0

I added this in my htaccess:

RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} ^SemrushBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Yandex [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^dotbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^CCBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^spbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mjbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MauiBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SeznamBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Barkrowler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MegaIndex [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^AlphaBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Cliqzbot [NC,OR]
RewriteCond %{REQUEST_URI} wp-login.php$ [NC]
#RewriteRule .*  - [L,F]
RewriteRule . - [F,L]
#RewriteRule ^.* - [F,L]

Looking for example MJ12bot in my access logs with this:

grep --text MJ12bot /var/log/apache2/access.log

I should not see towards the end HTTP/1.1 403?

Instead I see:

HTTP/1.1" 200 10841 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"

is something not working? What?

for wp-login.php seems to work, I see:

HTTP/1.1" 403 24328 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"

What's wrong?

  • Take out the "^". Of those bots in your list that I have seen before, only SemrushBot doesn't obey, or even look at, the robots.txt file. – Doug Smythies Jul 27 '18 at 13:21
  • Seems to work now, thank you very much. But what do you mean by "Semrushbot doesn't obey "? Can't it be blocked via htaccess? Do I necessarily use robots.txt? – alebal Jul 27 '18 at 23:06
  • Yes, it can be blocked by .htaccess (and indeed that is how I do it). I just meant that if you were have a robots.txt file, the others in your list that I know of (which isn't all of them) seem to obey a DISALLOW directive and so I don't think the .htaccess directive is needed. – Doug Smythies Jul 27 '18 at 23:16

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.