My access.log are full of:

Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
CCBot/2.0 (http://commoncrawl.org/faq/)
Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)

and of: - - [16/Jul/2018:06:30:44 +0000] "GET /wp-login.php HTTP/1.1" 404 11462 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" - - [16/Jul/2018:06:30:47 +0000] "GET /wp-login.php HTTP/1.1" 404 7645 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" - - [16/Jul/2018:06:30:57 +0000] "GET /wp-login.php HTTP/1.1" 404 12145 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" - - [16/Jul/2018:06:31:04 +0000] "GET /wp-login.php HTTP/1.1" 404 11096 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" - - [16/Jul/2018:06:31:07 +0000] "GET /wp-login.php HTTP/1.1" 404 11177 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" - - [16/Jul/2018:06:31:13 +0000] "GET /wp-login.php HTTP/1.1" 404 7736 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"

And none of my sites is wordpress.

Currently I block access to these via UFW blocking the IPs, but they have become thousands and every day the number increases.

Is there any way to say to Ubuntu for example:

If the user-agent (or requested URL) contains "Mjbot" block it.

I've seen several tools like Fail2ban, modsecurity, etc, but they seem to complicate things rather than simplify them.

I'm looking for a really simple solution.

  • I think you definitely need an additional module, as the mentioned in the question, to issue commands from Apache2 to the firewall. Here is a guide with few basics how to do that: askubuntu.com/a/922144/566421 – pa4080 Jul 17 '18 at 8:24

This can be done easily by the use of mod-rewrite. Just modify the start of your .htaccess to say

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} *Mjbot* [NC]
  RewriteRule .* - [F,L]

You can replace *Mjbot* with a regex of your choice

The RewriteCond directive sets up a condition for the url to be 'rewritten' or modified. In this case, it checks if the variable HTTP_USER_AGENT matches the regular expression *Mjbot* with no case sensitivity [NC]

The RewriteRule directive tells apache to respond to all requests under this .htaccess (.*) to be responded with a 403 Forbidden (the F flag) and to stop the rewriting process immediately after (the L flag)

Furthermore you can also prevent such requests from cluttering your access log by use of the already packaged mod_setenvif

To do that, to your .htaccess add this line:

SetEnvIf User-Agent "*Mjbot*" dontlog
  • Maybe I'm wrong, but so they access the site and then are blocked by htaccess, I would rather block them on access, as does UFW with IP. And then in this server I have almost 50 sites (and 50 htaccess) and every day comes some new access that I do not want, are 50 htaccess to change every day, it becomes an endless work... – alebal Jul 16 '18 at 19:25
  • @alebal This need not be in the .htaccess in that case. It could very well be in the root httpd.conf for it to apply to all websites. Since the User-Agent is actually part of the request body, unlike the IP address, any ufw based solution would have to search for a string in each packet and that sounds like a way to get DDoS'd quite easily – Amith KK Jul 16 '18 at 19:56
  • 1
    @alebal What I told Amith in our chat room here is valid - UFW doesn't out of the box come with the capacity to do searching within the request's contents to find a user-agent string. There is the capability to do that in iptables (which ufw under the hood utilizes), but it's not something you can add to UFW directly. Not to mention there are hundreds of User Agent strings you're probably wanting to block at that point, so you would have to manage a couple hundred rules simultaneously. – Thomas Ward Jul 16 '18 at 20:27
  • Sorry, a doubt... I should add all the code block (Rewritecond... etc.) in my httpd. conf file (which I don't have, my I think it's called apache2. conf) or create a htaccess in the directory www (root of my server)? – alebal Jul 17 '18 at 20:10
  • @alebal If you want to do it on all websites, your apache2.conf. if you want it for specific websites that you serve, in your .htaccess for those websites – Amith KK Jul 17 '18 at 20:14

I think it's not working...

I have this rules in apache2.conf/httpd.conf

RewriteCond %{REQUEST_URI} wp-login.php$ [NC]
RewriteRule .* - [F,L]

If I text a URL with curl, I see this:

curl -Is https://www.alebalweb-blog.com/wp-login.php | head -n 1
HTTP/2 404 

Shouldn't I see 403?

In fact if I add the same rule directly on the .htaccess of the site I see this:

curl -Is https://www.alebalweb-blog.com/wp-login.php | head -n 1
HTTP/2 403 

So I did a little bit of research on the URL rewrite rules in apache2.conf/httpd.conf, and found lots of confusing people and a lot of confusion.

Among the many things some create a bit of clarity but also other confusion, such as this: http://tltech.com/info/rewriterule-in-htaccess-vs-httpd-conf/

Here it is said that the rules of url rewrite must be enclosed between tags:


But wanting to use apache2.conf/httpd.conf I did not understand in which, there are already 3 in my apache2.conf:

<Directory />
<Directory /usr/share>
<Directory /var/www/>

I tried to add a tag:


around my code at the beginning of the file, Apache didn't like it.

So I tried so:

<Directory />

apache does not complain, but it does not seem to work.

So I tried so:

<Directory /var/www/>

again apache does not complain, but it does not seem to change anything.

Then I tried, not to add a tag, but enter the URL rewrite rules directly in the existing tags, but apache did not like.

Then I searched again and found this: https://stackoverflow.com/questions/344588/how-can-i-implement-a-global-rewritecond-rewriterule-in-apache-that-applies-to

Here they talk about new options for url rewrite: InheritDown, InheritDownBefore, IgnoreInherit.

But again, wanting to use apache2.conf/httpd.conf, where should I put these new options and where my rules?

I'm sure all this can work, but I need to figure out where every single piece has to go to make everything work properly.

Can someone help me please?


Maybe you should buy a book about fail2ban, if there is one.

You posted some more questions in an answer. Wouldn't it make more sense to work on your initial question?

If you want to know my answer/opinion: Change from Apache to Nginx and do something like that:

if ($http_user_agent ~* "AhrefsBot")
    rewrite .* /error.php?e=403 last;

In error.php you can get more data from the requester via $_SERVER and mail it to you or better put it in a log file.

It is even more fun to prohibit SQL in GET parameters:

if ($query_string ~* "(alter|delete|update|select)(?![a-z._-])")
    set $lalala "bad_main_word";

    set $lalala "${lalala}_and_bad_sub_word";

if ($query_string ~ "Put some RegEx here to whitelist your stuff, check your mail/log!")
    set $lalala "whitelist";

if ($lalala = "bad_main_word_and_bad_sub_word")
    rewrite .* /error.php?e=403 last;

Of course, you still need the following.

location = /error.php {
    alias /var/www/error.php;
    fastcgi_index error.php;
    fastcgi_pass php;
    include fastcgi_params;

error_page 403 /error.php?e=403;
  • I don't want use nginx – alebal Jul 21 '18 at 20:23
  • OK, too bad, it's really better/no rubbish. ;) – uav Jul 21 '18 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.