Block access via access.log

Question

My access.log are full of:

Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])
CCBot/2.0 (http://commoncrawl.org/faq/)
Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)

and of:

85.55.237.132 - - [16/Jul/2018:06:30:44 +0000] "GET /wp-login.php HTTP/1.1" 404 11462 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
51.253.65.140 - - [16/Jul/2018:06:30:47 +0000] "GET /wp-login.php HTTP/1.1" 404 7645 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
113.161.75.37 - - [16/Jul/2018:06:30:57 +0000] "GET /wp-login.php HTTP/1.1" 404 12145 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
114.94.208.2 - - [16/Jul/2018:06:31:04 +0000] "GET /wp-login.php HTTP/1.1" 404 11096 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
112.134.172.163 - - [16/Jul/2018:06:31:07 +0000] "GET /wp-login.php HTTP/1.1" 404 11177 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
144.48.75.34 - - [16/Jul/2018:06:31:13 +0000] "GET /wp-login.php HTTP/1.1" 404 7736 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"

And none of my sites is wordpress.

Currently I block access to these via UFW blocking the IPs, but they have become thousands and every day the number increases.

Is there any way to say to Ubuntu for example:

If the user-agent (or requested URL) contains "Mjbot" block it.

I've seen several tools like Fail2ban, modsecurity, etc, but they seem to complicate things rather than simplify them.

I'm looking for a really simple solution.

Answer 1

This can be done easily by the use of mod-rewrite. Just modify the start of your .htaccess to say

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} *Mjbot* [NC]
  RewriteRule .* - [F,L]
</IfModule>

You can replace *Mjbot* with a regex of your choice

The RewriteCond directive sets up a condition for the url to be 'rewritten' or modified. In this case, it checks if the variable HTTP_USER_AGENT matches the regular expression *Mjbot* with no case sensitivity [NC]

The RewriteRule directive tells apache to respond to all requests under this .htaccess (.*) to be responded with a 403 Forbidden (the F flag) and to stop the rewriting process immediately after (the L flag)

Furthermore you can also prevent such requests from cluttering your access log by use of the already packaged mod_setenvif

To do that, to your .htaccess add this line:

SetEnvIf User-Agent "*Mjbot*" dontlog

Answer 2

I think it's not working...

I have this rules in apache2.conf/httpd.conf

RewriteCond %{REQUEST_URI} wp-login.php$ [NC]
RewriteRule .* - [F,L]

If I text a URL with curl, I see this:

curl -Is https://www.alebalweb-blog.com/wp-login.php | head -n 1
HTTP/2 404 

Shouldn't I see 403?

In fact if I add the same rule directly on the .htaccess of the site I see this:

curl -Is https://www.alebalweb-blog.com/wp-login.php | head -n 1
HTTP/2 403 

So I did a little bit of research on the URL rewrite rules in apache2.conf/httpd.conf, and found lots of confusing people and a lot of confusion.

Among the many things some create a bit of clarity but also other confusion, such as this: http://tltech.com/info/rewriterule-in-htaccess-vs-httpd-conf/

Here it is said that the rules of url rewrite must be enclosed between tags:

<Directory></Directory>

But wanting to use apache2.conf/httpd.conf I did not understand in which, there are already 3 in my apache2.conf:

<Directory />
<Directory /usr/share>
<Directory /var/www/>

I tried to add a tag:

<Directory>

around my code at the beginning of the file, Apache didn't like it.

So I tried so:

<Directory />

apache does not complain, but it does not seem to work.

So I tried so:

<Directory /var/www/>

again apache does not complain, but it does not seem to change anything.

Then I tried, not to add a tag, but enter the URL rewrite rules directly in the existing tags, but apache did not like.

Then I searched again and found this: https://stackoverflow.com/questions/344588/how-can-i-implement-a-global-rewritecond-rewriterule-in-apache-that-applies-to

Here they talk about new options for url rewrite: InheritDown, InheritDownBefore, IgnoreInherit.

But again, wanting to use apache2.conf/httpd.conf, where should I put these new options and where my rules?

I'm sure all this can work, but I need to figure out where every single piece has to go to make everything work properly.

Can someone help me please?

Answer 3

Maybe you should buy a book about fail2ban, if there is one.

---

You posted some more questions in an answer. Wouldn't it make more sense to work on your initial question?

---

If you want to know my answer/opinion: Change from Apache to Nginx and do something like that:

if ($http_user_agent ~* "AhrefsBot")
{
    rewrite .* /error.php?e=403 last;
}

In error.php you can get more data from the requester via $_SERVER and mail it to you or better put it in a log file.

It is even more fun to prohibit SQL in GET parameters:

if ($query_string ~* "(alter|delete|update|select)(?![a-z._-])")
{
    set $lalala "bad_main_word";
}

if ($query_string ~* "1=1|ALL|DISTINCT|ON|AS|FROM|WHERE|GROUP|BY|HAVING|WINDOW|UNION|INTERSECT|EXCEPT|ORDER|ASC|DESC|USING|NULLS|FIRST|LAST|LIMIT|OFFSET|ROW|ROWS|FETCH|NEXT|ONLY|FOR|NO|KEY|SHARE|OF|NOWAIT|LATERAL|NATURAL|TABLE|CURRENT|RETURNING")
{
    set $lalala "${lalala}_and_bad_sub_word";
}

if ($query_string ~ "Put some RegEx here to whitelist your stuff, check your mail/log!")
{
    set $lalala "whitelist";
}

if ($lalala = "bad_main_word_and_bad_sub_word")
{
    rewrite .* /error.php?e=403 last;
}

Of course, you still need the following.

location = /error.php {
    alias /var/www/error.php;
    fastcgi_index error.php;
    fastcgi_pass php;
    include fastcgi_params;
    internal;
}

error_page 403 /error.php?e=403;