It turns out that fail2ban can be modified to include this as an unwanted event. Change/add a ssh config file in filter.d to include a filter for access as root: https://serverfault.com/questions/340565/how-to-block-all-root-login-attempts-using-denyhosts-and-or-fail2ban
Then fail to ban will more-quickly respond to those logins, but not as fast as dropping them on the spot (which it turns out would require changing code and recompiling sshd)
Lowering the bantime to a few hours prevent a huge list of blocked ips the firewall would have to sort through, reducing overhead and connection time. The failed password, and login-as-root bantime can be configured separately. This way if a user has CAPS on, its only a few minutes, but brute force bots get blocked for 2 hours. (noticed the default bantime is 10 minutes, and most bots simply wait 10:01 minutes and resume their attempts)
Reducing the log-levels cuts back on all the log files and their memory overhead. Reducing the fail2ban findtime, reduces the amount of logfiles fail2ban holds in memory!
Having the correct settings is important, because all 3 of these systems create a memory usage love triangle! Not blocking the bots soon enough, results in huge log files which means more memory usage by rsyslog. Fail2ban holds these logs in memory to scan (
# of attacks/second * by
findtime). And the sshd threads backup from all the concurrent attempts the bots are permitted to make. Keep the
bantime too short and the bots come right back creating more sshd threads. Have
bantime set too long and the firewall has to run through a long list of permissions held in memory.
bantime: 5-10 minutes for bad passwords
bantime: 2-6 hours for attempts as root (not days, certainly not 99999999)
log-files: Warn (not info!)
findtime: 5-15 minutes on SSH (log memory usage reduced by quick bans)
Also config the max attempts permitted in sshd to be 2 because it will dump them to the watched log file much faster
The upcoming change in brute force attacks (which is why it was harder on us) is to run them concurrently. If you are going to block them after some attempts are made, then start 20x ssh connections and try 20 brute forces at once! All the configs above are only effective against a single ip, single connection attack. They are less effective if multiple ssh connections are run at the same time.
This can be blocked at the firewall preventing more than N connections from a given IP at one time: https://serverfault.com/questions/275669/ssh-sshd-how-do-i-set-max-login-attempts