0

I have a vps on main server. This vps has many web sites but traffic is normal. Yesterday server cpu usage went up suddenly. I don't know what this line means.

enter image description here

enter image description here

What is it and what should I do about it?

6
  • I dont know how to use wipefs. what is task of this? – avalkab Jul 26 '17 at 15:07
  • Each disk and partition has some sort of signature and metadata/magic strings on it. The metadata used by operating system to configure disks or attach drivers and mount disks on your system. You can view such partition-table signatures/metadata/magic strings using the wipefs command. The same command can erase filesystem, raid or partition-table signatures/metadata. – Ziazis Jul 27 '17 at 6:18
  • I dont want to do something wrong. I have to do carefully. How can i fix this? Which commands with? – avalkab Jul 27 '17 at 8:45
  • Look inside that script which is run, it is not the default path nor does it use the default parameters. So cat /etc/rc.d/init.d/wipefs (you can post the content in your post aswell) if it runs nothing that you want and you don't want it to be run at systemstart just move it out of /etc/rc.d or just remove it completly. – Ziazis Jul 27 '17 at 8:53
1

So I found wipefs running on a server, turns out it was a script that had come in through an outdated glassfish service. Please run dpkg --verify to see if your package has been compromised. You will need to get a copy of the original wipefs to put back the correct file. Check that your system hasn't been compromised and remove all the /etc/rc* scripts that it creates.

1
  • The other file that is involved is gety which actually spins up the fake wipefs service – Andre Van Zuydam Nov 17 '17 at 6:47
0

I fixed it by renaming /etc/rc.d/init.d/wipefs as suggested by @Ziazis

Look inside that script which is run, it is not the default path nor does it use the default parameters. So cat /etc/rc.d/init.d/wipefs. If it runs nothing that you want and you don't want it to be run at systemstart just move it out of /etc/rc.d or just remove it completely.

0

I had a similar case in a server running java/glassfish. In my case wipefs was run by /bin/wipefs and was also using 100% of my CPU. I found out that this system was compromised. What follows is the most important parts of my investigation.

A typical investigation to find out whether this is malicious code

# man wipefs

According to its description on the man page this executable has no reason running -- more so running for a long time and consuming a lot of CPU.

# locate wipefs | grep '/wipefs$' | xargs md5sum
f3798d1cdea6d4e6d18219c6d4380b4d  /bin/wipefs
f3798d1cdea6d4e6d18219c6d4380b4d  /etc/init.d/wipefs
c23a54b144df0e4cb7a2f5c2b87dec4c  /sbin/wipefs

Also it's not normal to have 3 copies of it in different places. And it's not at all normal to have an executable in /etc.

Googling the md5 of the suspicious /bin/wipefs you get results that suggest hacking/virus.

But let's proceed:

# dpkg --verify
??5??????   /bin/ss
??5?????? c /etc/sudoers
??5?????? c /etc/mysql/my.cnf
??5??????   /bin/netstat
??5?????? c /etc/crontab

dpkg --verify lists a few files that have been altered since installation. That is surely a sign of compromise for executables (like /bin/ss and netstat). Based on the description of the ss and netstat (man ss, man netstat) it's obvious that we have malicious code here that is trying to hide itself. A note of caution here: If dpkg -V reports nothing then don't put your guard down because it's not unlikely that the virus/hacker has taken steps to fool it.

In my case crontab had a line to run /bin/wipefs every 12 minutes. Again completely abnormal.

Let's look at the strings in wipefs:

strings /bin/wipefs
...
$stratum+tcp://pool.minexmr.cn:8888
...
Try "xmrminer" --help' for more information.

So we have code for "mining various cryptocoins" here.

Before turning off the machine it's good to have a look at the process with strace (if you're comfortable with it) or look at the files in /proc/ -- at least cat cmdline and ls -la fd. In my case the later pointed to an open log file with this content:

# head /tmp/mcalog 
CMD: /bin/wipefs -B -o stratum+tcp://pool.minexmr.cn:8888 -u 49ijJ3HJUg1b2MGnDmnEDJWdphGzWXgtbbBENx43NJiAUZWf8cSGryiZtYVZz3dgRcZH3Leokoqqi8SfRexMW32aFfvoHBp -p x -k
[2018-03-05 12:00:02] huge pages: available, enabled
[2018-03-05 12:00:02] cpu: ...
[2018-03-05 12:00:02] stratum url: stratum+tcp://pool.minexmr.cn:8888
[2018-03-05 12:00:02] backup url: none
[2018-03-05 12:00:02] Pool set diff to 80000.1
[2018-03-05 12:00:02] Stratum detected new block

So again a reference to pool.minexmr.cn. Let's use this bit of info to scan our system for other files that are probably related to this malicious code:

find /    -mount -type f -exec sh -c 'grep -q "\.minexmr\.\|wipefs" "{}"' \; -print ; find /tmp -mount -type f -exec sh -c 'grep -q "\.minexmr\.\|wipefs" "{}"' \; -print ; find /opt -mount -type f -exec sh -c 'grep -q "\.minexmr\.\|wipefs" "{}"' \; -print

--most likely malicious--
/etc/crontab
/var/tmp/gety
/bin/wipefs
/tmp/mcalog
/opt/glassfish3/glassfish/domains/domain1/applications/Sarketsdr/gety

--maybe malicious maybe not--
/sbin/swaplabel
/sbin/blkid
/sbin/wipefs
/usr/share/locale-langpack/en_GB/LC_MESSAGES/util-linux.mo
/usr/share/command-not-found/programs.d/amd64-main.db
/bin/mount

--probably no problem--
/var/cache/man/index.db
/var/lib/dpkg/info/util-linux.list
/var/lib/dpkg/info/util-linux.md5sums
/var/lib/mlocate/mlocate.db
/var/log/syslog
/var/log/syslog.1
/var/log/apport.log.1
/usr/lib/udisks2/udisksd

Note this file /opt/glassfish3/glassfish/domains/domain1/applications/Sarketsdr/gety. It's probably a good indication of the open door to our system.

So now we are 100% sure we are hacked and we have these things to do: 1) if possible keep a backup of this system for investigation, if not keep as much copies of files and the output of commands as possible 2) restore backup and check if it's clean -or- re-install OS 3) find out what we can do to avoid getting hacked again (at least not in the same way :-).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.