How can you tell if a server has been hacked?

Question

Burglary protection

Good afternoon, colleagues.

A few days ago I had an unfortunate incident.
The leased VPS showed signs of hacking. Twice. In particular, RKhunter said that something has changed.

The situation is complicated by the fact that breaking the seover (in my opinion) could not happen from outside. There is reason to believe that this came from the host machine.

So, let's start from the moment when I reinstalled the system from the image, arguing that rather than cleaning up the consequences, it's easier to burn everything and rebuild it. At this moment the most interesting began. For a maximum of 5 hours, a repeated hacking occurred. This time it was confirmed by as many as 4 tests: rkhunter, tripwire, md5 -c against a pre-made reference file, plus a comparison of md5 infected and reinstalled files.

At that time, the system looked like this:
All packages are up to date.
sshd is configured only for authorization by keys and only for operation of the op 2 version of the protocol. Root access via ssh is denied.
installed and configured nginx, php-fpm, mysql. The web server showed two sites on the latest version of wordpress.
/var and /tmp were mounted with noexec.

Actually, only nginx, mysql and ssh stuck out. All the latest versions, all run by their own non-privileged users.

And this is where the fun begins. 600+ files in the /bin, /usr/bin, /sbin, /usr/sbin directories turned out to be changed.
The set is standard - ps, ls, lsmod and so on to hide the rootkit. BUT, only the super user can change all these files. I began to think that even if they found a 0-day exploit in the applications and got into the system through it, somehow bypassed noexec, then there was only one way to rise to the superuser - through a vulnerability in the kernel.
I looked at the kernel version (the hoster uses openvz, so it was not possible to change it).
It turned out like this: 2.6.18-308.el5.028stab099.3

As far as I understand, there are a lot of exploits for 2.6.18. Plus, the guys from openvz themselves say that this version of the kernel is no longer supported ... somewhere, moreover, 3 years already. (http://wiki.openvz.org/Download/kernel/2.6.18 This page was last modified on 22 October 2009, at 17:57.)

I write to the hoster and say “Maybe it’s worth checking? I'm not your only client. And in general, it would be time to upgrade. ”

To which I get the answer “We are fine, but your system has no signs of hacking. resizing files and checksums is not an indicator.”

This conversation lasted with varying degrees of activity for about 10 days. There are logs :)

Now it seems that there is no special point in reinstalling the system. If the host is hacked, I think at 0:00 it will check if there are fresh uninfected containers and correct the situation (it went something like this in time).

Even if the host is not hacked, the kernel version is also sad. This will perfectly allow you to repeat the hack (which happened almost immediately after the first reinstallation).

Actually questions:

1. Could it be that the system files have changed by themselves?
2. Is such a change, confirmed in 4 different ways, evidence of a hack?
3. If the host machine is hacked - what is the right thing to do? In addition to my problems, there is also a problem with the fact that other users are not aware.
4. Maybe I made a mistake somewhere and I’m wrong and I’m raising a panic in vain, blaming the guys.

For objectivity - www.habrastorage.com/images/colobrchc.png

PS If it suddenly coincided that we have the same hoster - check the system ... you never know :)

Answer 1

You can try sending suspicious files to virustotal. My files corrupted by the rootkit were identified as infected.

Answer 2

What does lsattr show for changed files? Usually, after replacing files that are designed to hide rootkits, attributes are hung on them so that when updating / reinstalling packages, these files do not change.

Answer 3

By the way, do you have prelink there for an hour? It is most likely that he walked through the binary files.

Answer 4

Why don't you want to switch to hardware virtualization? Saving? But how much time is wasted.
You now have a hoster who can go to your container with the vzctl enter command and he does not need your keys, this entry will not even be recorded.
1. Could it be that the system files have changed by themselves?
System files could not change themselves if you did not have yum -y update in your cron.
See logs /var/log/yum.log
2. Is such a change, confirmed in 4 different ways, evidence of a hack?
No, it's not.
3. If the host machine is hacked - what is the right thing to do? In addition to my problems, there is also a problem with the fact that other users are not aware.
Change the virtualization system, not save.

Answer 5

lkml.org/lkml/2011/9/30/425 There about checking not only the integrity of installed packages, but also their signatures. Well, and a little more stuff.

Answer 6

Prelink costs
Hashes were not found on the Internet :(
I rerolled the system. I installed extensions, took a picture of the system. We are waiting.