How to fight (find and destroy) a rootkit on a Linux server (in my case, Ubuntu Server)?

Question

linux

So, the situation, unfortunately, is standard: I was not worried about protecting the server, and for this, it was broken. Everything is fair, no claims to fortune. But now the question arose about avoiding mistakes in the future.
The VPS is running Ubuntu Server 10.10 (the firewall was not turned on), when they started brute-forcing anyone from it, my hoster got abused, and he gave me a day to close the holes and report, otherwise my account will be killed. Hoster Hetzner.de - well done, others probably would have immediately banged the account, but these gave time to eliminate the gaps.
Next, I will describe the actions that I took.
I ask qualified Khabrovites to suggest:

1. What did I do after the attack , and what of the necessary did not do?
   
2. How now to look for a rootkit?
   
3. If there are any comprehensive manuals on basic protection in Russian or plain English, then poke into them, please.
   

I further distorted my IPs, and left the IPs of the bad guys unchanged (in case someone googles salvation from them).

~# sockstat | grep 22<br/>
root sshd 1795 tcp4 79.47.35.666:22 96.156.140.666:54113 ESTABLISHED # моё<br/>
root sshd 17351 tcp4 *:22 *:* LISTEN<br/>
root sshd 17364 tcp4 79.47.35.666:22 2.2.44.3:51557 ESTABLISHED # чужое<br/>
root sshd 17365 tcp4 79.47.35.666:22 2.2.44.3:51557 ESTABLISHED # чужое<br/>
root sshd 18871 tcp4 79.47.35.666:22 96.156.140.666:57163 ESTABLISHED # моё

I tried to see who is using ssh at all:

~# ps ax | grep ssh<br/>
 1795 ? Ss 0:02 sshd: [email protected]/0<br/>
17351 ? Ss 0:00 /usr/sbin/sshd -D<br/>
18871 ? Ss 0:00 sshd: [email protected] # это точно не моё<br/>
18886 ? Ss 0:00 /usr/lib/openssh/sftp-server<br/>
30370 ? Ss 0:00 sshd: root [priv]<br/>
30371 ? S 0:00 sshd: root [net]<br/>
30373 pts/0 S+ 0:00 grep --color=auto ssh

Banged the process:
~# kill 18871
Turned on the firewall, added the rules:

~# ufw status<br/>
Status: active<br/>
<br/>
To Action From<br/>
-- ------ ----<br/>
Apache ALLOW Anywhere<br/>
Postfix ALLOW Anywhere<br/>
Anywhere ALLOW 97.157.140.172<br/>
Anywhere ALLOW 194.247.190.1<br/>
22 LIMIT Anywhere<br/>
21 LIMIT Anywhere<br/>
<br/>
22 DENY OUT Anywhere<br/>
21 DENY OUT Anywhere

And sockstat now says that no one climbs on IP 2.2.44.3.
I went to look for who and how came to the server.

~# cat /var/log/auth.log | grep &quot;Accepted &quot;<br/>
...<br/>
Mar 13 23:23:22 ubuntuserver sshd[27438]: Accepted password for webmaster from 114.80.100.241 port 37966 ssh2<br/>
Mar 13 23:23:22 ubuntuserver sshd[27439]: Accepted password for webmaster from 114.80.100.241 port 47732 ssh2<br/>
...<br/>
Mar 15 07:39:58 ubuntuserver sshd[6320]: Accepted password for webmaster from 79.117.72.150 port 1217 ssh2<br/>
...

I had such a user with access to the console (the user was created a long time ago, it is possible that his password was funny or that the password was set on fire by one of the employees on the local computer through a virus, because the ftp password is the same and it was saved in clients ).
But this webmaster was not in sudoers. It turns out that I have some kind of rootkit?
Installed rkhunter and chkrootkit
The first one found the following suspicions:

[14:57:13] Checking for hidden files and directories [ Warning ]<br/>
[14:57:13] Warning: Hidden directory found: /dev/.udev<br/>
[14:57:13] Warning: Hidden directory found: /dev/.initramfs<br/>
[14:57:13] Warning: Hidden file found: /dev/.blkid.tab: ASCII text<br/>
[14:57:13] Warning: Hidden file found: /dev/.blkid.tab.old: ASCII text

But on the Internet bugs.launchpad.net/ubuntu/+source/rkhunter/+bug/86153 they say that he swears in vain.
The second found only this:

Checking `chkutmp'... The tty of the following user process(es) were not found in /var/run/utmp !

But these were my own root sessions running inside screen.
And ... it turns out that everything else is clean.
How to search for a rootkit now?

Answer 1

Rebuild the system from scratch. This is most correct for the server.

Answer 2

> But this webmaster was not in sudoers. It turns out that I have some kind of rootkit?
Nothing works, most likely some kind of exploit. It is better to roll over the system from scratch and any non-static (php scripts etc.) from the backup. Otherwise, there is a chance that something will not be found.

Answer 3

1. Try the unhide utility first to find the hidden process (if there is one)
2. reinstall all packages. Here is the script:
#!/bin/bash
for pkg in `dpkg --get-selections|awk '{print $1}'| egrep -v '(dpkg|apt)'`; do apt-get install --reinstall -y $pkg; done
3. Use tripwire to check file integrity (for the future)

Answer 4

Do not be greedy, hire an expensive admin, let him do everything for you and tell you later what and how he did, and what and how you need to do. This is a one-time fee, as a result, if you stretch it for the duration of the server, it will not be expensive.

Answer 5

I once met with a rootkit, the rkhunter did not find it, it was about 4 years ago, I don’t remember the details, he skillfully hid from the TOP ps and much more, but he was blessed with an outgoing connection.
further along the chain of this connection, a “kernel module” was found ... reinstallation resolved the issue.
also recently, one of my friends had a site on a null engine that had not changed for 5 years, and through it they downloaded a file (shell) to me, collected the “client irk” and also hooked up to a remote server. the process was running as user apache
After clearing a lot, I found a file that was used by apache. a simple f3 on the executable file showed the connection parameters with the irk server, I went there and talked with the admin of that channel, he had more than 250 such bots.
he steered all the servers directly from the channel, he said that he was selling servers to spammers, he said how he got to me, he said that he was from Malaysia. sells servers from 5 to 100 bucks.
asked me to give him a shell only for “IRKproxy” nothing more :))
it makes sense to look for “php shell” or similar “things”
I had a perl picture by the way, a jpg file inside with a pearl script for assembling the irk daemon :))
experience I then got a lot :)

Answer 6

In this case, I do data migration to an intermediate server, reinstall the current one and migrate back. Well, or just to another clean server.
Linux is a complex system. Every system administrator wants to know where the rootkit sits. But there are too many places like this. I missed something and all the many days of work for nothing. In the future, the reinstall is easier.

Answer 7

It is strange in general that the hoster informed you about this after complaints, and not you yourself found out.
By the way, due to the stinginess of server holders towards normal admins, this happens, although to maintain an average server remotely, it is quite normal to have at least a remote admin.
That is, it turns out that you did not properly monitor the server, nor did you monitor it, did not look at the logs - so what do you want?
Ideally, set up a server from scratch, hire a person to adequately set up protection. Further, either pull up knowledge yourself, or leave a person on a salary, at least for the initial time.
I don’t understand how you can not track visits via ssh - this is the time. (at least logwatch would have been screwed)
I don’t understand how you can leave ssh on a standard port and not block the number of connection attempts to it from one IP - that’s two.
And I don’t understand at all how you can dive into the ocean without knowing how to swim - these are three.

Answer 8

If you have a kernel with module support, then there is also the possibility of kernel rootkits.
Reinstalling the server is the easiest and fastest way to fix the problem.

Answer 9

You should also configure SSH if you think: disable protocol version 1, disable root login, disable password authorization and leave only by keys. Take care of setting up access rights: remove all users from the wheel group (ideally, leave each user in their own group altogether), remove sudo.
You can look behind the rootkit if it didn’t get into the kernel using the lsof utility - in Linux it’s difficult to hide work with any files, look for all sorts of entries, like (deleted), maybe you’re lucky. But if the hacker got root, then he most likely put a module into the kernel that is able to hide, which is quite simple and students do it in a month, as it were, the power of Linux is playing against him here. Only reinstallation will save you from this. And take care of root in the future, like the apple of an eye.

Answer 10

rearrange the SSH daemon, its binary can be replaced, and log in with a password, but it will not be logged, and the attacker will easily log in, change the password at least 10 times