How to catch a malicious process?

Question

linux

Situation : There is a server running Linux (Ubuntu 12.04) without a graphical shell. Previously, the work was more or less flawless. But recently, some of the software began to behave inappropriately.

It was noticed that access to the site is difficult (to put it mildly). I connect via SSH and observe terrible brakes at the same time. It took about 1.5 minutes from the moment the “top” command was entered to the output of information. But the information received was not informative - the most active of the processes used 30-40% of the processor time (2 cores, i.e. we have the most available 200% per process, but I haven’t seen this for a long time), I also didn’t see anything suspicious for memory consumption. The "df" command indicates that there is enough free space.

There are two hypotheses: 1) some process is obviously loading the disk subsystem by shoveling/creating files (or other file system objects); 2) some processes stubbornly send / download something over the network.

I ask for help with commands that can show in dynamics which processes: 1) read / write a large number of files; 2) create maximum network traffic.

Answer 1

The title of the question and your nickname together deliver :)
Related: iotop, nethogs, iptraf

Answer 2

> 1) read/write a large number of
iotop files

Answer 3

Let's just hope it's not a rootkit. Otherwise, it will be a little more difficult to find it.

Answer 4

detailed description of all tops.
habrahabr.ru/post/114082/
It may also come in handy:
iftop - real-time channel load monitoring
ethstats - shows the number of connections and channel load
cat /proc/net/ip_conntrack - shows active connections
a lot of interesting things can be found out with the stat command, although The above should be enough for you to diagnose the problem.

Answer 5

There is also iostat, sar, pidstat, etc from the sysstat package. Very helpful in situations like this.

Answer 6

The network does not give overload, if something is network and slows down, then because of the disks, for example, torrents quite seriously load the disk subsystem.
First, check the readings of sar -d 1 10. If the disks are loaded, dig further towards iotop, iostat, if not, see the memory using the free or top command.

Answer 7

Another option: the RAM ends and active swapping begins. You can see both by top (low free memory, lots of swap used) and by iotop (active kswap).

Answer 8

look at the memory with the same top - sometimes the kernel starts swapping and all the resources go to this business

Answer 9

By the way, as noted above about rootkits, I advise you to check the machine, but if the rootkit is already running, it may well hide, for such situations they usually raise a copy of the machine on a virtual machine, with traffic monitoring and a total check of the system - hashes and signatures of packages and checksums