How to find a miner on linux, if everything is fine in the logs?

Question

linux

Gentlemen, the task of researching a black box or a cat in a poke or a horse in a vacuum (call it what you want).
We have node with ubuntu server. [ Linux version 4.4.0-87-generic ([email protected]) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ]
There is malicious code on the server (presumably some kind of miner) The task is to find it.
According to the logs, everything seems to be fine. There was no compromise of the system from outside. There are suggestions that it came either with some distribution kit or the repository was compromised. There is no whitelist of packages and applications. There is no monitoring of file changes. The period when this happened
RKHUNTER didn't find anything, tell me where to dig and what to look for ?

Answer 1

It is rightly said that a miner must eat a processor in order to be useful. All cores on the server are loaded almost at 100%.
The top command will help you
A netstat -nap | grep -i 'established' will show all established TCP sessions
and tcpdump will help catch outgoing traffic.
Just recently I struggled with a miner on a leaky server.
They upload via php to the /tmp folder,
then they look to see if there is a miner in the ps afx memory and run it via nohup.
Since other people are involved in leaky sites, I had to prohibit php from running processes.
It helped with the miner. Then the fight against spam began, and using the server to hack and attack other sites. I had to cut off Apache's ability to initiate outgoing connections and drop all UDP traffic. This is the only possibility, since developers cannot patch sites for half a year. Sites are still crappy, but miners don't take root, spam doesn't get sent, and attacks don't happen. The host does not complain. The sites are working.

Answer 2

The miner loads the CPU, the processes that load the percent have fallen, use strace if necessary

Answer 3

Make sha1sum of all files on the patient.
Raise the second one with the same set of packages.
Check out what's different.
Define from .deb it or otherwise.
If otherwise, then look with "handles" - monitor ports and calls to ip dns
ps: etckeeper is not there either?

Answer 4

I discovered one of the last such miners simply by stopping all the services known to me and looking at the list of running processes. once a service shutdown caused a "drop" in traffic and so localized the location of another miner