I found a suspicious process in the system (Ubuntu Server), which did not start itself. What actions to take?

Question

linux

Having nothing to do on the VPS, I launched topand saw with sorrow a strange process:

24051 webtin    20   0  9084 6680 1364 S    19  4.5 173:22.97 perl

It is strange in that perlfor me it is an absolutely unfamiliar contraption and I myself would most likely never have launched any pearl script. And it turns out that it has been working for several days and at certain moments it eats out 19% of the processor.
For me webtin, this is a system user (who is allowed a bash shell, but ssh login is prohibited). Under this user (via apache-mpm-itk) a couple of Wordpress sites work, and about a month ago I already cleaned the infection from them in the form of type files w36122986n.phpand w25363287n.phpwith evals inside. I updated Wordpress then, but, perhaps, I didn’t clean out all the bad things.
In administration, I, unfortunately, am a noob, so I would be very grateful for specific recommendations on analyzing and correcting the situation.

Answer 1

By the way: there is a small chance that the person before we feed the script to the pearl saved it to disk.
Then even if he deleted it after the start of execution, then / proc will still have its descriptor by which you can see the original file

Answer 2

Maybe it's the hoster that deploys images with this script?

Answer 3

as already suggested, strace will help to find out what system calls it makes, lsof - what files and sockets it opened.
cat /proc/24051/status | grep PPid
will tell you who created this process.

Answer 4

strace to the rescue

Answer 5

needs to be investigated further, but most likely it is malware. Judging by the connected libs, it works with the network. There are options here: connecting to the command center, sending spam, brute force some remote services. I met a similar one when I worked as an admin in a hosting support.
Searching for sources is difficult, because most often in this case the program is downloaded, launched and the file is immediately nailed, leaving only the process in memory.
Just nail this process and watch the UPU further - whether such processes will appear again. If they do, you have an unpatched remote vulnerability.

Answer 6

There was a similar malware, the source code was not found - it closes at startup and deletes it.
From memory, it seems, tied either to a vulnerability in the old sshd or to a substitution of sshd.
But I can be confused.

Answer 7

A little backstory. Once I went to a virtual machine in my jurisdiction, I noticed that ssh slows down strangely, after which I found a bunch of Python scripts in the top. The debriefing showed that they were brutalizing some Australian SIP provider and, apparently, they were flooded through a vulnerability in a web application that someone had once written self-written there. The application was not critical, so the virtual machine was immediately stopped, copied for further audit, and a new one from the template was deployed for the application, and, of course, it was delegated to programmers, incl. deal with leaky code. In general, it is a good practice that the compromised machine is not treated, but completely reinstalled, having removed a copy for auditing before that, and sometimes even simply taken offline and there it is picked right downtime already.
This is practical advice. See what's going on with the network. See if there is anything strange on netstat (open ports and sockets), you can even listen to tcpdump traffic. You can really trace the pearls and see where the legs grow from.
In general, restart Apache, if the process does not disappear (and most likely it will, I don’t remember Apache using pearl), kill it to hell with pid and kill -9. Then sit and watch if he appears again. View all possible logs and user directories with your hands, generally climb the entire file system. Then install rkhunter and another utility, I forgot the name. Scan there for strange activity. But it's still best to rearrange the system.
In general, it is necessary to install updates on time and monitor security issues at least superficially, regarding the distro and the packages used, all sorts of cms and so on.