Why did Debian start eating up all the RAM?

Question

Debian

Good afternoon. Faced such a problem.
There is a web server running Debian.
I noticed that the server started to work slower, got into the logs and saw that spam was being sent smoothly to non-existent mailboxes on my domain.
Now, after starting the server, the RAM is being filled with processes, literally after 5 minutes of operation, the server hangs dead with the messages "Out of memory killed process"
Has anyone encountered a similar one?
And how can you find out why it happened? Through what hole malefactors got on the server.

Answer 1

If the server is physically unavailable, then To begin with, it will turn on the firewall and disable outgoing connections on it, and incoming ones too, except for ssh.
Then see what processes are eating up memory. Kill a parent. View startup scripts for malware.
If the server is physically accessible, then boot from the SD, for example, and go through the file system ..
Well, they didn’t say what it is on the server ..
If the root password is not selected, then php scripts (leaky CMS) are most likely broken.
In general, firewalls with the closure of client connections rule from such situations. Even if it is broken, it is difficult to go beyond the server. On firehol it is done in 3 lines, I recommend it, strongly!
Well, in the case, first turn off the mail, then the web server, then the cron, and so on, until you find the cause. Well, look at the issuance of top, who and what users are eating memory and producing processes.