How to find the cause of the server crash?

Question

linux

Good afternoon.
Perhaps someone can suggest what to do with such a problem, not only referring to Google.
The problem is this: a week ago it all started with the fact that all the memory ended abruptly and the culprit seemed to be mysql, the hoster said increase the server memory, then the people said look for heavy requests. In general, it was not possible to find it, they suffered, they suffered, but suddenly the memory stopped eating, but the channel began to clog. He also raised the topic - supposedly the load comes from the database, but in the end, after a couple of days everything returned to normal. Here a couple of days a new misfortune again sites do not work on the server. The memory is not fully clogged with an approximate usage of 8 out of 16. atop shows that the channels are free, the load on the CPU from mysqld is 600-900%. I start saving to a file what kind of requests are now in the queue, and there are only 5 of them with a timing of 0 and half of them are sleep. those. some kind of garbage - there are no requests, but the site is lying and the load of mysqld is supposedly hoo.
I start htop and see that the value of tasks in the region of 400
restarted the server completely, no reaction, everything is the same, no clearance is visible,
I decided to go differently and started stopping services, I stop mysqld first, I see that the load seems to have fallen, but the tasks remained, I open the site in the browser expecting to see an error connecting to the database, but instead the site hangs, after a minute I gave out 504, I update the 502 page, but there are no errors about connecting to the database. I stop all sorts of memcached, httpd and nginx , refresh the page in the browser and get 502 or 504 again instead of check the connection.
while tasks are around 100 and the CPU is almost at zero

total from yesterday (from 16) to 23:55 with periodic flashes for 10 minutes only worked. Then, from 0:30 to 5:00, it fell off again and only from 5 in the morning the site started working

I wrote to the hoster about what I see, what the administration of your server got on you, you want us to figure out what kind of problems you have 3000/hour

Please help tips on what to do, what to try

Answer 1

you can limit the number of connections to MySQL if it's memory under load.
and so you need to look at the logs, and the parameters, etc. it is not clear what is happening.

Answer 2

1) Deny access to mysql from outside!
2) Limit access to the server by setting the number of connections from one IP
3) Install a firewall and disable connections to all ports except ssh and http/https
4) Set mysql logs for long queries
5) Analyze the application logs, as mysql queries it looks like you are coming from inside the application
6) What kind of application do you have there? Isn't it taken?

Answer 3

Is there a site backup? At the beginning of February, they hacked my site on the old Bitrix, only by comparing it with a clean backup, I managed to find and clean out the malware scattered throughout the site. True, they did not create any special problems, the hack was discovered almost by accident.