What utilities to use to investigate server load on centos?

Question

System administration

Hello.
Help in "investigating" the sudden load of the server.
Sites on wordpress work on this server, the sites are not large and the traffic is also not high.

And here are the surges for yesterday:

Please tell me what commands to use in order to find the cause of these sudden loads. Which process loaded the server.
Please describe how you would act in such a situation.
Here's what it shows sar -u

02:10:01 AM     all     85.92      0.00     14.06      0.00      0.00      0.02
02:20:01 AM     all     83.60      0.00     12.96      0.16      0.00      3.27

Answer 1

In my opinion, the easiest option without setting up different composite monitoring systems (zabbix and etc) is atop . By default, it gives statistics for a 10 minute interval, but this can be changed. Considering that you have CentOS, do not forget to add it to autostart (it is not automatically added there). At least to answer the question "what process is loading" it will often help and indicate in which direction to dig.
Unfortunately, if it turns out that this is some kind of apache, mysql, php, it would still be highly desirable for you to catch the process itself at the time of loading and see what it is doing directly, collecting information about it with all sorts of strace, lsof or more specific tools. It will also be useful to configure advanced logging for it.
As Alfss rightly pointed out, communication with logs, most likely, cannot be avoided. Rarely do they even make it clear, even without monitoring, what the problem is. However, their analysis is laborious and does not always lead to results, especially if you do not know what to look for and how the logs usually look (when there are no failures).

Answer 2

It is worth starting by analyzing the web server logs for the last 3 days in order to find out the number of requests processed per hour. For this purpose, you can use the q utility .
The load can be created by search bots and scanners of all sorts of malware.

Answer 3

htop, atop, strace, iotop for starters. What then can grow into a full-scale investigation and collection of statistics / processing of logs with zabbix, elk, some kind of intrusion prevention system, or whatever you choose, with a likely return to strace again.