What could be causing the high load?

Question

linux

Good afternoon.
The server is heavily loaded, and there are not so many visitors.
Intel(R) Xeon(R) CPU X5650 @ 2.67GHz 1596.000 Mhz X 24 4x512GB
HDD (Hardware RAID-10)
64GB RAM

]ab -n 5000 -c 10
Server Software: nginx/1.0.15
Server Hostname: webserver
Server Port: 80
Document Path: /
Document Length: 45318 bytes
Concurrency Level: 10
Time taken for tests: 340.275 seconds
Complete requests: 5000
Failed requests : 4552
(Connect: 0, Receive: 0, Length: 4552, Exceptions: 0)
Write errors: 0
Total transferred: 228864441 bytes
HTML transferred: 226594441 bytes
Requests per second: 14.69 [#/sec] (mean)
Time per request: 680.549 [ms] (mean)
Time per request: 68.055 [ms] (mean, across all concurrent requests)
Transfer rate: 656.82 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 29 49 203.9 33 3066
Processing: 568 631 39.3 624 1380
Waiting: 505 558 35.8 551 1313
Total: 599 680 208.0 659 3900 Percentage of the requests served
within a certain time ( ms )

top - 14:03:59 up 7 days, 2:26, ​​1 user, load average: 5.85, 3.99, 2.29
Tasks: 486 total, 10 running, 476 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.2%us , 8.2%sy, 0.0%ni, 66.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0
%
st free, 20301296k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
340 root 39 19 0 0 0 R 100.0 0.0 9169:33 [kipmi0]
3314 mysql 20 0 18.2g 2.2g 5312 S 72.3 02.5 50: usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/my
24534 old_kame 20 0 297m 50m 6160 R 11.3 0.1 0:00.34 /usr/bin/php- cgi
php
6140 S 11.3 0.1 0:00.34 /usr/bin/php-cgi php
24537 old_kame 20 0 288m 40m 6064 S 9.6 0.1 0:00.29 /usr/bin/php-cgi php
24538 old_kame 20 0 281m 36m 5924 0 00.20 /usr/bin/php-cgi php
24539 old_kame 20 0 281m 36m 5924 R 6.3 0.1 0:00.19 /usr/bin/php-cgi php
24540 old_kame 20 0 278m 32m 5924 R 5.3 0.1 0:00.16 /usr/bin/bin/bin
/usr/bin/php-cgi php 24542
old_kame 20 0 267m 21m 5924 R 2.3 0.0 0:00.07 /usr/bin/php-cgi php
14296 Apache 20 0 98152 4824 1852 S 1.0 0.0 1: 14.22 NGINX: Worker Process 445 Apache
20 0 299m 14M 3456 S 0.7 0.0 0: 09.42 /
USR / SBIN / HTTPD 7652 Apache 20 0 298m 12M 1960 S 0.7 0.0 0: 07.47 / usr/sbin/httpd 24434 root 20 0 15308
1568 948 R 0.7 0.0 0:00.05 [ksoftirqd/3] 121 root 20 0 0 0 0 S 0.3 0.0 0:07.97 [events/22] 471 apache 20 0 310m 25m 3604 S 0.3 0.0 0:10.13 /usr/sbin/httpd 475 apache 20 0 315m 29m 1980 S 0.3 0.0 0:10.18 /usr/sbin/httpd
2169 root 20 0 0 0 0 S 0.3 0.0 1:25.64 [kondemand/0] 2180 root 20 0 0 0
0 S 0.3 0.0 0:21.36 [kondemand/11]
kondemand/20]
2191 root 20 0 0 0 0 S 0.3 0.0 0:06.55 [kondemand/22]
2192 root 20 0 0 0 0 S 0.3 0.0 1:00.06 [kondemand/23]
7654 apache 20 0 413m 129m 3588 S 0.3
/usr/sbin/httpd
9882 apache 20 0 412m 127m 3384 S 0.3 0.2 0:02.72 /usr/sbin/httpd 1
root 20 0 19364 1544 1232 S 0.0 0.0 0:01.74 /sbin/init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kthreadd]
3 root RT 0 0 0 0 S 0.0 0.0 0:00.11 [migration/0]
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 [migration/ 0]

iostat -dx 2
Linux 2.6.32-431.20.3.el6.x86_64 (webserver) 08/04/2014 _x86_64_ (24 CPUs)
Device: rrqm/s wrqm/sr/sw/s rsec/s wsec/s avgrq-sz avgqu- sz await svctm %util
sda 0.02 7.35 0.32 2.54 35.90 77.72 39.72 0.01 2.34 0.36 0.10
dm-0 0.00 0.00 0, 34 9.72 35.88 77.72 11.30 0.36 35.78 0.10 0.10
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0 .00 0.98 0.88 0.00
dm-2 0.00 0.00 0.00 0.00 0.01 0.00 7.95 0.00 0.89 0.50 0.00
Device: rrqm /s wrqm/sr/sw/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 34.50 0.00 15.50 0.00 400.00 25.81 0.00 0.06 0.03 0.05
dm-0 0.00 0.00 0.00 50.00 0 .00 400.00 8.00 0.01 0.14 0.01 0.05
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/sr /sw/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 32.00 0.00 4.00 0.00 284.00 71.00 0.00 0.00 0, 00 0.00
dm-0 0.00 0.00 0.00 35.50 0.00 284.00 8.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0, 00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/sr/sw/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 92.00 0.00 31.00 0.00 984.00 31.74 0.04 1.26 0.05 0.15
dm-0 0.00 0.00 0.00 123.00 0 .00 984.00 8.00 0.12 0.95 0.01 0.15
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/sr/sw/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 29.00 0.00 10.00 0.00 308.00 30.80 0.01 0.60 0.05 0.05
dm-0 0.00 0.00 0.00 38.50 0.00 308.00 8.00 0.01 0.22 0.01 0.05
dm- 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0 .00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/sr/sw/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0 .00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0, 00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0 .00 0.00 0.00
Device: rrqm/s wrqm/sr/sw/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 39.50 0.00 2.50 0 .00 336.00 134.40 0.00 1.00 0.20 0.05
dm-0 0.00 0.00 0.00 42.00 0.00 336.00 8.00 0.04 1.00 0.01 0.05
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Answer 1

First, ditch php-cgi in favor of, for example, mpm-itk. And install some PHP accelerator, if not already.
Second, install/run mysqltuner. Just do not blindly follow the recommendations, read first about the impact of the variables proposed for change.
Thirdly, disable the ipmi service. I suspect you don't need it.

Answer 2

Well, judging by top, PHP eats most of it, show the output of netstat -antp or even
There is a suspicion of a small DoS, well, either the PHP scripts are not configured optimally and eat a lot of resources on their own, and you still haven’t attached the ab output.

Answer 3

It can be seen that everything is eating php for you, look at the Apache or nginx logs and it will show which pages load the server.
and you posted the top while testing the ab?