How to find out which processes are wasting CPU system time?

Question

MySQL

Hello.
There is a virtual machine with CentOS 7 on board, 12 cores and 85G RAM.
The mysql server is running on this machine, the database weighs 300G.
We have a problem with the fact that the CPU System Time rises to 30% - 40% and the system starts to blunt.
Here is the output of top:
top - 12:10:29 up 4 days, 6:05, 5 users, load average: 5.20, 4.70, 5.16
Tasks: 332 total, 1 running, 331 sleeping, 0 stopped, 0 zombie
%Cpu(s): 36.7 us, 18.4 sy, 0.0 ni, 42.8 id, 1.8 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 90574896 total, 90266192 used, 308708 free, 306288 buffers
KiB Swap: 0 total, 0 used, 0 free. 33777148 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29422 mysql 20 0 58.544g 0.051t 7328 S 664.3 60.6 4698:12 mysqld
46008 root 20 0 123800 1884 1176 R 0.7 0.0 0: 00.04 top
74 root 20 0 0 0 0 S 0.3 0.0 2:25.17 rcu_sched
1374 root 20 0 0 0 0 S 0.3 0.0 18:13.28 jbd2/sdb1-8
8718 root 20 0 190040 4936 1808 S 0, 3 0.0 3:32.98 vmtoolsd
1 root 20 0 47592 3608 1948 S 0.0 0.0 0:04.15 systemd2
root 20 0 0 0 0 S 0.0 0.0 0:00.05 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:36.52 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u128:0
8 root rt 0 0 0 0 S 0.0 0.0 0:01.42 migration/0
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
11 root 20 0 0 0 0 S 0.0 0 ,0 0:00.00 rcuob/1
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3
14 root 20 0 0 0 0 S 0, 0 0.0 0:00.00 rcuob/4
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6
17 root 20 0 0 0 0 S 0, 0 0.0 0:00.00 rcuob/7
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/8
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/9
20 root 20 0 0 0 0 S 0, 0 0.0 0:00.00 rcuob/10
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/11
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/12
23 root 20 0 0 0 0 S 0, 0 0.0 0:00.00 rcuob/13
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/14
25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/15
26 root 20 0 0 0 0 S 0, 0 0.0 0:00.00 rcuob/16 The
question is how to debug this situation, by what means can we determine which system process starts to work so actively.
Thanks to all who responded! =)

Answer 1

Well, probably for a start it is worth launching mytop during such periods.
top shows that the main "swallow" is mysqld

Answer 2

system time can be easily eaten by the application, if you have nothing but a muscle there, then it is logical that this is a muscle
look at the show process sheet at the moment of blunts, for example,
look at the iotop

Answer 3

how to correctly diagnose by CPU, which means CPU user time