How can I track where the bottleneck is on the server?

Question

PHP

Hello. I collected many different servers, and never had problems before. But now some kind of nonsense is happening ..
There is a server, nginx + php + mysql
At some random, but regular time (not so that I accidentally caught a bug, that's all. The bug is caught constantly, without any dependence at first glance) server on constipation can think for 8-15 seconds.

There are many different websites/web applications on the server. This is especially noticeable in web applications based on the REST architecture. Here, directly from the Google Chrome DevTools console, I "spam" Ajax with requests to the server. They all take ~200ms, but all of a sudden 5-6 requests (I'm spamming 1 request per second) are executed for 10 seconds, and then everything is fine.

I suspected that there might be some problems with MySQL, but on a pure yii2 web application without using a database, exactly the same thing happens.

If you ping the server cyclically - ping 66-67ms, there is no loss. WinMTR



The core24 node was very embarrassing, but their support claims that there are no problems with this, and the problems are on my side ..

I have been trying to solve the problem all day, but I have no idea which side to approach.
Versions:

Linux changeme 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
PHP 7.4.6 (cli) (built: May 14 2020 10:02:44 ) ( NTS )
nginx version: nginx/1.17.1
Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic-log - mariadb.org binary distribution

Answer 1

I would start by enabling slow_query_log in MySQL .
If you think that it’s not the database, then you need to dig deeper into server monitoring (cpu, inodes)
Also, sessions in files can cause such a problem, and the slowing request is the one on which the launch falls GC.
Well, be sure to check the php-fpm logs to see if there are any messages that there are not enough workers, since depending on the fpm settings, it can be blunt at the moment when new workers are created with a dynamic pool or blunt because the limit of workers ends.
Browser+session request spam can also be the cause of the problem due to session locks .

Answer 2

In addition to regular checks of the entire network and logs, I recommend adding Server-Timing profiling headers . Then it will be possible to see directly in the browser (in the Network / Network tab) at what stages the dullness occurs, it will look like this
:.
I don't know if yii can do Server-Timing...