How to debug stickiness in nginx+php-fpm?

Question

PHP

There is a fairly loaded web server (about 200k uniques per day, several online stores on the same platform) with a bunch specified in the subject.
It does not carry any other functions except for the web server, statics and the database are on separate servers. It copes with its load quite well, it uses 10-12GB of RAM out of 16 during peak hours, both processors cope. But about a month ago, a disaster overtook us: at certain, unrelated intervals, the number of php processes in a couple of minutes reaches a thousand, the memory ends and the server glues flippers. Only killing all hanging processes helps, after that the server continues to work in normal mode.
Since all this has been going on for a month (and the server crashes several times a day), we systematically tried:
- optimized mysql queries
- replaced nginx + php-fpm on the apache server - removed the
database server to another machine to free up memory - increased the allowed number of processes by 10 times, now at the time of the mysql crash, the processlist is almost empty, no requests longer than 1-2 seconds hangs
— logged from php all outgoing requests for external services that are made from php and can wait for them (one function is used for this in the CMS) — there is no special activity at the time of crashes
— nginx debug is enabled, no errors
— updated the software to the latest stable versions
- we monitored everything that is possible in cacti - except that the number of processes takes off and the memory ends, nothing special
I'm exhausted. The problem can be both with the server and with scripts. Maybe PHP scripts are looping somewhere? What do you advise how to debug the problem?

Answer 1

Have you looked at the logs, what is happening at this moment? Someone simply runs
ab -n 10000 -c 2000 host.com/
Whatever memory runs out, change in /etc/php5/fpm/pool.d/www.conf pm = dynamic to pm = static and pm.max_children = 30
Do the spawned fpm workers have cpu activity at the time of the hang ?

Answer 2

So that PHP processes under load do not multiply like mushrooms, you need to limit their number. To whom the process is not enough now - they are stupidly standing in the nginx queue.
Otherwise, your server can be put stupidly "hold F5" + a more or less decent channel.
If you do not use a caching mechanism and do not have restrictive iptables, try holding F5 on the main page in the browser and look at htop on the server. You will learn a lot of interesting things :)
Leaving Apache for Nginx is not yet a solution to the problem - you just reduced the amount of memory consumed per process, and not the amount of total memory.
PS The server dies when the RAM for the processes runs out, and the swap starts to saw - then "water enters the hold faster than you have time to scoop it out."

Answer 3

You would have to twist your scripts under some kind of profiler. xdebug.org/docs/profiler - something like this, or something else.