Php-fpm is CPU intensive. What could be the problem?

Question

PHP

Hello! Help deal with the problem.
Php fpm creates high LA + CPU load.
We have: API Web server ngnix+phpfpm (2000 processes) + redis + memcached.
The database is on another server, everything is ok with it.
Load 3500-6000 requests per second.
Architecture - the client sends a request, we search the database, access the memcache, access the radish for an array of exceptions, run the received data (300 rows from mysql) in a loop to create an array, pull the value we need from the array, write the result to the database, we write the result to the radish and display the result to the client.

Problem: at 3500-4300 requests per second, LA 28-30 + CPU load 30-40%. When increasing requests to 5000-6000 per second. LA begins to grow seriously and the processor consumption rises to 70-90 percent. Most likely a

piece of iron AMD EPYC 7502P + nvmessd drives

Maybe you need to change the processor in the ratio - fewer cores but more frequency?
Processor choice:
2 x Intel® Xeon® SP Gold (2pcs)
Intel® Xeon® W-2295
Intel® Xeon® W-2145
AMD EPYC 7502P
Intel® Core™ i9-12900K
AMD Ryzen™ 7 3700X

Thanks for your responses

Answer 1

1. Install prometheus + node_exporter. Metrics are very important, without them it's just pointing your finger at the sky
2. If the system is loaded by php code, then put xhprof and profile requests (for example, 1% of requests), then you can see which code gives the load on the processor
3. If the database is on that same server, install mysql_exporter as well