How to determine the bottleneck on the server?

Question

PHP

Fellow experts. I ask for help to understand and understand where the bottleneck is on my server. There are several different servers. A php project is uploaded there: some kind of api. When 1 person works with api - everything is ok, when there are a lot of them - it immediately starts to lag and the client may not wait for a response.
Here is the configuration of one of the servers: this is a free amazon instance. OS - some version of Amazonian, probably based on ubuntu or debian. Processor with 1 core, 600 MB of RAM. I understand that these are show-offs, but servers with a more powerful configuration experience the same problems, although in my opinion they should work at higher loads. The server uses php 5.4, mysql 5, nginx. There is almost no work with the database, since every 2 hours all api data is cached using memcached and is always returned from the cache.
So I'm running a load test from the site loader.io which simulates 30 clients per minute, each client makes requests to 6 different APIs. Each endpoint generally gives cached data + some small actions in the database like update the access date. With this configuration and 30 clients per minute, the test crashes due to a large number of timeouts. I can't figure out what's wrong. CPU load - 100 almost always. Takes up almost everything - php. But why does it load the percent so much, if there is work for a couple of seconds ...
Can someone briefly describe the instructions for determining the bottleneck on the server: something in nginx (maybe the setting is incorrect), or I somehow configure PHP incorrectly, Or something is wrong with the database. In setting up, I'm a complete 0, I would like to hear the advice of people who have experience. Thank you!
P.S. here is one of the graphs, it shows how the server properly processes requests, but then abruptly starts to pedal and bends: upd: profiler results

Answer 1

Scatter timers around the code, then to taste - you can write them somewhere, you can just open the code in parallel with the test and see the results.
At least it will be clear where it slows down. Now explicit for the solution is clearly not enough.
Hypothetically (the version from the head), for example, there may be some kind of crooked cache that chokes, constantly regenerates - here is the download of 100%
upd , another hypothetical version - for example, you have some kind of heavy transactions that interfere with different script flows.

Answer 2

www.xdebug.org/docs/profiler - use only for tests, in production it should be disabled.
There are suspicions that you have some kind of trouble with resolving DNS servers with memcache and / or database. Can you check?

Answer 3

new relic