Why does the Yandex Market bot not have time to receive a response from nginx in 150 seconds, even the 304 Not Modified code?

Question

Nginx

Once in about 20 minutes, a Yandex bot visits the site to download fresh price lists or make sure that they have not changed. As a rule, the operation completes successfully, but once a day (time changes, more often at night) it fails to receive a response from the server in the allotted 150 seconds and it disconnects with a 408 Request Timeout error. Moreover, as a rule, such unsuccessful attempts happen two in a row.

In the nginx logs, at the same time, at the time corresponding to errors in the Market admin panel, the response is generated normally, it takes a maximum of 30 seconds if the archive with price lists is not cached (tried to disable caching for the directory with price lists, it did not help), if the response code is 304 -- server responds instantly.

At the same time, the fact that there really is some kind of problem is signaled by the fact that the time of the failed load in the admin panel is a minute or two earlier than the nginx response time in the logs, i.e. Apparently something seriously delays the answer.

The load on the server in those moments when this happens is normal (according to ISPManager), and in general during the day is quite low, there are no interruptions in the operation of the site itself either.

How can it be debugged, where to dig?

nginx settings that might be relevant:

proxy_read_timeout 1600;
proxy_send_timeout 1600;
proxy_connect_timeout 1600;
keepalive_timeout  150;
client_body_timeout 150;
client_header_timeout 150;

Also tried the following, didn't help:

send_timeout 1600;
fastcgi_read_timeout 1600;
proxy_max_temp_file_size 2048m;
proxy_buffers 16 16k;
proxy_buffer_size 16k;

Answer 1

Does this file form the backend? The fact that this happens at night suggests the idea of ​​backups or other types of service work. Do you have VDS/VPS?
They suggest adding the following log format to the main config:

log_format  with_time '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" rqt=$request_time urst=$upstream_response_time';

The server config specifies:

access_log  /путь_до_файла_лога/access.log with_time;

This will allow you to see the time the client took the response (large value - a problem with the client / network to the client) and the response time of the back (large value - problems on the back). Depending on the results, you need to think further where to go.

Answer 2

dig into the net

Answer 3

How can it be debugged, where to dig?

At the same time, the fact that there really is some kind of problem is signaled by the fact that the time of the failed load in the admin panel is a minute or two earlier than the nginx response time in the logs, i.e. Apparently something seriously delays the answer.

- And what is the response time in the rest of the hours?
- I would simulate the request myself, and at the same time watch nginx and the database