Nginx drops gRPC connection when testing via Locust: Received http2 header with status: 504. How to write a failover service on grpc?

Question

Python

How to make it right so that the connection does not break and processes all requests? Or aborted when I think the request is complete, because nginx is holding the connection between containers for a while, if I understood correctly.

I tried to do something similar to the answer . The error remains the same.

#nginx version: nginx/1.19.10
http {
    fastcgi_read_timeout 300;
    proxy_read_timeout 300;
    uwsgi_read_timeout 300;
    grpc_read_timeout 300;
}

server {
    listen 8558 http2;
    charset utf-8;
    location / {
        grpc_pass grpc://stt:8119;
    }
}

Full error description from grpc to python:

<_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "Received http2 header with status: 504" debug_error_string = "{"created":"@1641818642.427412518","description":"Received http2 :status header with non- 200 OK status","file":"src/core/ext/filters/http/client/http_client_filter.cc","file_line":130,"grpc_message":"Received http2 header with status: 504","grpc_status" :14,"value":"504"}" >

The error appears after several requests (number of users and spawn rate 200). At the same time, after each request, the average processing speed increases. Can you suggest why this might be?


UPD
Removed nginx, tried to make requests directly to grpc - no error. But if request balancing is required, how then to be?