Why doesn't nginx redistribute the load when one of the backends goes down?

Question

Nginx

Good afternoon, I have a question about setting up balancing on nginx. I needed to make an nginx balancer that distributes the load between backends (nginx + rails). I ran into a problem that when one of the backends is disabled, nginx does not redistribute the load, i.e. the load on the remaining backends remains the same, but the RPS on the balancer drops. The balancer tries every time to send requests to the downed server. I can't figure out what the problem is.
Here is the balancer config:

upstream backend {
  server 10.132.0.2 max_fails=1 fail_timeout=30s;
  server 10.142.0.4 max_fails=1 fail_timeout=30s;
  server 10.142.0.5 max_fails=1 fail_timeout=30s;
  server 10.142.0.6 max_fails=1 fail_timeout=30s;
}

server {
  listen 80;
  server_name 0.0.0.0;
  root /app/public;
  allow all;

  location /nginx_status {
    stub_status;
  }

  location /assets/ {
    error_page 404 = @store;
    expires max;
  }

  location @store {
    proxy_store on;
    proxy_store_access user:rw group:rw all:r;
    proxy_pass http://backend;
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto http;
    proxy_set_header Host $http_host;
  }

  location / {
    proxy_pass http://backend;
    proxy_next_upstream error timeout invalid_header http_502;
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto http;
    proxy_set_header Host $http_host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_redirect off;
  }
}

The RPS graph on the balancer (blue) and backends (red), the yellow backend is turned off (on it RPS immediately drops to 0, there is an error on the graph).

Answer 1

As I understand it, it was necessary to fix the configs on the backends: increase the number of workers and set multi_accept on;. Now everything is working fine.

Answer 2

use Traefik
, it is designed specifically to quickly catch the appearance / disappearance of backends and quickly switch them

Answer 3

Look at the logs. Where what happens with the requests that you lost.
Alternatively, it is possible that your backend does not process requests, but accepts new ones. Or something similar. Then you need to tune the timeouts on the proxies.
Well, have you lost any requests from the front? To answer you need to know their fate.