How to set up nginx file server correctly?

Question

Nginx

There is a Dedik with 32 GB of RAM and 1 GB of port, and the problem with it is that it is used as a file server via the http protocol, i.e. the user simply clicks on the server.com/file.exe link and downloads the file. But it hangs tightly when 20-30 people are downloading from it at the same time. I found the settings on the Internet, now it works with them. If without them, then generally a maximum of 10 simultaneous downloads. Settings:

sendfile                          on;
    tcp_nopush                      on;
    tcp_nodelay                     on;
    client_header_buffer_size       1k;
    client_body_buffer_size         2k;
    client_max_body_size            500m;
    large_client_header_buffers     4 8k;
    send_timeout                    10;
    keepalive_timeout               20 20;
    reset_timedout_connection       on;
    server_tokens                   off;
    server_name_in_redirect         off;
    server_names_hash_max_size      512;
    server_names_hash_bucket_size   512;
    open_file_cache max=20000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors off;

Please tell me what needs to be fixed, added to increase the number of connected users?

Answer 1

https://docs.nginx.com/nginx/admin-guide/web-serve...
There may be problems at the OS level, configure according to this guide

Answer 2

How are the resource indicators? Look at HDD usage first (I/o delay). If the httpS protocol (ssl) is used, then encryption can lie down cpu if there is some kind of atom. Look at the readings under load.

Answer 3

1. See how many CPU Cores. If you increase the number of worker_processes and worker_connections
As it turned out, with a large number of file downloads, performance increases if you increase worker_processes. Auto doesn't help much. But without fanaticism. For 32 cores I set 96
worker_processes.
2.
multi_accept on;
accept_mutex off;
use epoll;
3.
directio, depending on the size of the files, you need to select.
4.
thread_pool more than 128 I don't see the point in my case
5. Combo. If soft raid do nothing. If iron raid with ssd or hdd, or just ssd or just hdd change io scheduler from bfq to noop
With bfq, even a 50 TB iron raid with a bunch of disks dies. Io rises above 3 sec.
That's all for Ubuntu 18.04.
By fiddling with these settings, you can fully use a 10 Gbit / s server with a 28 TB file cache. And io when writing no more than 10 ms and 4000 active connections