No response from c nginx server to some syn?

Question

linux

Good afternoon.
Faced with the following situation.
There is a server with Debian Squeeze, it has nginx 0.7.67 installed (from the repository).
In principle, everything works without problems, but we noticed that sometimes it is not possible to connect to the port. SYN comes, no answer.
I tried to enable syncookies, it helped a little (errors appeared in the logs about “possible SYN flooding on port 80. Sending cookies.”). But still there were problems with the connection. There is no synflood, it's just a lot of legal traffic.
After that, I increased the following parameters:

net.core.somaxconn = 128000<br>
net.core.netdev_max_backlog = 10000<br>
net.ipv4.tcp_max_syn_backlog = 128000<br>

'
Njinx increased the listen backlog to 65536.

The "possible SYN flooding" messages are gone, but it still won't connect to the server intermittently. I checked with a simple script that raises 20 threads and opens and closes sockets to the 80th port in each thread.
Out of 1000 attempts to open a socket, about 30-50 fall off by timeout (2 seconds), while the rest connect almost instantly.

With all this, dmesg is empty, and nginx's error.log is also empty.

Njinx config snippet:

user                    www-data;<br>
worker_processes        2;<br>
worker_rlimit_nofile    65535;<br>
error_log               /var/log/nginx/error.log;<br>
pid                     /var/run/nginx.pid;<br>
<br>
events {<br>
        worker_connections              65535;<br>
        use                             epoll;<br>
}<br>
server {<br>
    listen 80 default backlog=65536;<br>
.....<br>
}<br>

On the server 2 cores, LoadAverage keeps less than one.

In netstat'e something like this:

# netstat -ant | grep tcp | tr -s ' ' ' ' | awk '{print $6}' | sort | uniq -c<br>
     22 CLOSING<br>
   3729 ESTABLISHED<br>
    815 FIN_WAIT1<br>
   3807 FIN_WAIT2<br>
    138 LAST_ACK<br>
      5 LISTEN<br>
    167 SYN_RECV<br>
     37 SYN_SENT<br>
   1104 TIME_WAIT<br>

In stub_status of nginx:

Active connections: 5985 <br>
server accepts handled requests<br>
 35200 35200 34437 <br>
Reading: 341 Writing: 223 Waiting: 5421 <br>

Just in case, SYNs definitely reach the server, took tcpdump, they are visible in it.

Tell me, please, what to watch, where to dig, who came across?

Answer 1

As a result, large backlogs and disabled syncs helped:

net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_syncookies = 0
net.core.somaxconn = 262144

nginx.conf:

user www-data;
worker_processes 4;
worker_rlimit_nofile 100000;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
        worker_connections 65535;
        use epoll;
}
...
server {
           listen 80 default backlog=65000;
...
}

Answer 2

You have only two worker_processes
and worker_connections 65535;
if you have a lot of legal traffic, then maybe it just doesn’t fit?
Increase worker_processes

Answer 3

This can't be said from the logs, but increasing nf_conntrack might help?
This is a little from another opera, but suddenly.