Problems with TCP connections in Linux. How to win?

Question

linux

I can not catch in any way - what network connections rest against.
The essence of the server:
The server collects statistics from client applications. The server is running NGINX with a simple LUA script that sends the data received from the client (very small JSON) to RabbitMQ.
The load is generally not large - 30,000 RPM (500RPS).
The following problem is constantly observed: Those clients who currently send data to the server do not experience problems. New clients, when trying to establish an HTTP connection, wait 20-30 seconds each with a periodic full timeout.
If instead of a script in LUA, for each request, just give a JSON file, then the picture does not change in any way.
Those. looks like a problem with setting up a network in Linux at the stage of establishing a TCP connection with a large number of such
hardware

Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz Hexa-core
64GB Ram
LAN 1Gbit

CPU loaded at 20%, RAM at 20%.
IOTop doesn't show any meaningful disk accesses.
The NGINX configuration is something like this:

user www-data;
worker_processes auto;
worker_rlimit_nofile 262143;

events {
  worker_connections 262143;
  multi_accept on;
  use epoll;
}

http {
  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 10;
}

After re-reading a lot of optimizations for HighLoad in sysctl, I wrote the following:

fs.file-max=100000
vm.swappiness=10
net.netfilter.nf_conntrack_max=1548576
net.ipv4.ip_local_port_range=10000 65000
net.ipv4.tcp_tw_reuse=1
net.core.somaxconn=15600
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_tw_recycle=1
net.core.rmem_default=31457280
net.core.rmem_max=12582912
net.core.wmem_default=31457280
net.core.wmem_max=12582912
net.core.netdev_max_backlog=65536
net.core.optmem_max=25165824
net.ipv4.tcp_rmem=8192 87380 16777216
net.ipv4.udp_rmem_min=16384
net.ipv4.tcp_wmem=8192 65536 16777216

ifconfig eth0 txqueuelen = 10000 is set.
Obviously somewhere there is some kind of bottlneck, but I don’t understand where. There is no special understanding of the TCP stack in Linux, so help is needed.
For example, here is the output, which, unfortunately, does not tell me anything personally, but TimeWait is too much and what Closed is.

[email protected] / # ss -s
Total: 296670 (kernel 296893)
TCP:   310953 (estab 2959, closed 307810, orphaned 149, synrecv 0, timewait 14403/0), ports 0

Transport Total     IP        IPv6
*         296893    -         -
RAW       0         0         0
UDP       18        12        6
TCP       3143      3140      3
INET      3161      3152      9
FRAG      0         0         0

How to understand - where the "network" gets stuck? What resources \values ​​rests on? With the help of what in general can this be debugged and what settings should I try again?

Answer 1

Is there a firewall on the server?
Is there nating on the server? You may have run into these settings, by default iptables has very limited settings for the number of Nat sessions

Answer 2

spin the backlog.

Answer 3

I will re-read a lot of optimizations for HighLoad

HighLoad is 1000 clusters of 50 racks, 10 blades connected via Infiniband, working 24/7/365, with 5 minutes downtime per year.
---
Do you need IPv6? Not! Cut down!
Ping to the router from outside, from the server?
RSS? delay?
ifconfig eth0 txqueuelen = 10000 is too much. This is for 10Gb network cards and beyond.
routing table?
tc?
...
Further paid.

Answer 4

# sysctl net.ipv4.tcp_syncookies
# sysctl net.ipv4.tcp_max_syn_backlog
# cat /proc/ngix_master_PID/limits
# cat /proc/ngix_worker_PID/limits

Answer 5

Look in top for the status of nginx processes

Answer 6

enter a subscription, interrupt without a subscription with advertising inserts :)

Answer 7

you can hang some kind of advertisement, maybe even teasers

Answer 8

Sell ​​broadcasts, no?
From where, by the way, do you get traffic with broadcasts?