How to configure centos 7 for a large number of tcp connections?

Question

linux

Actually, tell me how to optimize the installed centos 7 for a large number of tcp connections.
Now, when there are more than 1500 connections, some ping requests do not receive a response or are received with a long delay.
Dropping ssh connection sometimes.
Doesn't stick to iron. The channel also does not rest.
There are enough resources to handle many times more connections...
dmesg:

[3119162.879543] nf_conntrack: table full, dropping packet
[3119162.879590] nf_conntrack: table full, dropping packet
[3119162.879632] nf_conntrack: table full, dropping packet
[3119162.879784] nf_conntrack: table full, dropping packet
[3119167.872258] net_ratelimit: 11305 callbacks suppressed
[3119167.872264] nf_conntrack: table full, dropping packet
[3119167.872412] nf_conntrack: table full, dropping packet
[3119167.872519] nf_conntrack: table full, dropping packet
[3119167.872673] nf_conntrack: table full, dropping packet
[3119167.873027] nf_conntrack: table full, dropping packet
[3119167.873032] nf_conntrack: table full, dropping packet
[3119167.873214] nf_conntrack: table full, dropping packet
[3119167.873274] nf_conntrack: table full, dropping packet
[3119167.873465] nf_conntrack: table full, dropping packet
[3119167.873489] nf_conntrack: table full, dropping packet
[3119172.865080] net_ratelimit: 11237 callbacks suppressed
[3119172.865085] nf_conntrack: table full, dropping packet
[3119172.865139] nf_conntrack: table full, dropping packet
[3119172.865160] nf_conntrack: table full, dropping packet
[3119172.865414] nf_conntrack: table full, dropping packet
[3119172.865539] nf_conntrack: table full, dropping packet
[3119172.865890] nf_conntrack: table full, dropping packet
[3119172.866458] nf_conntrack: table full, dropping packet
[3119172.867984] nf_conntrack: table full, dropping packet
[3119172.868072] nf_conntrack: table full, dropping packet
[3119172.868115] nf_conntrack: table full, dropping packet
[3119177.858188] net_ratelimit: 11322 callbacks suppressed
[3119177.858193] nf_conntrack: table full, dropping packet
[3119177.859415] nf_conntrack: table full, dropping packet
[3119177.859420] nf_conntrack: table full, dropping packet
[3119177.859593] nf_conntrack: table full, dropping packet
[3119177.860184] nf_conntrack: table full, dropping packet
[3119177.860235] nf_conntrack: table full, dropping packet
[3119177.860498] nf_conntrack: table full, dropping packet
[3119177.860578] nf_conntrack: table full, dropping packet
[3119177.860586] nf_conntrack: table full, dropping packet
[3119177.860598] nf_conntrack: table full, dropping packet
[3119182.853116] net_ratelimit: 11196 callbacks suppressed
[3119182.853121] nf_conntrack: table full, dropping packet
[3119182.854188] nf_conntrack: table full, dropping packet
[3119182.854257] nf_conntrack: table full, dropping packet
[3119182.855281] nf_conntrack: table full, dropping packet
[3119182.855539] nf_conntrack: table full, dropping packet
[3119182.855790] nf_conntrack: table full, dropping packet
[3119182.856007] nf_conntrack: table full, dropping packet
[3119182.856199] nf_conntrack: table full, dropping packet
[3119182.856267] nf_conntrack: table full, dropping packet
[3119182.856463] nf_conntrack: table full, dropping packet

Google advises to increase the values ​​of the variables below. As a result of attempts to achieve the absence of drops, I set a very huge value. As a result, the server began to accept about 500 more connections, but the drops remained.

# sysctl net.nf_conntrack_max=10000000
net.nf_conntrack_max = 10000000
# sysctl net.netfilter.nf_conntrack_max=10000000
net.netfilter.nf_conntrack_max = 10000000

The value of net.netfilter.nf_conntrack_count is kept at approximately the same level:

# sysctl net.netfilter.nf_conntrack_count
net.netfilter.nf_conntrack_count = 101061

At the end of the dmesg output there is such a thing nf_conntrack version 0.5.0 (16384 buckets, 65536 max) BUT I DID CHANGE THE VALUE!!!

[3122746.117846] nf_conntrack: table full, dropping packet
[3122746.118746] nf_conntrack: table full, dropping packet
[3122746.119005] nf_conntrack: table full, dropping packet
[3122746.120058] nf_conntrack: table full, dropping packet
[3122746.120358] nf_conntrack: table full, dropping packet
[3124678.296758] Ebtables v2.0 unregistered
[3124732.966132] ip_tables: (C) 2000-2006 Netfilter Core Team
[3124732.981407] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[3124732.987953] ip6_tables: (C) 2000-2006 Netfilter Core Team
[3124733.005746] Ebtables v2.0 registered

Rebooted the server. It seems that while the packages are not dropped.
Or just at night there are fewer connections, so there are no drops.

Answer 1

we look at the logs and dmesg
1500 connections about nothing I have the seventh one that keeps it on the default settings without any problems.

Answer 2

cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

##############
# nat tuning #
##############

# Увеличиваем количество conntrack соединений, необходимо менять значение
# по умолчанию только на высоконагруженных серверах.
# ядро 2.6 лучше воспринимают размеры таблиц равные степени 2, так что можно установить
# CONNTRACK_MAX и HASHSIZE равные 1048576 (2^20) etc...
# net.ipv4.netfilter.ip_conntrack_max = 4194304
# net.nf_conntrack_max = 4194304
net.ipv4.netfilter.ip_conntrack_max = 9437184
net.nf_conntrack_max = 9437184


# increase TCP max buffer size setable using setsockopt()
# изменение размеров буферов для приема и отправки данных через сокеты, tcp-memory
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths
# net.ipv4.tcp_rmem = 4096 87380 16777216
# net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_rmem = 4096 16777216 16777216
net.ipv4.tcp_wmem = 4096 16777216 16777216

# Максимальное число сокетов в состоянии TIME-WAIT одновременно. При превышении этого порога лишний сокет разрушается 
# и пишется сообщение в системный журнал. Цель этой переменной – предотвращение простейших разновидностей DoS-атак
# Значение по-умолчанию –     180000
net.ipv4.tcp_max_tw_buckets = 1800000

# Переменная задает максимальное число осиротевших (не связанных ни с
# одним процессом) сокетов. Если это число будет превышено, то такие
# соединения разрываются, а в системный журнал пишется предупреждение. Это
# ограничение существует исключительно ради предотвращения простейших
# разновидностей DoS-атак. 
net.ipv4.tcp_max_orphans = 262144
# net.ipv4.tcp_max_orphans = 65536

# net.ipv4.tcp_max_syn_backlog = 262144

# When in a non-Napi (or Interrupt) mode, this counter indicates that the stack is dropping packets.
# net.core.netdev_max_backlog = 10000
net.core.netdev_max_backlog = 30000

# net.core.somaxconn = 262144

# These ensure that TIME_WAIT ports either get reused or closed fast.
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

# защита от syn-флуда
net.ipv4.tcp_syncookies=1

# запрет приёма ICMP-редиректов
# net.ipv4.conf.all.accept_redirects = 0
# net.ipv4.conf.default.accept_redirects = 0

# игнорируем широковещательные ICMP-запросы
net.ipv4.icmp_echo_ignore_broadcasts=1

# игнорируем пакеты, в которых указан путь до источника
net.ipv4.conf.all.accept_source_route=0

# Укажем диапазон портов которые разрешено использовать в качестве
# локальных. По умолчанию этот диапазон достаточно мал, и при высокой
# нагрузке вам их может просто не хватить
# net.ipv4.ip_local_port_range = 16384 61000
net.ipv4.ip_local_port_range = 1024 65535

# Уменьшим время которое используется для сообщений
# о поддержке keep alive соединений
net.ipv4.tcp_keepalive_time = 7200

# Уменьшим время до закрытия TCP соединения, данный параметр стоит менять
# только на высоко нагруженных серверах.
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 7200

# Другие таймауты:
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 60
net.ipv4.netfilter.ip_conntrack_udp_timeout = 60
net.ipv4.netfilter.ip_conntrack_generic_timeout = 3600

# В случае kernel panic reboot через 10 секунд (panic может возникнуть при высокой нагрузке и irqbalance, например)
kernel.panic = 10

# чтобы не переполнялась арп-таблица (пишет в логах kernel: ipv4: Neighbour table overflow)
net.ipv4.neigh.default.gc_thresh1 = 1024
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096

Answer 3

If the problem is with conntrack - there are two solutions.
1. Decide whether you need a counter on the server at all? Is there a stateful firewall? Can rewrite to stateless? And unload the module itself.
2. If necessary, increase the size of the net.nf_conntrack_max table

Answer 4

the hash table, which stores lists of conntrack records, should also be increased.
echo 65536 > /sys/module/nf_conntrack/parameters/hashsize