Why does script performance drop?

Question

linux

Hello.
Using a python script (using asyncio and aiohttp), I parse sites to collect statistics. After starting the parsing, about 1k-1.5k sites are parsed very quickly, and then the brakes begin. Moreover, the server resources are almost not used, the network is also not loaded. The loop.create_connection method slows down. If you load any site in parallel with parsing, it also becomes stupid. I assume that the problem is with the OS (debian 7.8) but I don’t know in which direction to dig.

Answer 1

maybe running out of buffers or something else - you need to look at dmesg
also try adding to /etc/sysctl.conf
net.core.wmem_default = 2097152
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.core.somaxconn = 16384
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 1440000 net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.sync
.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_abort_on_overflow = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.netfilter.ip_conntrack_max = 262144 net.ipv4.netconf.all.accept_redirects
= 0
ipv4.conf.all.secure_redirects = 0
then
# sysctl -p
to roll back changes - remove options from /etc/sysctl.conf
and reboot.

Answer 2

You, apparently, run on your machine, maybe your provider limits you, seeing such a number of connections? Try the same script on the server/vps.