Pochmeu gigabit network card does not pump gigabit at MTU=9000?

Question

CentOS

Good afternoon!
2 servers connected by a patchcord directly. Installed centos7
At MTU = 1500 iperf issues:
[SUM] 0.0-30.0 sec 3.29 GBytes 942 Mbits/sec
and at mtu 9000:
[SUM] 0.0-60.0 sec 3.90 GBytes 558 Mbits/sec
and if also through a switch: (jumbo frames 10000B are allowed)
[SUM] 0.0-72.6 sec 1.83 GBytes 216 Mbits/sec
So not only iperf, but also rsync / ssh, nc is indignant.
How to treat it?

Answer 1

network model? If your finger is in the sky, then start with:

ethtool -K em0 tso off ufo off gso off gro off lro off rx off
ethtool -K em1 tso off ufo off gso off gro off lro off rx off
ethtool -G em1 rx 4096 tx 4096
ethtool -G em0 rx 4096 tx 4096
ethtool -L em0 combined 4
ethtool -L em1 combined 4
/sbin/ifconfig em0 txqueuelen 10000
/sbin/ifconfig em1 txqueuelen 10000
echo 000000ff> /sys/class/net/em0/queues/rx-0/rps_cpus
echo 000000ff> /sys/class/net/em1/queues/rx-0/rps_cpus

echo 1 > /proc/irq/44/smp_affinity
echo 2 > /proc/irq/47/smp_affinity
echo 4 > /proc/irq/48/smp_affinity
echo 8 > /proc/irq/49/smp_affinity

echo 1 > /proc/irq/46/smp_affinity
echo 2 > /proc/irq/50/smp_affinity
echo 4 > /proc/irq/51/smp_affinity
echo 8 > /proc/irq/52/smp_affinity

And there are so many nuances from the network model to the version of the driver used, for specific firewood you can say more precisely, well, the switch should have adult jumbo support, and not just for show.

Answer 2

And it makes sense to bother if you already have a small mtu
With MTU = 1500 iperf gives:
[SUM] 0.0-30.0 sec 3.29 GBytes 942 Mbits/sec

Answer 3

Can you show a traffic dump? There is a suspicion that TCP for some reason does not expose an adequate segment size.

Answer 4

Manufacturer of a network card, switch, switch, etc. "guarantees" the specified bandwidth at standard values ​​for its equipment. In your case, it is most likely that your hardware does not cope with the tasks of "collecting / parsing" packages. In general, changing the MTU and using jumbo frames is a very capricious thing and you should definitely do some experiments, which you did.