Packet loss, packet bursts?

Question

linux

I have a problem with the server, or rather, as far as I understand, with the network core.
For me, the speed of data transfer between processes, in the future between servers, is critical.
The problem lies in the delays in the transmission of packets between processes, for example, out of 100 packets, 90 will reach in 0 -1 ms, and 10 with a delay of up to 200ms, the packet size is very small. In the case of TCP, it can be assumed that the packet is dropped and when it is sent again, a queue appears, etc. and because of this, a sharp delay.
But with UDP, the situation is the same.
I tried to use various brokers such as zeromq, nanomsg the situation does not change, because of this I have to believe that the matter is in the configuration of the network core or hardware (which is less likely).
I checked on my computer, it's windows. No such problems were observed, message delivery time 0 -1.