SSD latency on an array under ESXi takes off, where to dig?

Question

Hard disks

The essence of the problem: if you put a load on the disk subsystem, the latency indicator flies up and everything stops working.
In the picture - load testing 1C

What was done:
Replaced SSD drives with another model
Disabled caching, buffering and other raid chips Disabled
the usual AHCI driver
Disks are less than half busy. But all the space is marked for VMFS6

Configuration:
OS: ESXi 6.7
CPU: i7-3770
Array: LSI Megaraid 9260 for 4 slots (512 RAM)
2 enterprise HDD slots in the mirror
2 SSD slots in the mirror (there were intel, now some micron 1100 )
Server in Hetzner

Symptoms:
The cache is enabled for everything and everyone, but when you turn it off, the situation only gets worse.
Linear copying of a 50GB file between datastores can be indicative. After 5 minutes of copying at a speed of 150MB, the speed drops sharply, and the delay increases sharply. In this case, in general, the entire system ceases to respond normally.
The same thing happens with, for example, a 1C load test, if you run 100 users, everything freezes abruptly and the latency on the SSD array takes off.
Copying between datastores from HDD to SSD of one file 50 GB

Here are the indicators of previous Intel SSDs with array cache disabled

:
. I/O latency increased from average value of 4580 microseconds to 210110 microseconds.

At the same time, the HDD array works exactly as it should (small and large files to display the operation of the raid cache)
On the neighboring server, the same RAID, ESXi version 6.5 in 2 SSD datacenter edition, which should not affect, but there is no more than 10ms at all jumping latency.

upd. Screenshot of esxtop when copying between datastores (HDD->SSD). Latency 20-50ms fluctuates

What to do, where to look?

Answer 1

Try soft raid, it's much more fun