Different SSD load for the same IOPS, why is that?

Question

linux

Guys, has anyone encountered the problem of different SSD loads in software raid 10/5? There are new samsung evo 860 16x1TB on the server, the load of each disk is growing in jumps. Although IOPS does not change.. OS - debian 9.X 64bit
Tried to reboot the server - it helped, in a day - two started again. On raid0, I haven't noticed this yet.

IOPS

This is how it starts to manifest itself (another server)

And this is how it should ideally be without problems

Any suggestions?
It is also very interesting why in software raid0 / 10 one part of the SSD works at one load level, the second part at another. Okay, on raid10 you can understand, there are two mirrors, but why is it like this on raid0? (we have 7 disks, 5 with one iops and load, 2 with another), just like on my graphs there are two lines. I remember before (on Debian 6-7) this was not the case.
Here is the output of mdadm
[email protected]:~# mdadm --detail /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Wed Sep 26 19:09:07 2018
Raid Level : raid10
Array Size : 7807188992 (7445.52 GiB 7994.56 GB )
Used Dev Size : 975898624 (930.69 GiB 999.32 GB)
Raid Devices : 16
Total Devices : 16
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Feb 11 04:14:00 2019
State : clean
Active Devices : 16
Working Devices : 16
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Name : AMS:2 ( local to host AMS)
UUID : e4df4aeb:61e8d8f3:e1ad893e:bae90821
Events : 166074
Number Major Minor RaidDevice State
0 8 3 0 active sync set-A /dev/sda3
1 8 35 1 active sync set-B /dev/sdc3
2 8 99 2 active sync set-A /dev/sdg3
3 8 51 3 active sync set-B /dev/sdd3
4 8 67 4 active sync set-A /dev/sde3
5 8 83 5 active sync set-B /dev/sdf3
6 8 115 6 active sync set-A /dev /sdh3
7 8 131 7 active sync set-B /dev/sdi3
8 8 19 8 active sync set-A /dev/sdb3
9 8 147 9 active sync set-B /dev/sdj3
10 8 163 10 active sync set-A /dev/sdk3
11 8 179 11 active sync set-B /dev/sdl3
12 8 195 12 active sync set-A /dev/sdm3
13 8 211 13 active sync set-B /dev/sdn3
14 8 227 14 active sync set-A /dev/sdo3
15 8 243 15 active sync set-B /dev/sdp3

Answer 1

It's not entirely clear what exactly is wrong.
For a correct understanding, it is important to define the terminology - for example, what do you understand by load? Well, it is just as important what exactly and how the monitoring utility measures, the charts of which you are looking at. It looks like it just shows how fast the disk is processing requests.

the load on each disk jumps up. Although IOPS does not change..

Where does the conclusion that the load is growing come from? Where is the load information? As far as I understand, the first graph shows the busyness of the disk (busy). There is not a word about the load, maybe there is no load at all.

It is also very interesting why in software raid0 / 10 one part of the SSD works at the same load level

Again, a rather strange statement. Where does the different level of load come from in the raid? Well, it can't be that more was recorded on one disc than on another. Obviously the load is the same.
The only difference is disk usage. One disk is faster to handle the load, and the other is slower, that's all.

Answer 2

For example, here: https://habr.com/en/company/webzilla/blog/227927/

During the test, something strange was observed in blktop: some of the disks were loaded into the ceiling, some were almost idle. Moreover, those who show low performance are loaded into the ceiling, and “fast” drives are idle. Moreover, disks sometimes change places - that is, a disk that was previously loaded at 100% suddenly shows a higher speed and less load, and vice versa, a disk that was loaded at 50% suddenly turns out to be loaded at 100% and at the same time shows a lower speed. Why?
And then it hit me.