Low build speed 6 soft raid in linux

Question

linux

Centos 5.5
22 hard drives, sata2 seagate
I have not collected raids in Linux for a long time, but if my memory serves me right, the speed was usually 50-90 megabytes per second.
Now 8-20 megabytes per second (on two different machines).

I create a raid using the command
mdadm --create --verbose /dev/md0 --level=6 --raid-devices=22 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev /sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1 Build

speed
cat /proc/mdstat
Personalities: [raid6] [raid5] [raid4]
md0: active raid6 sdw1[21] sdv1[20] sdu1[19] sdt1[18] sds1[17] sdr1[16] sdq1[15] sdp1[14] sdo1[13] sdn1[12] sdm1[11] sdl1[ 10] sdk1[9] sdj1[8] sdi1[7] sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
39070288640 blocks level 6, 64k chunk, algorithm 2 [22/22] [UUUUUUUUUUUUUUUUUUUUUU]
[>..................] resync = 0.3% (5973976/1953514432) finish=3862.7min speed=8402K/ sec

unused devices: CPU not loaded
top — 20:00:42 up 5:51, 0 users, load average: 1.35, 1.16, 0.77
Tasks: 106 total, 1 running, 105 sleeping, 0 stopped, 0 zombie
Cpu0: 0.0% us, 15.7%sy, 0.0%ni, 73.2%id, 0.0%wa, 0.0%hi, 11.0%si, 0.0%st
Cpu1: 0.0%us, 3.3%sy, 0.0%ni, 96.7%id, 0.0%wa , 0.0%hi, 0.0%si, 0.0%st
Cpu2: 0.0%us, 4.3%sy, 0.0%ni, 95.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3: 0.0%us, 1.0%sy, 0.0%ni, 99.0%id , 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2050688k total, 283396k used, 1767292k free, 7456k
buffers

%CPU %MEM TIME+ COMMAND
5587 root 10 -5 0 0 0 S 19.0 0.0 2:10.15 md0_raid5
5588 root 10 -5 0 0 0 D 7.0 0.0 0:43.88 md0_resync
5633 root 15 0 12740 1084 804 R.0.07
1 root 15 0 10348 688 572 S 0.0 0.0 0:00.84 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/
1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/2
9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/
2
11 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/3
12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/
3
14 root 10 -5 0 0 0 S 0.0 0.0 0:00.02 events/0
15 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/1

Any ideas?

Answer 1

> I have not collected raids in Linux for a long time, but if my memory serves me right, the speed was usually 50-90 megabytes per second.
for RAID what level? The 6th will eat more resources anyway.
In general, for 22 disks it is better to have a hardware raid with BBU

Answer 2

have you tried?
cat /proc/sys/dev/raid/speed_limit_m*
200000
1000
echo 10000 >/proc/sys/dev/raid/speed_limit_min

Answer 3

rather cruel configuration for software.
I would suggest inserting several hardware controllers, collecting raids on them, which can be combined into a stripe/mirror by soft-trade as desired.
and so I'm afraid you can run into a corny bus / irq, etc.
what does iostat/vmstat show?
read articles about tuning? cache size for raid stripes, etc. optimized?

Answer 4

By the way, what wheels? not with 4kb sectors? maybe corny at the junction of sectors hit?

Answer 5

-c, --chunk=
Specify chunk size of kibibytes. The default is 64.
I would play with it

Answer 6

Sorry for offtopic.
Is it true that this soft raid works stably only on Cent? And then they convince me that everything will be stable on Ubuntu.

Answer 7

A post that has received an invite from the sandbox can be published in principle in any (at least a specialized blog) even with zero karma. Write to [email protected]

Answer 8

Here they say that they have already fixed it.