DRBD - low write performance?

Question

linux

Hello.
I have the following config:
2 x IBM x3630, 14 2Tb SATA disks each, raid10 of 12 disks is assembled on each, 2 in hot-spare.
Gigabit link between servers.
DRBD over raid10 devices (md-raid).
Performance /dev/md127:
throughput ~ 900 MB/s
fio with the following config:
config fio

[readtest]
blocksize=4k
filename=/dev/md/raid10
rw=randread
direct=1
buffered=0
ioengine=libaio
iodepth=16

produces something like this:
Output fio

readtest: (groupid=0, jobs=1): err= 0: pid=5009
  read : io=38632KB, bw=3502.5KB/s, iops=875 , runt= 11030msec
    slat (usec): min=4 , max=135 , avg=15.63, stdev= 4.91
    clat (msec): min=1 , max=149 , avg=18.19, stdev=12.39
     lat (msec): min=2 , max=149 , avg=18.21, stdev=12.39
    bw (KB/s) : min=    0, max= 3736, per=61.09%, avg=2139.33, stdev=1733.77
  cpu          : usr=1.16%, sys=2.03%, ctx=9085, majf=0, minf=36
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=9658/0/0, short=0/0/0

     lat (msec): 2=0.01%, 4=0.38%, 10=21.22%, 20=49.09%, 50=26.52%
     lat (msec): 100=2.63%, 250=0.16%

875 iops at 18ms latency. I'm satisfied with these numbers, everything is fine.
fio for writing - similarly.
fio config

[writetest]
blocksize=4k
filename=/dev/md/raid10
rw=randwrite
direct=1
buffered=0
ioengine=libaio
iodepth=16

fio output

writetest: (groupid=0, jobs=1): err= 0: pid=5023
  write: io=169624KB, bw=3912.7KB/s, iops=978 , runt= 43353msec
    slat (usec): min=2 , max=20841 , avg=10.85, stdev=101.29
    clat (usec): min=15 , max=169027 , avg=16321.19, stdev=33566.14
     lat (usec): min=267 , max=169040 , avg=16332.46, stdev=33566.13
    bw (KB/s) : min= 2936, max= 7334, per=100.26%, avg=3922.26, stdev=526.96
  cpu          : usr=1.02%, sys=1.50%, ctx=40727, majf=0, minf=18
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/42406/0, short=0/0/0
     lat (usec): 20=0.01%, 250=0.01%, 500=33.40%, 750=9.47%, 1000=5.28%
     lat (msec): 2=18.85%, 4=13.12%, 10=1.09%, 20=0.87%, 50=1.70%
     lat (msec): 100=12.06%, 250=4.15%

978 iops with latency up to 17ms. It's great too.
Then the most interesting begins.
I create a drbd device with the following config:

resource r0 {
  device     /dev/drbd0;
  disk       /dev/md/raid10;
  meta-disk  internal;
  on storage00 {
    address 192.168.254.10:7789;
  }
  on storage01 {
    address 192.168.254.11:7789;
  }
  net {
    max-buffers 8000;
    max-epoch-size 8000;
  }
}

And I get something that cannot be processed by the mind, namely:
Throughput for reading almost does not degrade (which was to be expected, reading comes from a local node).
Write throughput is fixed at exactly 60 MB/s. Not enough, I expected 110 MB / s from gigabit, especially since the initial sync of the drbd device took place just at a speed of 110 MB / s.
Again fio with a StandAlone device. Read:
fio config

[readtest]
blocksize=4k
filename=/dev/drbd0
rw=randread
direct=1
buffered=0
ioengine=libaio
iodepth=16

fio output

readtest: (groupid=0, jobs=1): err= 0: pid=5214
  read : io=154380KB, bw=3500.5KB/s, iops=875 , runt= 44103msec
    slat (usec): min=5 , max=417 , avg=17.87, stdev= 5.28
    clat (msec): min=1 , max=209 , avg=18.25, stdev=12.51
     lat (msec): min=1 , max=209 , avg=18.27, stdev=12.51
    bw (KB/s) : min= 3048, max= 3840, per=100.16%, avg=3505.55, stdev=113.92
  cpu          : usr=1.02%, sys=2.17%, ctx=36213, majf=0, minf=37
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=38595/0/0, short=0/0/0

     lat (msec): 2=0.01%, 4=0.47%, 10=21.24%, 20=48.90%, 50=26.46%
     lat (msec): 100=2.79%, 250=0.13%

Entry:
fio config

[readtest]
blocksize=4k
filename=/dev/drbd0
rw=randread
direct=1
buffered=0
ioengine=libaio
iodepth=16

fio output

writetest: (groupid=0, jobs=1): err= 0: pid=5229
  write: io=2396.0KB, bw=109341 B/s, iops=26 , runt= 22439msec
    slat (msec): min=8 , max=67 , avg=37.40, stdev= 9.43
    clat (usec): min=440 , max=741029 , avg=553594.77, stdev=83784.27
     lat (msec): min=40 , max=783 , avg=590.99, stdev=86.72
    bw (KB/s) : min=    6, max=  131, per=98.23%, avg=104.12, stdev=19.29
  cpu          : usr=0.30%, sys=0.11%, ctx=601, majf=0, minf=20
  IO depths    : 1=0.2%, 2=0.3%, 4=0.7%, 8=1.3%, 16=97.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/599/0, short=0/0/0
     lat (usec): 500=0.17%
     lat (msec): 50=0.17%, 100=0.17%, 250=0.83%, 500=14.36%, 750=84.31%

Reading did not degrade or almost did not degrade. The record has degraded many times over (I remind you, drbd is in StandAlone mode)
In addition, there is a terrible delay on slat - more than 37 ms just for processing an io request by the disk stack.
What am I doing wrong with drbd? Is this abnormal behavior when just one more layer (drbd over md) cuts performance forty times?
DRBD 8.3, 3.5.0-27-generic kernel, Ubuntu 12.04 LTS system. The io scheduler is cfq.
Help?

Answer 1

Transfer metadata to a separate device. In general, yes, this is a well-known feature of drbd.

Answer 2

60Mb/s on Standalone is very little.
What version of DRBD? 8.3 is not enough, a minor release is also important.
Is the write cache on the raid definitely enabled?
Try no-barrier, no-flushes.
Check syslog and kern.log.

Answer 3

I seem to have missed something. Is this a soft raid?