Linux, why is writing to disk faster than reading?

Question

linux

I decided to test the discs here. Found an article on how to do it right . But the results confuse me, the IOPS record is clearly higher, although direct=1 is clearly specified in the settings. Here are the test parameters:

[global]
blocksize=4k
filename=/dev/sda2
direct=1
buffered=0
ioengine=libaio
iodepth=32

[readtest]
rw=randread
[writetest]
rw=randwrite

And these are the results:

readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
writetest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.3
Starting 2 processes
^Cbs: 2 (f=2): [rw] [0.0% done] [256KB/472KB/0KB /s] [64/118/0 iops] [eta 05d:16h:47m:12s]
fio: terminating on signal 2
Jobs: 1 (f=1): [r_] [0.0% done] [176KB/412KB/0KB /s] [44/103/0 iops] [eta 05d:16h:55m:03s]
readtest: (groupid=0, jobs=1): err= 0: pid=30307: Wed Mar 19 10:03:26 2014
  read : io=32616KB, bw=213912B/s, iops=52, runt=156133msec
    slat (usec): min=5, max=150, avg=19.58, stdev= 7.60
    clat (msec): min=7, max=7383, avg=612.65, stdev=610.66
     lat (msec): min=7, max=7383, avg=612.67, stdev=610.66
    clat percentiles (msec):
     |  1.00th=[   21],  5.00th=[   37], 10.00th=[   62], 20.00th=[  169],
     | 30.00th=[  215], 40.00th=[  314], 50.00th=[  416], 60.00th=[  562],
     | 70.00th=[  766], 80.00th=[ 1045], 90.00th=[ 1352], 95.00th=[ 1614],
     | 99.00th=[ 2474], 99.50th=[ 3163], 99.90th=[ 6194], 99.95th=[ 6325],
     | 99.99th=[ 7373]
    bw (KB  /s): min=   25, max=  365, per=100.00%, avg=223.15, stdev=56.56
    lat (msec) : 10=0.04%, 20=0.76%, 50=7.49%, 100=3.77%, 250=23.17%
    lat (msec) : 500=19.79%, 750=14.23%, 1000=9.09%, 2000=19.11%, >=2000=2.56%
  cpu          : usr=0.17%, sys=0.33%, ctx=8155, majf=0, minf=57
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=8154/w=0/d=0, short=r=0/w=0/d=0
writetest: (groupid=0, jobs=1): err= 0: pid=30308: Wed Mar 19 10:03:26 2014
  write: io=69824KB, bw=458320B/s, iops=111, runt=156004msec
    slat (usec): min=4, max=176, avg=15.77, stdev= 6.61
    clat (usec): min=322, max=2154.7K, avg=285938.25, stdev=313795.74
     lat (usec): min=342, max=2154.8K, avg=285954.76, stdev=313796.55
    clat percentiles (usec):
     |  1.00th=[ 1096],  5.00th=[ 4512], 10.00th=[ 9152], 20.00th=[26496],
     | 30.00th=[67072], 40.00th=[152576], 50.00th=[187392], 60.00th=[232448],
     | 70.00th=[342016], 80.00th=[481280], 90.00th=[716800], 95.00th=[1011712],
     | 99.00th=[1351680], 99.50th=[1515520], 99.90th=[1777664], 99.95th=[1908736],
     | 99.99th=[2056192]
    bw (KB  /s): min=    7, max= 1505, per=100.00%, avg=468.19, stdev=230.34
    lat (usec) : 500=0.07%, 750=0.39%, 1000=0.33%
    lat (msec) : 2=1.23%, 4=2.38%, 10=6.55%, 20=6.46%, 50=9.66%
    lat (msec) : 100=5.09%, 250=30.14%, 500=18.98%, 750=9.71%, 1000=3.77%
    lat (msec) : 2000=5.21%, >=2000=0.02%
  cpu          : usr=0.37%, sys=0.57%, ctx=17404, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=17456/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=32616KB, aggrb=208KB/s, minb=208KB/s, maxb=208KB/s, mint=156133msec, maxt=156133msec
  WRITE: io=69824KB, aggrb=447KB/s, minb=447KB/s, maxb=447KB/s, mint=156004msec, maxt=156004msec

Disk stats (read/write):
  sda: ios=8387/17529, merge=27/5, ticks=5004436/5029920, in_queue=10042680, util=100.00%

Is that how it should be?

Answer 1

hm. Try 2GB or more to test more. and better for 100MB, 500MB, 1GB, 2GB. the result will be different, but we will get at least general statistics