DELL R720xd服务器RAID的IOPS测试详解

一、服务器配置

品牌：DELL
型号：PowerEdge R710
CPU：2 * Intel(R) Xeon(R) CPU E5-2670 @ 2.60GHz
内存：24 * 8G
阵列卡：PERC 6/i
硬盘：8 * 300G 15k SAS + 4*4T 7.2k
系统：Centos 7

二、开始测试

1.1、6块盘做RAID 0(300G 15k SAS)

# fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.2.8
Starting 30 threads
Jobs: 30 (f=30): [m(30)] [100.0% done] [20288KB/7520KB/0KB /s] [1268/470/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=12776: Fri Jan 22 16:26:54 2016
read : io=1862.9MB, bw=19070KB/s, iops=1191, runt=100030msec
clat (usec): min=1000, max=273733, avg=15116.91, stdev=12049.09
lat (usec): min=1000, max=273733, avg=15117.38, stdev=12049.09
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 8],
| 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 14],
| 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 29], 95.00th=[ 38],
| 99.00th=[ 63], 99.50th=[ 75], 99.90th=[ 106], 99.95th=[ 126],
| 99.99th=[ 163]
bw (KB /s): min= 151, max= 1149, per=3.34%, avg=636.73, stdev=138.37
write: io=801536KB, bw=8012.1KB/s, iops=500, runt=100030msec
clat (msec): min=2, max=454, avg=23.88, stdev=21.12
lat (msec): min=2, max=454, avg=23.88, stdev=21.12
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10],
| 30.00th=[ 12], 40.00th=[ 15], 50.00th=[ 18], 60.00th=[ 21],
| 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 48], 95.00th=[ 65],
| 99.00th=[ 106], 99.50th=[ 127], 99.90th=[ 184], 99.95th=[ 208],
| 99.99th=[ 269]
bw (KB /s): min= 56, max= 553, per=3.34%, avg=267.30, stdev=75.82
lat (msec) : 2=0.01%, 4=1.13%, 10=34.87%, 20=36.43%, 50=23.42%
lat (msec) : 100=3.70%, 250=0.45%, 500=0.01%
cpu : usr=0.09%, sys=0.23%, ctx=170535, majf=0, minf=8
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=119224/w=50096/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
READ: io=1862.9MB, aggrb=19070KB/s, minb=19070KB/s, maxb=19070KB/s, mint=100030msec, maxt=100030msec
WRITE: io=801536KB, aggrb=8012KB/s, minb=8012KB/s, maxb=8012KB/s, mint=100030msec, maxt=100030msec
 
Disk stats (read/write):
sdb: ios=119189/58207, merge=0/32, ticks=1795936/1427321, in_queue=3224051, util=100.00%

1.2、4块盘做RAID 0(4T 7.2k)

# fio -filename=/dev/sdc1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.2.8
Starting 30 threads
Jobs: 30 (f=30): [m(30)] [100.0% done] [8896KB/4160KB/0KB /s] [556/260/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=12887: Fri Jan 22 16:34:54 2016
read : io=915504KB, bw=9144.5KB/s, iops=571, runt=100116msec
clat (usec): min=691, max=492461, avg=46226.85, stdev=39581.50
lat (usec): min=691, max=492461, avg=46227.33, stdev=39581.50
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 19],
| 30.00th=[ 23], 40.00th=[ 27], 50.00th=[ 33], 60.00th=[ 41],
| 70.00th=[ 51], 80.00th=[ 68], 90.00th=[ 97], 95.00th=[ 129],
| 99.00th=[ 194], 99.50th=[ 219], 99.90th=[ 289], 99.95th=[ 322],
| 99.99th=[ 383]
bw (KB /s): min= 31, max= 621, per=3.35%, avg=306.12, stdev=75.57
write: io=366624KB, bw=3661.2KB/s, iops=228, runt=100116msec
clat (usec): min=306, max=421747, avg=15478.87, stdev=17938.58
lat (usec): min=307, max=421748, avg=15480.04, stdev=17938.58
clat percentiles (usec):
| 1.00th=[ 892], 5.00th=[ 2064], 10.00th=[ 3376], 20.00th=[ 5024],
| 30.00th=[ 6752], 40.00th=[ 8640], 50.00th=[10432], 60.00th=[12864],
| 70.00th=[16064], 80.00th=[20864], 90.00th=[31872], 95.00th=[46848],
| 99.00th=[88576], 99.50th=[107008], 99.90th=[166912], 99.95th=[236544],
| 99.99th=[346112]
bw (KB /s): min= 19, max= 414, per=3.41%, avg=125.01, stdev=61.41
lat (usec) : 500=0.01%, 750=0.13%, 1000=0.26%
lat (msec) : 2=0.96%, 4=2.51%, 10=11.64%, 20=24.35%, 50=36.94%
lat (msec) : 100=16.28%, 250=6.74%, 500=0.17%
cpu : usr=0.04%, sys=0.12%, ctx=81065, majf=0, minf=8
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=57219/w=22914/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
READ: io=915504KB, aggrb=9144KB/s, minb=9144KB/s, maxb=9144KB/s, mint=100116msec, maxt=100116msec
WRITE: io=366624KB, aggrb=3661KB/s, minb=3661KB/s, maxb=3661KB/s, mint=100116msec, maxt=100116msec
 
Disk stats (read/write):
sdc: ios=57180/43825, merge=3/97, ticks=2640067/646381, in_queue=3287460, util=100.00%

2.1、6块盘做RAID 5(300G 15k SAS)

# fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.2.8
Starting 30 threads
Jobs: 30 (f=30): [m(30)] [100.0% done] [9200KB/3568KB/0KB /s] [575/223/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2491: Fri Jan 22 16:56:26 2016
 read : io=898000KB, bw=8974.6KB/s, iops=560, runt=100061msec
 clat (msec): min=1, max=220, avg=22.07, stdev=15.68
 lat (msec): min=1, max=220, avg=22.07, stdev=15.68
 clat percentiles (msec):
 | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12],
 | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 18], 60.00th=[ 21],
 | 70.00th=[ 25], 80.00th=[ 31], 90.00th=[ 42], 95.00th=[ 53],
 | 99.00th=[ 83], 99.50th=[ 94], 99.90th=[ 125], 99.95th=[ 141],
 | 99.99th=[ 176]
 bw (KB /s): min= 26, max= 669, per=3.35%, avg=300.58, stdev=108.64
 write: io=358896KB, bw=3586.8KB/s, iops=224, runt=100061msec
 clat (msec): min=15, max=401, avg=78.47, stdev=37.42
 lat (msec): min=15, max=401, avg=78.47, stdev=37.42
 clat percentiles (msec):
 | 1.00th=[ 28], 5.00th=[ 35], 10.00th=[ 41], 20.00th=[ 49],
 | 30.00th=[ 56], 40.00th=[ 63], 50.00th=[ 71], 60.00th=[ 80],
 | 70.00th=[ 89], 80.00th=[ 104], 90.00th=[ 127], 95.00th=[ 151],
 | 99.00th=[ 206], 99.50th=[ 233], 99.90th=[ 289], 99.95th=[ 318],
 | 99.99th=[ 396]
 bw (KB /s): min= 27, max= 253, per=3.33%, avg=119.39, stdev=33.01
 lat (msec) : 2=0.01%, 4=0.10%, 10=10.73%, 20=31.46%, 50=31.18%
 lat (msec) : 100=19.90%, 250=6.54%, 500=0.08%
 cpu : usr=0.04%, sys=0.11%, ctx=79363, majf=0, minf=9
 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued : total=r=56125/w=22431/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
 latency : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
 READ: io=898000KB, aggrb=8974KB/s, minb=8974KB/s, maxb=8974KB/s, mint=100061msec, maxt=100061msec
 WRITE: io=358896KB, aggrb=3586KB/s, minb=3586KB/s, maxb=3586KB/s, mint=100061msec, maxt=100061msec
 
Disk stats (read/write):
 sdb: ios=56100/23445, merge=0/17, ticks=1235317/1931813, in_queue=3168368, util=100.00%

2.2、4块盘做RAID 5(4T 7.2k)

# fio -filename=/dev/sdc1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.2.8
Starting 30 threads
Jobs: 30 (f=30): [m(30)] [100.0% done] [4480KB/2384KB/0KB /s] [280/149/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2572: Fri Jan 22 16:59:18 2016
 read : io=542784KB, bw=5422.9KB/s, iops=338, runt=100093msec
 clat (msec): min=3, max=481, avg=52.54, stdev=37.16
 lat (msec): min=3, max=481, avg=52.54, stdev=37.16
 clat percentiles (msec):
 | 1.00th=[ 13], 5.00th=[ 18], 10.00th=[ 21], 20.00th=[ 25],
 | 30.00th=[ 30], 40.00th=[ 36], 50.00th=[ 42], 60.00th=[ 50],
 | 70.00th=[ 60], 80.00th=[ 75], 90.00th=[ 99], 95.00th=[ 126],
 | 99.00th=[ 190], 99.50th=[ 221], 99.90th=[ 285], 99.95th=[ 326],
 | 99.99th=[ 424]
 bw (KB /s): min= 24, max= 432, per=3.35%, avg=181.79, stdev=67.00
 write: io=219472KB, bw=2192.7KB/s, iops=137, runt=100093msec
 clat (msec): min=14, max=503, avg=88.65, stdev=45.28
 lat (msec): min=14, max=503, avg=88.66, stdev=45.28
 clat percentiles (msec):
 | 1.00th=[ 27], 5.00th=[ 36], 10.00th=[ 42], 20.00th=[ 53],
 | 30.00th=[ 62], 40.00th=[ 70], 50.00th=[ 80], 60.00th=[ 90],
 | 70.00th=[ 102], 80.00th=[ 120], 90.00th=[ 147], 95.00th=[ 176],
 | 99.00th=[ 243], 99.50th=[ 269], 99.90th=[ 334], 99.95th=[ 355],
 | 99.99th=[ 433]
 bw (KB /s): min= 20, max= 195, per=3.44%, avg=75.48, stdev=31.45
 lat (msec) : 4=0.01%, 10=0.14%, 20=6.87%, 50=41.32%, 100=35.67%
 lat (msec) : 250=15.59%, 500=0.40%, 750=0.01%
 cpu : usr=0.02%, sys=0.09%, ctx=48738, majf=0, minf=5
 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued : total=r=33924/w=13717/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
 latency : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
 READ: io=542784KB, aggrb=5422KB/s, minb=5422KB/s, maxb=5422KB/s, mint=100093msec, maxt=100093msec
 WRITE: io=219472KB, aggrb=2192KB/s, minb=2192KB/s, maxb=2192KB/s, mint=100093msec, maxt=100093msec
 
Disk stats (read/write):
 sdc: ios=33898/14707, merge=3/12, ticks=1778522/1401726, in_queue=3186295, util=100.00%

3.1、6块盘做RAID 10(300G 15k SAS)

# fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.2.8
Starting 30 threads
Jobs: 30 (f=30): [m(30)] [100.0% done] [14689KB/5882KB/0KB /s] [918/367/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2447: Fri Jan 22 17:25:55 2016
 read : io=1405.9MB, bw=14390KB/s, iops=899, runt=100038msec
 clat (msec): min=1, max=214, avg=18.51, stdev=13.94
 lat (msec): min=1, max=214, avg=18.51, stdev=13.94
 clat percentiles (msec):
 | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9],
 | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 15], 60.00th=[ 17],
 | 70.00th=[ 21], 80.00th=[ 26], 90.00th=[ 36], 95.00th=[ 46],
 | 99.00th=[ 72], 99.50th=[ 84], 99.90th=[ 116], 99.95th=[ 133],
 | 99.99th=[ 163]
 bw (KB /s): min= 107, max= 894, per=3.34%, avg=480.74, stdev=121.36
 write: io=591968KB, bw=5917.5KB/s, iops=369, runt=100038msec
 clat (msec): min=3, max=253, avg=36.04, stdev=25.42
 lat (msec): min=3, max=253, avg=36.04, stdev=25.42
 clat percentiles (msec):
 | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 16],
 | 30.00th=[ 20], 40.00th=[ 24], 50.00th=[ 29], 60.00th=[ 36],
 | 70.00th=[ 43], 80.00th=[ 53], 90.00th=[ 70], 95.00th=[ 87],
 | 99.00th=[ 125], 99.50th=[ 143], 99.90th=[ 180], 99.95th=[ 192],
 | 99.99th=[ 227]
 bw (KB /s): min= 31, max= 445, per=3.33%, avg=197.11, stdev=55.92
 lat (msec) : 2=0.01%, 4=0.32%, 10=21.44%, 20=35.96%, 50=33.22%
 lat (msec) : 100=8.12%, 250=0.94%, 500=0.01%
 cpu : usr=0.07%, sys=0.17%, ctx=127979, majf=0, minf=7
 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued : total=r=89972/w=36998/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
 latency : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
 READ: io=1405.9MB, aggrb=14390KB/s, minb=14390KB/s, maxb=14390KB/s, mint=100038msec, maxt=100038msec
 WRITE: io=591968KB, aggrb=5917KB/s, minb=5917KB/s, maxb=5917KB/s, mint=100038msec, maxt=100038msec
 
Disk stats (read/write):
 sdb: ios=89947/44865, merge=0/32, ticks=1660513/1632659, in_queue=3294646, util=100.00%

3.2、4块盘做RAID 10(4T 7.2k)

# fio -filename=/dev/sdc1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.2.8
Starting 30 threads
Jobs: 30 (f=30): [m(30)] [100.0% done] [7280KB/2816KB/0KB /s] [455/176/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2535: Fri Jan 22 17:28:38 2016
 read : io=758976KB, bw=7586.2KB/s, iops=474, runt=100047msec
 clat (msec): min=2, max=594, avg=53.33, stdev=42.45
 lat (msec): min=2, max=594, avg=53.33, stdev=42.45
 clat percentiles (msec):
 | 1.00th=[ 10], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 23],
 | 30.00th=[ 28], 40.00th=[ 34], 50.00th=[ 41], 60.00th=[ 49],
 | 70.00th=[ 61], 80.00th=[ 77], 90.00th=[ 106], 95.00th=[ 139],
 | 99.00th=[ 212], 99.50th=[ 249], 99.90th=[ 338], 99.95th=[ 383],
 | 99.99th=[ 457]
 bw (KB /s): min= 52, max= 569, per=3.35%, avg=254.03, stdev=68.40
 write: io=300000KB, bw=2998.7KB/s, iops=187, runt=100047msec
 clat (usec): min=527, max=186677, avg=25049.31, stdev=22283.87
 lat (usec): min=528, max=186678, avg=25050.51, stdev=22283.89
 clat percentiles (usec):
 | 1.00th=[ 1688], 5.00th=[ 3856], 10.00th=[ 5408], 20.00th=[ 8032],
 | 30.00th=[10688], 40.00th=[13888], 50.00th=[17792], 60.00th=[22912],
 | 70.00th=[29824], 80.00th=[39168], 90.00th=[54016], 95.00th=[69120],
 | 99.00th=[107008], 99.50th=[122368], 99.90th=[148480], 99.95th=[162816],
 | 99.99th=[179200]
 bw (KB /s): min= 19, max= 377, per=3.44%, avg=103.23, stdev=51.31
 lat (usec) : 750=0.01%, 1000=0.04%
 lat (msec) : 2=0.34%, 4=1.15%, 10=7.00%, 20=17.65%, 50=42.58%
 lat (msec) : 100=22.56%, 250=8.33%, 500=0.34%, 750=0.01%
 cpu : usr=0.03%, sys=0.10%, ctx=67282, majf=0, minf=5
 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued : total=r=47436/w=18750/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
 latency : target=0, window=0, percentile=100.00%, depth=1
 
Run status group 0 (all jobs):
 READ: io=758976KB, aggrb=7586KB/s, minb=7586KB/s, maxb=7586KB/s, mint=100047msec, maxt=100047msec
 WRITE: io=300000KB, aggrb=2998KB/s, minb=2998KB/s, maxb=2998KB/s, mint=100047msec, maxt=100047msec
 
Disk stats (read/write):
 sdc: ios=47404/26003, merge=3/49, ticks=2525736/787504, in_queue=3314823, util=100.00%
四、结果总结
1、6*300G