fio storage performance manometry

1. Environmental Instructions:

This demo is a dellr620 physical server with centos7.6 x86_64-bit
The hardware-related information is as follows:

[root@mysql-redis scripts]# sh hw_msg1.sh 
+--------------------------------------------------------------+
| This Machine's Hyper-Threading is Enabled(recommend disable) |
+--------------------------------------------------------------+
Systembit    :    64
MEM info     :    20*4096 MB
Disk_totle   :    
Pro_SN_name  :    Product Name: PowerEdge R620 Serial Number: 5RV2962
System name  :    \S
Board_SN name:    Product Name: Serial Number:
CPU model    :    32  Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
CPU_phy_num  :    2
CPU_core_num :    8
CPU_logic_num:    16
Netcard info :    Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
+++++++++++++++++++++++++++++++++++
<<raid informations>>
 Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 893.75 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
--
Virtual Drive Information:
Virtual Drive: 1 (Target Id: 1)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 223.0 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
+++++++++++++++++++++++++++++++++++
<<Virtual Drive Information>>
Device Id: 3
PD Type: SATA
Raw Size: 894.252 GB [0x6fc81ab0 Sectors]
Inquiry Data: 56AB609UKCDU        TOSHIBA Q300.                           SAFM12.3
Device Speed: 6.0Gb/s 
Drive Temperature : N/A
Device Id: 0
PD Type: SATA
Raw Size: 223.570 GB [0x1bf244b0 Sectors]
Inquiry Data: PHDV723500E8240AGN  INTEL SSDSC2BB240G7                     N2010112
Device Speed: 6.0Gb/s 
Drive Temperature :25C (77.00 F)

2. Install fio

1.redhat linux system direct yum installation
yum install fio.x86_64
With direct yum installation on centos7.6, the default version is fio.3.7

2. Source Compilation and Installation
In order to test asynchronous io, libaio needs to be installed

yum install libaio-devel

wget  -P /data/soft http://github.com/axboe/fio/archive/fio-3.7.tar.gz
tar xf /data/soft/fio.3.7.tar.gz -C /usr/local/
cd /usr/local/fio.3.7
./configure
make && make install

3. Introduction of Important Parameters

Fio's parameters vary from release to release, but here's only a list of the most important parameters that are commonly used everyday
filename=/dev/sdb1 or/data/soft/test1: Specify the name of the test hard disk or the name of the test file (never select a system device test, which will cause the system to crash)
direct=1: The test process bypasses the machine's own buffer to make the test results more accurate.
rw=randwrite:Write randomly.rw Available Values: randwrite/randread/read/write/randrw/trim/randtrim/trim write; Random Write, Random Read, Sequential Read, Sequential Write, Mixed Random Read and Write,
bs=16k: The size of a single I/O block is 16KB. The size of a single I/O block has a greater impact on IOPS. Generally speaking, to get the largest IOPS, the smaller the bs, the better.Of course, this is also related to the smallest size of the file system.Generally, the smallest file partition size under linux is 512B,1KB,2KB,4KB,8KB and so on, which can be selected according to the file system type.
size=5g: Test file size is 5GB
Ioengine=libaio: The I/O engine uses libaio, an asynchronous I/O engine that submits a batch of I/Os at a time and then waits for them to complete, which reduces the number of interactions and is more efficient.FIO supports at least 13 different I/O engines.sync,mmap,libaio,network, etc.
Iodepth=4: The I/O queue depth is adjusted based on the degree of parallelism of the device.There are usually two ways of I/O access.That is, synchronous I/O, and asynchronous I/O. Synchronous I/O can only send one I/O request at a time and wait for the kernel to complete before returning.This is always less than 1 for a single thread. If you want iodepth to be greater than 1, you can do so through multithreaded concurrent execution.Asynchronous I/O submits multiple I/O requests at once.Wait for I/O to finish or harvest once at intervals.Iodepth is commonly used in asynchronous I/O models (libaio) to specify how many I/O requests are initiated at a time.Flash storage generally has multiple parallel units.Supports multiple I/O concurrent execution.I/O for better asynchronization
-numjobs=4:Use four threads to test the hard disk device For Raid devices, such as raid1,raid5, or devices with high parallelism, consider increasing the number of test threads-numjobs to test appropriately.

runtime=1000: The test time is 1000s. Works with the parameter size, which is a double limit, i.e. stops testing if any limit is reached.

rwmixread=70: In mixed read-write mode, 70% read
group_reporting: Set the display results, specify group_reporting summarizes information for each process.
-name iops_randwrite: Rename the test iops_randwrite
--outputTestResult.log: Test results specify output to fileTestResult.login

IV. Testing io performance

You can use fio simulation to test different types of io requests, including random write, sequential write, random read, sequential read.Mixed Random Read-Write

4.1 Write randomly:

 time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite   --output TestResult.log

Command description:
A direct asynchronous random write (libaio) test with 4 threads on the / data/soft/test1 file for 60s, a queue depth of 1, and a block size of 4 kb, named iops_randwrite.Output results are not displayed by four jobs but are summarized by group, resulting in IOPS randomly written from this file system under this pressure.For Raid devices, such as raid1,raid5, or devices with high parallelism, consider appropriately increasing the number of test threads - numjobs to test.
--outputTestResult.logOutput to specified log

Here is a random write of the specific execution process:

 [root@mysql-redis soft]# time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite 
iops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 4 processes
iops_randwrite: Laying out IO file (1 file / 5120MiB)
Jobs: 4 (f=4): [w(4)][100.0%][r=0KiB/s,w=68.2MiB/s][r=0,w=17.5k IOPS][eta 00m:00s]
iops_randwrite: (groupid=0, jobs=4): err= 0: pid=13505: Thu May 21 18:50:33 2020
  write: IOPS=39.2k, BW=153MiB/s (160MB/s)(9182MiB/60001msec)
    slat (usec): min=6, max=27813, avg=17.42, stdev=36.21
    clat (nsec): min=1177, max=123217k, avg=82183.17, stdev=323538.76
     lat (usec): min=53, max=123230, avg=99.84, stdev=325.69
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   51], 10.00th=[   57], 20.00th=[   62],
     | 30.00th=[   65], 40.00th=[   70], 50.00th=[   75], 60.00th=[   79],
     | 70.00th=[   84], 80.00th=[   91], 90.00th=[  103], 95.00th=[  117],
     | 99.00th=[  161], 99.50th=[  206], 99.90th=[  330], 99.95th=[  619],
     | 99.99th=[14615]
   bw (  KiB/s): min=17536, max=45304, per=25.12%, avg=39354.01, stdev=5145.44, samples=476
   iops        : min= 4384, max=11326, avg=9838.50, stdev=1286.36, samples=476
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=4.22%
  lat (usec)   : 100=84.07%, 250=11.50%, 500=0.12%, 750=0.03%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=4.33%, sys=19.34%, ctx=2824889, majf=0, minf=1185
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2350467,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=9182MiB (9628MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=0/2349456, merge=0/0, ticks=0/167181, in_queue=166516, util=96.46%

real    1m0.706s
user    0m11.167s
sys 0m47.017s

4.2 Random reading:

time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread 

Instructions on command parameters:
Command description: A direct asynchronous random read (libaio) test named iops_with a duration of 60s, a queue depth of 1, and a block size of 4 KB is performed on the / data/soft/test1 file using two threadsRandread. Output results are not displayed by two jobs, but are summarized by the group, resulting in IOPS randomly read from this file system under this pressure

Here is a random reading of the specific execution process:

 [root@mysql-redis39 ~]# time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread 
iops_randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [r(2)][100.0%][r=36.7MiB/s,w=0KiB/s][r=9389,w=0 IOPS][eta 00m:00s]
iops_randread: (groupid=0, jobs=2): err= 0: pid=24566: Fri May 22 06:21:46 2020
   read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec)
    slat (usec): min=7, max=1480, avg=23.09, stdev= 8.09
    clat (usec): min=8, max=123984, avg=185.97, stdev=570.65
     lat (usec): min=136, max=123999, avg=209.50, stdev=570.70
    clat percentiles (usec):
     |  1.00th=[  139],  5.00th=[  141], 10.00th=[  145], 20.00th=[  169],
     | 30.00th=[  174], 40.00th=[  178], 50.00th=[  182], 60.00th=[  188],
     | 70.00th=[  198], 80.00th=[  204], 90.00th=[  212], 95.00th=[  219],
     | 99.00th=[  255], 99.50th=[  273], 99.90th=[  322], 99.95th=[  537],
     | 99.99th=[ 1549]
   bw (  KiB/s): min=13824, max=19616, per=49.99%, avg=18713.87, stdev=1060.66, samples=238
   iops        : min= 3456, max= 4904, avg=4678.43, stdev=265.16, samples=238
  lat (usec)   : 10=0.01%, 100=0.01%, 250=98.87%, 500=1.08%, 750=0.02%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 20=0.01%, 250=0.01%
  cpu          : usr=4.11%, sys=15.44%, ctx=561546, majf=0, minf=498
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=561503,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=36.6MiB/s (38.3MB/s), 36.6MiB/s-36.6MiB/s (38.3MB/s-38.3MB/s), io=2193MiB (2300MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=560393/2, merge=0/0, ticks=102314/0, in_queue=101910, util=97.80%

real    1m0.706s
user    0m5.649s
sys 0m19.196s

The above indicators need attention:
Read: iops=9358, BW=36.6MiB/s (38.3MB/s) (2193MiB/60001msec) io=2193MiB (2300MB) These parameters give you insight into iops and traffic
Lat(usec): min=136, max=123999, avg=209.50, stdev=570.70 These parameters also need to be noted: the io delay averages 209.5 subtle.

4.3 Write in sequence:

time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write

Description: A direct asynchronous sequential write (libaio) stress test with a block size of 1 M and a queue depth of 1 is named bw_using four threads for 60s Write.Output results are not shown separately by four job s, but are presented as a group summary.

[root@mysql-redis ~]# time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write  
bw_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 5 processes
bw_write: Laying out IO file (1 file / 5120MiB)
Jobs: 5 (f=5): [W(5)][100.0%][r=0KiB/s,w=184MiB/s][r=0,w=184 IOPS][eta 00m:00s]
bw_write: (groupid=0, jobs=5): err= 0: pid=26278: Fri May 22 06:44:38 2020
  write: IOPS=319, BW=320MiB/s (335MB/s)(18.7GiB/60009msec)
    slat (usec): min=56, max=303, avg=140.02, stdev=29.86
    clat (msec): min=3, max=1092, avg=15.49, stdev=67.68
     lat (msec): min=3, max=1093, avg=15.63, stdev=67.68

4.4 Sequential reading:

time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read 
[root@mysql-redis ~]# time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read 
bw_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=259MiB/s,w=0KiB/s][r=259,w=0 IOPS][eta 00m:00s]
bw_read: (groupid=0, jobs=2): err= 0: pid=26789: Fri May 22 06:51:01 2020
   read: IOPS=255, BW=256MiB/s (268MB/s)(10.0GiB/40034msec)
    slat (usec): min=95, max=1149, avg=245.64, stdev=42.23
    clat (msec): min=4, max=130, avg= 7.57, stdev= 2.98
     lat (msec): min=4, max=130, avg= 7.81, stdev= 2.98

4.5 Mixed Read-Write

time fio  -rw=randrw  -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw

Command description:
Two threads did a direct asynchronous mixed read-write (libaio) stress test on the / data/soft/test2 file for 60s with a queue depth of 1 and a block size of 16 kb. The read-write ratio was pseudo 7:3 and the test was named iops_randrw

rwmixwrite=30 means that in mixed read-write mode, writing accounts for 30%.
-rwmixread=70 means 70% reading in mixed read-write mode.

Mixed Read-Write output:

[root@mysql-redis ~]# time fio  -rw=randrw  -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw

iops_randrw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [m(2)][100.0%][r=62.0MiB/s,w=27.2MiB/s][r=4030,w=1740 IOPS][eta 00m:00s]
iops_randrw: (groupid=0, jobs=2): err= 0: pid=27095: Fri May 22 06:55:08 2020
   read: IOPS=3931, BW=61.4MiB/s (64.4MB/s)(3686MiB/60001msec)
    slat (usec): min=8, max=6353, avg=36.20, stdev=19.77
    clat (nsec): min=1648, max=123466k, avg=346175.56, stdev=725094.33
     lat (usec): min=12, max=123516, avg=382.99, stdev=725.55
    clat percentiles (usec):
     |  1.00th=[  215],  5.00th=[  241], 10.00th=[  255], 20.00th=[  277],
     | 30.00th=[  293], 40.00th=[  310], 50.00th=[  322], 60.00th=[  334],
     | 70.00th=[  351], 80.00th=[  375], 90.00th=[  412], 95.00th=[  449],
     | 99.00th=[ 1221], 99.50th=[ 1467], 99.90th=[ 1631], 99.95th=[ 1827],
     | 99.99th=[16712]
   bw (  KiB/s): min=23328, max=33792, per=49.97%, avg=31431.39, stdev=1649.11, samples=238
   iops        : min= 1458, max= 2112, avg=1964.44, stdev=103.06, samples=238
  write: IOPS=1686, BW=26.4MiB/s (27.6MB/s)(1581MiB/60001msec)
    slat (usec): min=12, max=2005, avg=42.06, stdev=16.09
    clat (usec): min=3, max=123270, avg=225.73, stdev=432.84
     lat (usec): min=86, max=123320, avg=268.44, stdev=433.42
    clat percentiles (usec):
     |  1.00th=[   78],  5.00th=[   85], 10.00th=[  100], 20.00th=[  129],
     | 30.00th=[  149], 40.00th=[  174], 50.00th=[  202], 60.00th=[  233],
     | 70.00th=[  265], 80.00th=[  306], 90.00th=[  359], 95.00th=[  408],
     | 99.00th=[  510], 99.50th=[ 1090], 99.90th=[ 1549], 99.95th=[ 1614],
     | 99.99th=[ 5080]
   bw (  KiB/s): min= 9152, max=15329, per=49.97%, avg=13484.37, stdev=892.91, samples=238
   iops        : min=  572, max=  958, avg=842.76, stdev=55.80, samples=238
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=3.06%
  lat (usec)   : 250=22.22%, 500=72.91%, 750=0.76%, 1000=0.05%
  lat (msec)   : 2=0.96%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=4.64%, sys=15.91%, ctx=346233, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=235884,101198,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=61.4MiB/s (64.4MB/s), 61.4MiB/s-61.4MiB/s (64.4MB/s-64.4MB/s), io=3686MiB (3865MB), run=60001-60001msec
  WRITE: bw=26.4MiB/s (27.6MB/s), 26.4MiB/s-26.4MiB/s (27.6MB/s-27.6MB/s), io=1581MiB (1658MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=274477/101294, merge=0/0, ticks=88860/19119, in_queue=107576, util=95.82%

real    1m0.690s
user    0m6.300s
sys 0m19.718s

4.6SSD Performance Test Profile

The configuration file is as follows:

[root@mysql-redis scripts]# cat /data/soft/ssd-test.fio 
##2020-05-24
[global]
bs=4k
ioengine=libaio
iodepth=4
numjobs=1
group_reporting
size=5g
direct=1
runtime=60
directory=/data/soft
filename=ssd.test.file
#inflate-log=/data/soft/TestResult.log
#output=/data/soft/TestResult.log
[seq-read]
name=iops_seqread
rw=read
stonewall

[rand-read]
name=iops_randread
rw=randread
stonewall

[seq-write]
name=iops_seqwrite
rw=write
stonewall

[rand-write]
name=iops_randwrite
rw=randwrite
stonewall

[randrw]
name=iops_randrw
rwmixread=70
rw=randrw
stonewall

Here is what the fio test specifies for the profile output:

[root@mysql-redis soft]# fio ssd-test.fio 
iops_seqread: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randread: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_seqwrite: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randwrite: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randrw: (g=4): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.7
Starting 5 processes
iops_seqread: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [_(4),m(1)][90.5%][r=47.8MiB/s,w=20.7MiB/s][r=12.2k,w=5308 IOPS][eta 00m:21s]     
iops_seqread: (groupid=0, jobs=1): err= 0: pid=2142: Sun May 24 20:06:20 2020
   read: IOPS=57.5k, BW=225MiB/s (236MB/s)(5120MiB/22782msec)
    slat (usec): min=3, max=1943, avg= 5.13, stdev= 2.21
    clat (usec): min=25, max=12467, avg=63.13, stdev=202.65
     lat (usec): min=40, max=12472, avg=68.41, stdev=202.72
    clat percentiles (usec):
     |  1.00th=[   45],  5.00th=[   53], 10.00th=[   55], 20.00th=[   56],
     | 30.00th=[   56], 40.00th=[   56], 50.00th=[   57], 60.00th=[   57],
     | 70.00th=[   57], 80.00th=[   58], 90.00th=[   59], 95.00th=[   62],
     | 99.00th=[   69], 99.50th=[   79], 99.90th=[ 2073], 99.95th=[ 5080],
     | 99.99th=[10814]
   bw (  KiB/s): min=43579, max=193812, per=75.80%, avg=174446.82, stdev=47419.87, samples=45
   iops        : min=10894, max=48453, avg=43611.36, stdev=11855.06, samples=45
  lat (usec)   : 50=3.39%, 100=96.25%, 250=0.08%, 500=0.07%, 750=0.04%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.06%, 4=0.04%, 10=0.05%, 20=0.01%
  cpu          : usr=19.83%, sys=39.29%, ctx=1222141, majf=0, minf=114
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_randread: (groupid=1, jobs=1): err= 0: pid=2170: Sun May 24 20:06:20 2020
   read: IOPS=21.7k, BW=84.9MiB/s (89.0MB/s)(5094MiB/60001msec)
    slat (usec): min=3, max=1068, avg=12.62, stdev= 6.14
    clat (usec): min=42, max=38568, avg=167.76, stdev=315.76
     lat (usec): min=62, max=38588, avg=180.75, stdev=316.07
    clat percentiles (usec):
     |  1.00th=[  130],  5.00th=[  135], 10.00th=[  137], 20.00th=[  141],
     | 30.00th=[  143], 40.00th=[  147], 50.00th=[  151], 60.00th=[  155],
     | 70.00th=[  161], 80.00th=[  172], 90.00th=[  192], 95.00th=[  206],
     | 99.00th=[  251], 99.50th=[  273], 99.90th=[ 3982], 99.95th=[ 7308],
     | 99.99th=[14091]
   bw (  KiB/s): min= 5397, max=74196, per=79.00%, avg=68682.74, stdev=15713.88, samples=119
   iops        : min= 1349, max=18549, avg=17170.32, stdev=3928.48, samples=119
  lat (usec)   : 50=0.01%, 100=0.02%, 250=98.94%, 500=0.82%, 750=0.02%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.03%, 4=0.04%, 10=0.07%, 20=0.02%, 50=0.01%
  cpu          : usr=11.52%, sys=37.46%, ctx=668232, majf=0, minf=319
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1304064,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_seqwrite: (groupid=2, jobs=1): err= 0: pid=2231: Sun May 24 20:06:20 2020
  write: IOPS=47.1k, BW=184MiB/s (193MB/s)(5120MiB/27848msec)
    slat (usec): min=4, max=139, avg= 7.50, stdev= 2.96
    clat (usec): min=27, max=15374, avg=76.14, stdev=278.53
     lat (usec): min=49, max=15381, avg=83.78, stdev=279.26
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   59], 10.00th=[   60], 20.00th=[   61],
     | 30.00th=[   62], 40.00th=[   64], 50.00th=[   65], 60.00th=[   67],
     | 70.00th=[   68], 80.00th=[   71], 90.00th=[   73], 95.00th=[   74],
     | 99.00th=[   81], 99.50th=[   89], 99.90th=[ 3326], 99.95th=[ 3392],
     | 99.99th=[15139]
   bw (  KiB/s): min= 3769, max=175262, per=80.83%, avg=152166.87, stdev=43009.10, samples=55
   iops        : min=  942, max=43815, avg=38041.35, stdev=10752.22, samples=55
  lat (usec)   : 50=0.24%, 100=99.43%, 250=0.07%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.21%, 10=0.01%, 20=0.03%
  cpu          : usr=11.79%, sys=43.48%, ctx=652945, majf=0, minf=95
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_randwrite: (groupid=3, jobs=1): err= 0: pid=2262: Sun May 24 20:06:20 2020
  write: IOPS=47.2k, BW=185MiB/s (194MB/s)(5120MiB/27743msec)
    slat (usec): min=4, max=415, avg= 7.49, stdev= 2.82
    clat (usec): min=26, max=21032, avg=75.63, stdev=276.91
     lat (usec): min=48, max=21058, avg=83.27, stdev=277.60
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   60], 10.00th=[   60], 20.00th=[   62],
     | 30.00th=[   62], 40.00th=[   64], 50.00th=[   65], 60.00th=[   67],
     | 70.00th=[   68], 80.00th=[   70], 90.00th=[   73], 95.00th=[   74],
     | 99.00th=[   80], 99.50th=[   86], 99.90th=[ 3326], 99.95th=[ 3392],
     | 99.99th=[15139]
   bw (  KiB/s): min= 3175, max=142274, per=68.21%, avg=128901.20, stdev=33671.41, samples=55
   iops        : min=  793, max=35568, avg=32224.95, stdev=8417.81, samples=55
  lat (usec)   : 50=0.21%, 100=99.50%, 250=0.04%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.20%, 10=0.01%, 20=0.02%, 50=0.01%
  cpu          : usr=12.97%, sys=43.05%, ctx=654916, majf=0, minf=258
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_randrw: (groupid=4, jobs=1): err= 0: pid=2292: Sun May 24 20:06:20 2020
   read: IOPS=11.3k, BW=44.3MiB/s (46.5MB/s)(2659MiB/60001msec)
    slat (usec): min=4, max=594, avg=13.08, stdev= 6.88
    clat (usec): min=61, max=45073, avg=244.45, stdev=441.39
     lat (usec): min=134, max=45115, avg=257.84, stdev=441.61
    clat percentiles (usec):
     |  1.00th=[  137],  5.00th=[  143], 10.00th=[  151], 20.00th=[  176],
     | 30.00th=[  190], 40.00th=[  202], 50.00th=[  212], 60.00th=[  221],
     | 70.00th=[  233], 80.00th=[  258], 90.00th=[  330], 95.00th=[  359],
     | 99.00th=[  474], 99.50th=[ 1270], 99.90th=[ 6521], 99.95th=[10028],
     | 99.99th=[17695]
   bw (  KiB/s): min= 3072, max=51824, per=99.92%, avg=45336.55, stdev=11250.66, samples=119
   iops        : min=  768, max=12958, avg=11334.12, stdev=2812.66, samples=119
  write: IOPS=4860, BW=18.0MiB/s (19.9MB/s)(1139MiB/60001msec)
    slat (usec): min=6, max=874, avg=16.94, stdev= 7.42
    clat (usec): min=40, max=31265, avg=192.80, stdev=413.19
     lat (usec): min=56, max=31306, avg=210.09, stdev=413.32
    clat percentiles (usec):
     |  1.00th=[   52],  5.00th=[   58], 10.00th=[   62], 20.00th=[   78],
     | 30.00th=[  145], 40.00th=[  169], 50.00th=[  188], 60.00th=[  202],
     | 70.00th=[  212], 80.00th=[  227], 90.00th=[  251], 95.00th=[  277],
     | 99.00th=[  619], 99.50th=[ 1205], 99.90th=[ 5997], 99.95th=[ 9634],
     | 99.99th=[16581]
   bw (  KiB/s): min= 1184, max=22176, per=99.94%, avg=19428.64, stdev=4847.41, samples=119
   iops        : min=  296, max= 5544, avg=4857.13, stdev=1211.85, samples=119
  lat (usec)   : 50=0.12%, 100=7.22%, 250=74.36%, 500=17.22%, 750=0.24%
  lat (usec)   : 1000=0.10%
  lat (msec)   : 2=0.52%, 4=0.06%, 10=0.11%, 20=0.04%, 50=0.01%
  cpu          : usr=7.88%, sys=29.46%, ctx=340101, majf=0, minf=374
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=680630,291634,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=5120MiB (5369MB), run=22782-22782msec

Run status group 1 (all jobs):
   READ: bw=84.9MiB/s (89.0MB/s), 84.9MiB/s-84.9MiB/s (89.0MB/s-89.0MB/s), io=5094MiB (5341MB), run=60001-60001msec

Run status group 2 (all jobs):
  WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5120MiB (5369MB), run=27848-27848msec

Run status group 3 (all jobs):
  WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5120MiB (5369MB), run=27743-27743msec

Run status group 4 (all jobs):
   READ: bw=44.3MiB/s (46.5MB/s), 44.3MiB/s-44.3MiB/s (46.5MB/s-46.5MB/s), io=2659MiB (2788MB), run=60001-60001msec
  WRITE: bw=18.0MiB/s (19.9MB/s), 18.0MiB/s-18.0MiB/s (19.9MB/s-19.9MB/s), io=1139MiB (1195MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=3294785/2912828, merge=0/3, ticks=430527/238028, in_queue=666582, util=99.09%
[root@mysql-redis soft]# 

V. Summary

Recommendations for fio test hard disks:
1. It is recommended that sequential I/O and larger blocksize be used to test the vomiting volume and delay of the device
2. It is recommended to use random I/O and smaller blocksize to test the IOPS and latency of the device
3. When configuring numjobs and io depth to test the underlying storage performance, it is recommended that you have a better understanding of whether synchronous or asynchronous I O is used (whether it is a multi-process concurrent i/o request or a batch of i/o requests submitted at once)
Tip: If you are interested in fio measuring disk performance, you can refer to ezfio (directly atGithub.comSearch ezfio)
Important note: fio supports direct I/O testing of bare devices. When testing bare devices, it is important to avoid the bare devices where the system is located, otherwise the system will run.
For example:

[root@mysql-redis soft]# df -h / /boot 
File System Capacity Used Available%Mountpoint
/dev/sdb5       207G  7.5G  200G    4% /
/dev/sdb1       297M  120M  177M   41% /boot

/dev/sdb/dev/sdb5/dev/sdb1 These devices are absolutely prohibited from fio testing.
At the same time, when testing the bare equipment of other devices, it is necessary to determine in advance whether there is important data on the bare equipment, and make a good backup of the important equipment in advance.

This is my simple learning summary of fio test hard disk for reference only. If there are users who directly copy the application causing losses, this blog does not bear any responsibility.
Reference documents:
http://www.ssdfans.com/?p=6087
https://linux.die.net/man/1/fio
https://blog.csdn.net/don_chiang709/article/details/92628623

Tags: Linux MySQL Redis iOS

Posted on Sun, 24 May 2020 12:20:30 -0400 by tha_mink