How Linux views and tests disk IO performance

1. View disk IO performance  

1.1 top command

The top command judges the current disk IO performance by viewing the wa% value of the CPU. If this value is too large, it is likely that the disk IO is too high. Of course, it may also be due to other reasons, such as too high network io.

The meanings of other parameters of the top command are shown in top command details

1.2 sar command

sar command is an artifact to analyze system bottlenecks. It can be used to view the performance of CPU, memory, disk, network and so on.

The sar command to view the current disk performance is:

[root@server-68.2.stage.polex.io var ]$ sar -d -p 1 2
Linux 3.10.0-693.5.2.el7.x86_64 (server-68)     03/11/2019     _x86_64_    (64 CPU)

02:28:54 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
02:28:55 PM       sda      1.00      0.00      3.00      3.00      0.01      9.00      9.00      0.90
02:28:55 PM       sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:28:55 PM polex_pv-rootvol      1.00      0.00      3.00      3.00      0.01      9.00      9.00      0.90
02:28:55 PM polex_pv-varvol      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:28:55 PM polex_pv-homevol      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

02:28:55 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
02:28:56 PM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:28:56 PM       sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:28:56 PM polex_pv-rootvol      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:28:56 PM polex_pv-varvol      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:28:56 PM polex_pv-homevol      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:          sda      0.50      0.00      1.50      3.00      0.00      9.00      9.00      0.45
Average:          sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    polex_pv-rootvol      0.50      0.00      1.50      3.00      0.00      9.00      9.00      0.45
Average:    polex_pv-varvol      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    polex_pv-homevol      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

The "- d" parameter means to check the disk performance, the "- p" parameter means to display the dev device according to the name of sda, sdb,..., "1" means to take the value every 1s, and "2" means to take the value twice in total.

  • await: average wait time (in milliseconds) per device I/O operation.  

  • svctm: average service time (in milliseconds) per device I/O operation.

  • %util: a few percent of a second is spent on I/O operations.   

For disk IO performance, there are generally the following criteria:

Under normal circumstances, svctm should be less than the await value, and the size of svctm is related to disk performance. The load of CPU and memory will also affect the svctm value, and too many requests will indirectly lead to the increase of svctm value.

The value of await generally depends on the value of svctm, I/O queue length and I/O request mode. If the value of svctm is very close to await, it means that there is almost no I/O waiting and the disk performance is good. If the value of await is much higher than the value of svctm, it means that the I/O queue waiting is too long and the applications running on the system will slow down, At this time, the problem can be solved by replacing a faster hard disk.

%The value of util is also an important indicator to measure disk I/O. if% util is close to 100%, it means that the disk generates too many I/O requests, and the I/O system is working at full load, so the disk may have a bottleneck. In the long run, it is bound to affect the performance of the system. This problem can be solved by optimizing programs or replacing higher and faster disks.

By default, sar displays data from the nearest 0:00; If you want to continue to view the report of the previous day; You can view the sar logs saved in / var/log/sa /

[root@server-68.2.stage.polex.io var ]$ sar -d -p -f  /var/log/sa/sa11  | more
Linux 3.10.0-693.5.2.el7.x86_64 (server-68)     03/11/2019     _x86_64_    (64 CPU)

09:50:01 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
10:00:01 AM       sda      0.51      0.00      9.06     17.82      0.02     37.65     14.65      0.74
10:00:01 AM       sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:00:01 AM polex_pv-rootvol      0.22      0.00      2.50     11.56      0.01     37.44     14.10      0.31
10:00:01 AM polex_pv-varvol      0.30      0.00      6.55     21.97      0.01     38.55     14.73      0.44
10:00:01 AM polex_pv-homevol      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:10:01 AM       sda      0.79      3.45     13.18     21.06      0.04     51.81     11.03      0.87
10:10:01 AM       sdb      0.04      3.45      0.00     86.33      0.00      0.25      0.25      0.00
10:10:01 AM polex_pv-rootvol      0.26      0.00      3.08     11.85      0.01     50.21     17.88      0.46
10:10:01 AM polex_pv-varvol      0.54      3.45     10.10     24.95      0.03     52.58      7.49      0.41
10:10:01 AM polex_pv-homevol      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:20:01 AM       sda      0.65      0.00     10.43     16.11      0.03     38.67     10.99      0.71
10:20:01 AM       sdb      0.04      3.46      0.00     86.33      0.00      0.08      0.08      0.00

1.3 iostat command

Iostat is mainly used to monitor the IO load of system equipment. When iostat runs for the first time, it displays various statistical information since the system startup. After running iostat, it will display the statistical information since the last time the command was run. Users can obtain the required statistical information by specifying the number and time of statistics.

[root@server-68.2.stage.polex.io var ]$ iostat -dxk 1 2
Linux 3.10.0-693.5.2.el7.x86_64 (server-68)     03/11/2019     _x86_64_    (64 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.06    0.32    2.08     1.44    72.78    61.81     0.14   59.32    0.51   68.37   5.68   1.36
sdb               0.00     0.00    0.03    0.00     1.15     0.00    86.32     0.00    0.17    0.17    0.00   0.16   0.00
dm-0              0.00     0.00    0.00    0.24     0.02     1.56    13.22     0.01   44.55    6.36   44.71  13.25   0.32
dm-1              0.00     0.00    0.32    1.90     1.32    71.22    65.30     0.14   62.43    0.49   72.79   4.75   1.06
dm-2              0.00     0.00    0.00    0.00     0.00     0.00    26.79     0.00   28.06    4.68   38.98   5.18   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    3.00     0.00    16.00    10.67     0.26   86.33    0.00   86.33  42.33  12.70
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    3.00     0.00    16.00    10.67     0.26   86.33    0.00   86.33  42.33  12.70
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

The result of this command is the same as above    sar -d -p   one   two    Commands are similar. In actual use, we mainly look at   await   svctm  % util parameter.

1.4 vmstat command

  The vmstat command is simple to use:

[root@server-68.2.stage.polex.io var ]$ vmstat  2
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 93221488   4176 69117136    0    0     0     1    0    0  4  1 94  0  0
 2  0      0 93226048   4176 69117128    0    0     0     0 33326 36671 18  2 80  0  0
 1  0      0 93218776   4176 69117104    0    0     0     9 26225 21588 18  2 80  0  0
 1  0      0 93226072   4176 69117072    0    0     0     0 13271 25857  5  0 94  0  0
 0  0      0 93223984   4176 69117040    0    0     0     5 34637 24444 20  2 78  0  0
11  0      0 93219248   4176 69117184    0    0     0     0 30736 20671  8  2 90  0  0

 

In the output result, bi bo can represent the current performance of the disk:

  • bi    The number of blocks received by block devices per second. Block devices here refer to all disks and other block devices on the system. The default block size is 1024 byte s.

  • bo   The number of blocks sent by the block device per second. For example, when we read a file, Bo must be greater than 0. bi and Bo are generally close to 0, or IO is too frequent and needs to be adjusted.

2. Test disk IO performance

2.1 hdparm command

The hdparm command provides a command line interface for reading and setting IDE or SCSI hard disk parameters. Note that this command can only test the read rate of the disk.

For example, test the read rate of sda disk:

[root@server-68.2.stage.polex.io var ]$ hdparm -Tt /dev/polex_pv/varvol

/dev/polex_pv/varvol:
 Timing cached reads:   15588 MB in  2.00 seconds = 7803.05 MB/sec
 Timing buffered disk reads: 1128 MB in  3.01 seconds = 374.90 MB/sec

The test results show that the read rate with cache is 7803.05MB/s, and the actual read rate of disk is 374.90mb/s.

2.2 dd command

The Linux dd command is used to read, convert, and output data. dd can read data from standard input or file, convert the data according to the specified format, and then output it to file, device or standard output.

We can use the replication function of the dd command to test the IO performance of a disk. It should be noted that the dd command can only roughly measure the IO performance of the disk, which is not very accurate.

Test write performance command:

[root@server-68.2.stage.polex.io var ]$ time dd if=/dev/zero of=test.file bs=1G count=2 oflag=direct
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB) copied, 13.5487 s, 159 MB/s

real    0m13.556s
user    0m0.000s
sys    0m0.888s 

You can see that the disk write rate of this partition is 159M/s, where:

  • /The dev/zero pseudo device will generate a null character stream and will not generate IO for it.
  • The if parameter is used to specify the file read by the dd command.
  • The of parameter specifies the file to be written by the dd command.
  • The bs parameter represents the size of the block written each time.
  • The count parameter is used to specify the number of blocks written.
  • The offlag=direc parameter must be specified when testing IO, which means that it is written directly to the disk without using cache.

Test read performance command:

[root@server-68.2.stage.polex.io var ]$ dd if=test.file of=/dev/null  iflag=direct
4194304+0 records in
4194304+0 records out
2147483648 bytes (2.1 GB) copied, 4.87976 s, 440 MB/s

As you can see, the read rate of this partition is 440MB/s

2.3 fio command

The fio command is a special command for testing iops, which is more accurate than the dd command. The fio command has many parameters. Here are some examples for your reference:

Sequential read:

fio -filename=/var/test.file -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=test_r

Random write:

fio -filename=/var/test.file -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=test_randw

Sequential write:

fio -filename=/var/test.file -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=test_w

Mixed random read / write:

fio -filename=/var/test.file -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=test_r_w -ioscheduler=noop

 

Reference link:

https://elf8848.iteye.com/blog/2168876

https://linuxtools-rst.readthedocs.io/zh_CN/latest/tool/sar.html

http://blog.sina.com.cn/s/blog_62b832910102w3zt.html

https://www.cnblogs.com/ggjucheng/archive/2013/01/13/2858810.html

https://linux.cn/article-6104-1.html

http://www.runoob.com/linux/linux-comm-hdparm.html

 
 

Tags: Linux

Posted on Sat, 04 Dec 2021 16:42:23 -0500 by dvd420