sar command details

sar command details

sar is a very powerful performance analysis tool. It can obtain the performance indicators of the system, such as cpu, waiting queue, disk I/O, memory / network, etc

sar [options] [-o filename] [delay [count] ]
Parameter Description:
-A: Displays the health status of all resources in the system
-b: Displays disk IO and tranfer rate information. Like iostat information, it is the overall IO statistics
-d: Displays the disk activity during the refresh interval. You can specify one or more devices. Unlike -b, it displays a single disk
IO and transfer information of the device.
: it is recommended to use the display friendly device name with - p, otherwise the device name with primary and secondary device numbers will be displayed by default
-P: Displays the usage of a specified cpu or CPUs. The specified method is, - P 0,1,2,3 or ALL- u: Displays the overall average usage of each cpu- The difference between u and - P is easily distinguished by the following example.
-r: Displays the memory usage during the refresh interval
-n: Displays the network operation status. It can be followed by DEV/NFS/NFSD/ALL and other parameters.
: DEV indicates the display of network interface information, NFS and NFSD respectively indicate the display of traffic information of NFS client and server, and ALL indicates the display of network interface information
Show all information.
-q: Displays the size of the waiting queue
-o filename: save the results to a file
delay: status refresh interval
count: how many refresh times in total

1: Statistics of cpu usage

[root@rac1 ~]# sar -P ALL 1 2 
Linux 2.6.32-431.el6.x86_64 (rac1) 	09/26/2021 	_x86_64_	(16 CPU)

04:08:38 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:08:39 PM     all     14.66      0.00      1.57      0.19      0.00     83.58
04:08:39 PM       0     10.42      0.00      2.08      0.00      0.00     87.50
04:08:39 PM       1     29.59      0.00      4.08      0.00      0.00     66.33
04:08:39 PM       2     27.00      0.00      6.00      1.00      0.00     66.00
04:08:39 PM       3      2.00      0.00      2.00      0.00      0.00     96.00
04:08:39 PM       4      1.98      0.00      0.99      0.00      0.00     97.03
04:08:39 PM       5     12.00      0.00      1.00      0.00      0.00     87.00
04:08:39 PM       6     26.00      0.00      3.00      0.00      0.00     71.00
04:08:39 PM       7      2.00      0.00      2.00      0.00      0.00     96.00
04:08:39 PM       8      1.00      0.00      1.00      0.00      0.00     98.00
04:08:39 PM       9      2.94      0.00      0.00      0.98      0.00     96.08
04:08:39 PM      10    100.00      0.00      0.00      0.00      0.00      0.00
04:08:39 PM      11     15.00      0.00      3.00      0.00      0.00     82.00
04:08:39 PM      12      0.00      0.00      0.00      0.00      0.00    100.00
04:08:39 PM      13      1.01      0.00      1.01      0.00      0.00     97.98
04:08:39 PM      14      1.02      0.00      0.00      0.00      0.00     98.98
04:08:39 PM      15      1.00      0.00      1.00      0.00      0.00     98.00

04:08:39 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:08:40 PM     all     15.87      0.00      2.45      0.06      0.00     81.62
04:08:40 PM       0     16.16      0.00      6.06      0.00      0.00     77.78
04:08:40 PM       1      5.15      0.00      9.28      0.00      0.00     85.57
04:08:40 PM       2     21.43      0.00      7.14      0.00      0.00     71.43
04:08:40 PM       3      2.94      0.00      0.00      0.00      0.00     97.06
04:08:40 PM       4      2.91      0.00      0.97      0.97      0.00     95.15
04:08:40 PM       5      2.97      0.00      2.97      0.00      0.00     94.06
04:08:40 PM       6      3.03      0.00      2.02      0.00      0.00     94.95
04:08:40 PM       7      2.02      0.00      0.00      0.00      0.00     97.98
04:08:40 PM       8      2.00      0.00      0.00      0.00      0.00     98.00
04:08:40 PM       9      2.97      0.00      1.98      0.99      0.00     94.06
04:08:40 PM      10    100.00      0.00      0.00      0.00      0.00      0.00
04:08:40 PM      11     89.11      0.00     10.89      0.00      0.00      0.00
04:08:40 PM      12      0.98      0.00      0.00      0.98      0.00     98.04
04:08:40 PM      13      1.98      0.00      0.00      0.00      0.00     98.02
04:08:40 PM      14      1.00      0.00      0.00      0.00      0.00     99.00
04:08:40 PM      15      1.01      0.00      0.00      0.00      0.00     98.99

Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all     15.27      0.00      2.01      0.13      0.00     82.60
Average:          0     13.33      0.00      4.10      0.00      0.00     82.56
Average:          1     17.44      0.00      6.67      0.00      0.00     75.90
Average:          2     24.24      0.00      6.57      0.51      0.00     68.69
Average:          3      2.48      0.00      0.99      0.00      0.00     96.53
Average:          4      2.45      0.00      0.98      0.49      0.00     96.08
Average:          5      7.46      0.00      1.99      0.00      0.00     90.55
Average:          6     14.57      0.00      2.51      0.00      0.00     82.91
Average:          7      2.01      0.00      1.01      0.00      0.00     96.98
Average:          8      1.50      0.00      0.50      0.00      0.00     98.00
Average:          9      2.96      0.00      0.99      0.99      0.00     95.07
Average:         10    100.00      0.00      0.00      0.00      0.00      0.00
Average:         11     52.24      0.00      6.97      0.00      0.00     40.80
Average:         12      0.50      0.00      0.00      0.50      0.00     99.00
Average:         13      1.50      0.00      0.50      0.00      0.00     98.00
Average:         14      1.01      0.00      0.00      0.00      0.00     98.99
Average:         15      1.01      0.00      0.50      0.00      0.00     98.49


[root@rac1 ~]# sar -u ALL 1 2 
Linux 2.6.32-431.el6.x86_64 (rac1) 	09/26/2021 	_x86_64_	(16 CPU)

04:09:31 PM     CPU      %usr     %nice      %sys   %iowait    %steal      %irq     %soft    %guest     %idle
04:09:32 PM     all      8.17      0.00      0.75      0.06      0.00      0.00      0.00      0.00     91.01
04:09:33 PM     all      9.16      0.00      1.32      0.13      0.00      0.00      0.06      0.00     89.34
Average:        all      8.67      0.00      1.04      0.09      0.00      0.00      0.03      0.00     90.17

In the above example, the status information of all CPUs (0 ~ 15, 16 in total) per second is counted, and a summary is carried out every second, that is, all. Finally, the average value of each cpu and the summary all is calculated. What we really need to pay attention to is the idle value of the last average part. The smaller the idle value is, the less idle time the cpu is, and the higher the utilization rate of the cpu or the whole cpu is.

2: Statistics of memory usage

[root@rac1 ~]# sar -r 1 2 
Linux 2.6.32-431.el6.x86_64 (rac1) 	09/26/2021 	_x86_64_	(16 CPU)

04:13:19 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
04:13:20 PM  44428300  87727252     66.38   1215848  75231540  50284664     33.76
04:13:21 PM  44428416  87727136     66.38   1215848  75231504  50284664     33.76
Average:     44428358  87727194     66.38   1215848  75231522  50284664     33.76

Where kbdirty represents the size of dirty pages in memory, that is, how much data in memory should be flushed to disk.

3: Statistics of network traffic

3.1 viewing configuration files

View the / proc/net/dev file

[root@rac1 ~]# cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo:203426850197 646320725    0    0    0     0          0         0 203426850197 646320725    0    0    0     0       0          0
  eth0:442783371566 1701009301    0    0    0     0          0     50705 889277564264 1540231327    0    0    0     0       0          0
  eth2:2516904487490 3116720740    0    0    0     0          0     50721 2100534460110 2788465282    0    0    0     0       0          0

Attention column: receive and transmit indicate packet receiving and contracting respectively. Pay attention to the bytes of each network card to obtain the network card. Write a script to calculate the difference per second, which is the network traffic.

3.2 sar statistics network card interface data

The sar -n command counts the data of the network card interface.

[root@rac1 ~]# sar -n DEV  1 2 
Linux 2.6.32-431.el6.x86_64 (rac1) 	09/26/2021 	_x86_64_	(16 CPU)

04:19:53 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
04:19:54 PM        lo     22.68     22.68      1.86      1.86      0.00      0.00      0.00
04:19:54 PM      eth0    175.26    153.61     45.07     42.19      0.00      0.00      0.00
04:19:54 PM      eth2   1293.81    758.76    590.63    539.31      0.00      0.00      0.00

04:19:54 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
04:19:55 PM        lo    153.68    153.68     64.75     64.75      0.00      0.00      0.00
04:19:55 PM      eth0    316.84    283.16     64.12     70.75      0.00      0.00      0.00
04:19:55 PM      eth2   2034.74   1345.26   1337.26   1072.58      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo     87.50     87.50     32.98     32.98      0.00      0.00      0.00
Average:         eth0    245.31    217.71     54.50     56.32      0.00      0.00      0.00
Average:         eth2   1660.42   1048.96    960.06    803.17      0.00      0.00      0.00

The meaning of each column is as follows:

rxpck/s: number of packets received per second

txpck/s: number of packets sent per second

rxkB/s: data received per second, in kb

txkB/s: data sent per second, in kb

rxcmp/s: number of compressed packets received per second

txcmp/s: number of compressed packets sent per second

rxmcst/s: number of multicast packets received per second

4: View queue status

[root@rac1 ~]# sar -q
Linux 2.6.32-431.el6.x86_64 (rac1) 	09/26/2021 	_x86_64_	(16 CPU)

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
12:10:01 AM         0      1417      0.44      0.22      0.18
12:20:01 AM         2      1417      0.41      0.30      0.22
12:30:01 AM         1      1408      0.11      0.13      0.15
12:40:01 AM         1      1461      0.22      0.19      0.18
12:50:02 AM         2      1405      0.33      0.26      0.20
01:00:01 AM         2      1412      0.61      0.33      0.22
01:10:01 AM         1      1414      0.20      0.24      0.22
01:20:01 AM         0      1419      1.73      1.93      1.10
01:30:01 AM         1      1409      0.33      0.91      1.03
01:40:01 AM         0      1409      0.17      0.31      0.63
01:50:01 AM         0      1411      0.40      0.31      0.45
02:00:01 AM         1      1415      0.76      0.46      0.43
02:10:01 AM         4      1417      0.67      0.37      0.37
02:20:01 AM         0      1418      0.13      0.26      0.30
02:30:01 AM         0      1409      0.68      0.53      0.36
02:40:01 AM         0      1410      0.22      0.32      0.33
02:50:01 AM         0      1407      0.44      0.38      0.32
03:00:01 AM         1      1412      0.23      0.26      0.27
03:10:01 AM         1      1412      0.33      0.27      0.27
03:20:01 AM         0      1419      0.60      0.33      0.27
03:30:01 AM         0      1408      0.15      0.20      0.22
03:40:01 AM         2      1418      0.08      0.19      0.22
03:50:01 AM         0      1398      0.20      0.26      0.22
04:00:01 AM         1      1410      0.48      0.42      0.30
04:10:01 AM         1      1415      0.56      0.53      0.42
04:20:01 AM         0      1415      0.57      0.56      0.47
04:30:01 AM         1      1430      0.64      0.47      0.44
04:40:01 AM         0      1404      0.20      0.24      0.33
04:50:01 AM         1      1403      0.03      0.17      0.26

04:50:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
05:00:01 AM         0      1408      0.13      0.17      0.22
05:10:01 AM         1      1409      0.22      0.24      0.25
05:20:01 AM         1      1420      0.17      0.26      0.26
05:30:01 AM         0      1410      0.56      0.42      0.31
05:40:01 AM         0      1417      0.14      0.17      0.21
05:50:01 AM         0      1407      0.23      0.12      0.15
06:00:01 AM         0      1413      0.14      0.17      0.16
06:10:01 AM         0      1415      0.08      0.15      0.17
06:20:01 AM         1      1413      0.03      0.13      0.16
06:30:01 AM         1      1412      0.14      0.17      0.17
06:40:01 AM         0      1454      0.05      0.17      0.17
06:50:01 AM         1      1413      1.14      0.55      0.30
07:00:02 AM         1      1411      0.25      0.29      0.30
07:10:01 AM         1      1416      0.30      0.18      0.22
07:20:01 AM         0      1414      0.03      0.13      0.17
07:30:01 AM         2      1414      0.12      0.11      0.12
07:40:01 AM         1      1409      0.11      0.12      0.09
07:50:01 AM         0      1410      0.07      0.15      0.11
08:00:01 AM         4      1413      0.63      0.33      0.20
08:10:01 AM         3      1415      1.81      0.77      0.40
08:20:01 AM         0      1427      0.91      1.50      1.07
08:30:01 AM         1      1415      0.03      0.28      0.61
08:40:01 AM         3      1415      0.27      0.27      0.43
08:50:01 AM         1      1406      0.18      0.20      0.31
09:00:01 AM         1      1410      0.30      0.34      0.32
09:10:02 AM         1      1416      0.54      0.35      0.30
09:20:01 AM         1      1425      0.32      0.26      0.26
09:30:01 AM         0      1415      0.59      0.31      0.24
09:40:01 AM         3      1408      0.30      0.24      0.21

09:40:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
09:50:01 AM         1      1411      0.10      0.30      0.27
10:00:01 AM         0      1406      0.18      0.26      0.26
10:10:01 AM         4      1425      0.81      0.49      0.34
10:20:01 AM         1      1419      0.18      0.24      0.27
10:30:02 AM         1      1414      0.28      0.19      0.21
10:40:01 AM         2      1414      0.26      0.18      0.18
10:50:01 AM         0      1411      0.19      0.19      0.18
11:00:01 AM         2      1414      0.38      0.33      0.24
11:10:01 AM         3      1420      0.18      0.30      0.27
11:20:01 AM         1      1417      0.17      0.23      0.24
11:30:01 AM         7      1465      0.61      0.49      0.33
11:40:02 AM         2      1456      0.25      0.43      0.37
11:50:01 AM         2      1456      0.37      0.39      0.36
12:00:01 PM         2      1462      0.30      0.45      0.43
12:10:01 PM         5      1469      0.44      0.46      0.44
12:20:01 PM         4      1480      0.52      0.51      0.46
12:30:01 PM         2      1462      0.47      0.55      0.50
12:40:02 PM         6      1506      1.50      0.91      0.66
12:50:01 PM         5      1459      1.27      1.28      0.98
01:00:01 PM         6      1467      1.73      1.38      1.15
01:10:02 PM         3      1466      1.22      1.26      1.18
01:20:01 PM         7      1468      1.32      1.24      1.20
01:30:01 PM         4      1459      0.41      0.52      0.84
01:40:02 PM         5      1457      0.44      0.45      0.62
01:50:01 PM         2      1456      0.66      0.69      0.66
02:00:01 PM         1      1457      0.35      0.54      0.61
02:10:02 PM         5      1471      1.26      0.86      0.76
02:20:01 PM         1      1467      0.37      0.43      0.56
02:30:01 PM         2      1463      0.43      0.38      0.45

02:30:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
02:40:01 PM         3      1458      0.17      0.27      0.35
02:50:02 PM         3      1459      0.53      0.37      0.35
03:00:01 PM         3      1459      1.10      0.75      0.52
03:10:01 PM         2      1463      3.23      2.04      1.11
03:20:02 PM         1      1511      0.87      2.20      1.86
03:30:01 PM         6      1458      1.60      1.89      1.91
03:40:01 PM         3      1458      1.60      1.70      1.81
03:50:01 PM         2      1453      1.40      1.58      1.70
04:00:02 PM         4      1455      1.32      1.39      1.55
04:10:01 PM         1      1458      1.47      1.48      1.52
04:20:01 PM         3      1461      1.46      1.42      1.46
Average:            2      1430      0.54      0.52      0.48

Interpretation of the meaning of each column:

Runq SZ: the length of the waiting queue, excluding running processes

Plist SZ: the number of processes in the task list, that is, the total number of tasks

ldavg-N: average load of the system in the past 1 minute, 5 minutes and 15 minutes

Blocked: the current number of blocked tasks waiting for IO

5: Statistics of disk I/O

[root@rac1 ~]# sar -d -p 1 2 
Linux 2.6.32-431.el6.x86_64 (rac1) 	09/26/2021 	_x86_64_	(16 CPU)

04:30:36 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
04:30:37 PM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:37 PM       sdb      4.17      0.00     33.33      8.00      0.01      2.50      2.50      1.04
04:30:37 PM       sdc      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:37 PM       sdd   1573.96  34360.42    129.17     21.91      0.57      0.36      0.35     55.73
04:30:37 PM rootvg-lvroot      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:37 PM rootvg-lvswap      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:37 PM rootvg-lvopt      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:37 PM rootvg-lvvar      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:37 PM rootvg-lvhome      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:37 PM rootvg-lvgrid      4.17      0.00     33.33      8.00      0.01      2.50      2.50      1.04
04:30:37 PM rootvg-lvoracle      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:37 PM rootvg-lvorasoft      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

04:30:37 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
04:30:38 PM       sda      5.21      0.00    216.67     41.60      0.00      0.40      0.40      0.21
04:30:38 PM       sdb      4.17      0.00     33.33      8.00      0.01      2.25      2.25      0.94
04:30:38 PM       sdc      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:38 PM       sdd   1421.88  22343.75     84.38     15.77      0.47      0.33      0.32     46.04
04:30:38 PM rootvg-lvroot     21.88      0.00    175.00      8.00      0.02      0.90      0.05      0.10
04:30:38 PM rootvg-lvswap      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:38 PM rootvg-lvopt      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:38 PM rootvg-lvvar      5.21      0.00     41.67      8.00      0.00      0.20      0.20      0.10
04:30:38 PM rootvg-lvhome      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:38 PM rootvg-lvgrid      4.17      0.00     33.33      8.00      0.01      2.25      2.25      0.94
04:30:38 PM rootvg-lvoracle      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:38 PM rootvg-lvorasoft      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:          sda      2.60      0.00    108.33     41.60      0.00      0.40      0.40      0.10
Average:          sdb      4.17      0.00     33.33      8.00      0.01      2.38      2.37      0.99
Average:          sdc      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:          sdd   1497.92  28352.08    106.77     19.00      0.52      0.35      0.34     50.89
Average:    rootvg-lvroot     10.94      0.00     87.50      8.00      0.01      0.90      0.05      0.05
Average:    rootvg-lvswap      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    rootvg-lvopt      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    rootvg-lvvar      2.60      0.00     20.83      8.00      0.00      0.20      0.20      0.05
Average:    rootvg-lvhome      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    rootvg-lvgrid      4.17      0.00     33.33      8.00      0.01      2.38      2.37      0.99
Average:    rootvg-lvoracle      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    rootvg-lvorasoft      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

The IO usage from 04:30:36 to 04:30:37 and from 04:30:37 to 04:30:38 are counted respectively.

The meaning of each column is as follows:

tps: transfer per second, the transfer rate per second. One physical IO request is counted once

Transfer, but multiple logical IO requests may be combined to calculate a transfer.

rd_sec/s: the number of sectors read per second. The sector size is 512 bytes.

wr_sec/s: number of sectors written per second.

avgrq-sz: the average size of the requested write device, in sectors. (The average size (in sectors) of the requests that were issued to the device)

avgqu-sz: the average queue length of the request to write to the device. (The average queue length of the requests that were issued to the device.)

await: The average time for I/O requests issued to the device to be served

svctm: an untrusted column that will be removed in the future, so don't worry

%util: the most important column shows the bandwidth of the device. If this column is close to 100%, it indicates that the disk speed is saturated.

Tags: Linux

Posted on Sun, 26 Sep 2021 03:54:35 -0400 by baconbeastnz