Linux system running analysis tool - Summary of the use of the sarcommand

sar is the abbreviation of system activity reporter. At present, it is the most comprehensive system performance analysis tool on Linux. The activities of the system can be reported from the following aspects, including

File reading and writing
System call usage
Disk I/O
cpu efficiency
Memory usage
Process activities and IPC related content

sar installation and startup

yum install sysstat #Default path after installation find / -name sysstat /etc/sysconfig/sysstat /etc/cron.d/sysstat /etc/selinux/targeted/active/modules/100/sysstat #Start command systemctl start sysstat systemctl status sysstat systemctl stop sysstat

sar usage

1. Historical statistics

#The default log is saved under / var/log/sa /. It is saved by day and collected once every 10 minutes [root@node1 ~]# sar Linux 3.10.0-957.el7.x86_64 (k8s.node2) 06/26/2020 _x86_64_ (1 CPU) 08:00:01 AM CPU %user %nice %system %iowait %steal %idle 08:10:01 AM all 0.07 0.00 0.09 0.06 0.00 99.78 08:20:01 AM all 0.07 0.00 0.09 0.00 0.00 99.84 08:30:01 AM all 0.08 0.00 0.08 0.00 0.00 99.84 08:40:01 AM all 0.11 0.00 0.26 0.06 0.00 99.57 Average: all 0.08 0.00 0.13 0.03 0.00 99.76 #Or use the command sar -r -f /var/log/sysstat/sa26 [root@node1 ~]# sar -r -f /var/log/sa/sa26 Linux 3.10.0-957.el7.x86_64 (k8s.node2) 06/26/2020 _x86_64_ (1 CPU) 08:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 08:10:01 AM 427400 568496 57.08 2104 341612 448300 45.01 263976 166348 0 08:20:01 AM 426076 569820 57.22 2104 341620 450800 45.27 264840 166384 0 08:30:01 AM 427080 568816 57.12 2104 341608 448300 45.01 264072 166364 0 08:40:01 AM 400196 595700 59.82 2104 341740 452332 45.42 265584 166348 4 Average: 420188 575708 57.81 2104 341645 449933 45.18 264618 166361 1

2. Statistics of CPU usage

Sar-u 23 ා statistics every 2 seconds three times in total

[root@node1 ~]@ sar -u 2 3 Linux 3.10.0-957.el7.x86_64 (node1) 06/26/2020 _x86_64_ (1 CPU) 09:30:02 AM CPU %user %nice %system %iowait %steal %idle 09:30:04 AM all 0.00 0.00 0.50 0.00 0.00 99.50 09:30:06 AM all 0.00 0.00 0.00 0.00 0.00 100.00 09:30:08 AM all 0.50 0.00 0.00 0.00 0.00 99.50 Average: all 0.17 0.00 0.17 0.00 0.00 99.67 #View the usage of each CPU sar -p 2 3 #Output the result to the file under / tmp / at the same time sar -u -o /tmp/sa 2 3 #Read in from file sar -f /tmp/sa

#%User - CPU usage of user space

#%nice changes the CPU usage of the process that has been prioritized

#%CPU utilization of system kernel space

#%Percentage of iowait CPU waiting for IO

#%CPU used by virtual machine CPU of steel virtual machine

#%idle CPU

#In the above display, we mainly look at% iowait and% idle. If% iowait is too high, it means that there is an I/O bottleneck, that is, disk IO cannot meet the business requirements. If% idle is too low, it means that the CPU utilization is serious. We need to determine whether the CPU bottleneck is based on the memory utilization.

sar -q ා view average load

#Runq SZ run queue length (number of processes waiting to run, no more than 3 CP S per core)

#The number of processes and threads in plist SZ process list

#ldavg-1 average CPU load in the last minute, that is, the average value obtained by adding the load of multi-core CPU in the past minute and dividing by the number of cores, 5 minutes and 15 minutes and so on

#ldavg-5 average CPU load in the last five minutes

#ldavg-15 average CPU load in the last 15 minutes

3. Check memory usage

[root@node1 ~]@ sar -r Linux 3.10.0-957.el7.x86_64 (k8s.node2) 06/26/2020 _x86_64_ (1 CPU) 08:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 08:10:01 AM 427400 568496 57.08 2104 341612 448300 45.01 263976 166348 0 08:20:01 AM 426076 569820 57.22 2104 341620 450800 45.27 264840 166384 0 08:30:01 AM 427080 568816 57.12 2104 341608 448300 45.01 264072 166364 0 08:40:01 AM 400196 595700 59.82 2104 341740 452332 45.42 265584 166348 4 Average: 420188 575708 57.81 2104 341645 449933 45.18 264618 166361 1

#kbmemfree free free physical memory size

#Physical memory size in use of kbmemused

#%memused physical memory usage

#The size of physical memory used as buffer in kbbuffers kernel, kbbuffers and kbcached: these two values are buffer and cache in free command

#File size of kbcached cache

#kbcommit is the minimum memory required to ensure the normal operation of the current system, that is, the minimum memory required to ensure that memory does not overflow (physical memory + Swap partition)

#The value of commit is a percentage of kbcommit and total memory (physical memory + swap partition)

4. View system Swap partition information

[root@node1 ~]@ sar -w Linux 3.10.0-957.el7.x86_64 (k8s.node2) 06/26/2020 _x86_64_ (1 CPU) 08:00:01 AM proc/s cswch/s 08:10:01 AM 0.04 62.66 08:20:01 AM 0.02 56.84 08:30:01 AM 0.01 54.77 08:40:01 AM 0.05 64.78 Average: 0.03 59.76

#pswpin/s the number of swap page s per second from the swap partition to the system

#pswpott/s the number of swap page s exchanged from the system to swap per second

5. View I/O and transfer rate statistics

[root@node1 ~]@ sar -b Linux 3.10.0-957.el7.x86_64 (k8s.node2) 06/26/2020 _x86_64_ (1 CPU) 08:00:01 AM tps rtps wtps bread/s bwrtn/s 08:10:01 AM 0.15 0.01 0.14 0.16 3.22 08:20:01 AM 0.07 0.00 0.07 0.00 0.76 08:30:01 AM 0.06 0.00 0.06 0.00 0.71 08:40:01 AM 0.60 0.39 0.21 10.82 10.00 Average: 0.22 0.10 0.12 2.74 3.67

#tps the total IO per second of the disk, equal to tps in iostat

#Total IO read from disk by rtps per second

#wtps total IO written to disk per second

#bread/s total number of blocks read from disk per second

#bwrtn/s the total number of blocks written to disk per second

6. Statistics of disk usage

[root@node1 ~]@ sar -d Linux 3.10.0-957.el7.x86_64 (k8s.node2) 06/26/2020 _x86_64_ (1 CPU) 08:00:01 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 08:10:01 AM dev8-0 0.15 0.16 3.22 22.49 0.00 7.19 6.20 0.09 08:10:01 AM dev253-0 0.17 0.16 3.21 19.78 0.00 5.78 4.62 0.08 08:10:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 08:20:01 AM dev8-0 0.07 0.00 0.76 10.76 0.00 0.52 0.29 0.00 08:20:01 AM dev253-0 0.08 0.00 0.76 9.62 0.00 0.55 0.26 0.00 08:20:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 08:30:01 AM dev8-0 0.06 0.00 0.71 12.09 0.00 11.03 10.66 0.06 08:30:01 AM dev253-0 0.07 0.00 0.71 10.57 0.00 9.78 9.33 0.06 08:30:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 08:40:01 AM dev8-0 0.60 10.82 10.00 34.89 0.00 8.01 7.28 0.43 08:40:01 AM dev253-0 0.71 10.62 10.00 29.09 0.00 6.70 5.86 0.42 08:40:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev8-0 0.22 2.74 3.67 29.30 0.00 7.47 6.76 0.15 Average: dev253-0 0.26 2.69 3.67 24.84 0.00 6.28 5.45 0.14 Average: dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

#The name of DEV disk device. If you do not add - p, the device name similar to dev253-0 will be displayed, so adding - p is more direct

#tps: total number of I/O transfers per second

#rd_sec/s total number of sectors read per second

#wr_sec/s total number of sectors written per second

#Avgrq SZ average data size per disk I/O operation (sector)

#The average length of avgqu-sz disk request queue

#The average elapsed time of each request, including the waiting time of the request queue, from the request disk operation to the completion of the system processing. The unit is milliseconds (1 second equals 1000 milliseconds), which is equal to the seek time + queue time + service time

#The service processing time of svctm I/O, i.e. excluding the time in the request queue

#%The percentage of CPU occupied by util I/O requests. The higher the value is, the slower I/O is

7. Statistics of process, inode, file and lock table status

[root@node1 ~]@ sar -v Linux 3.10.0-957.el7.x86_64 (k8s.node2) 06/26/2020 _x86_64_ (1 CPU) 08:00:01 AM dentunusd file-nr inode-nr pty-nr 08:10:01 AM 67612 1152 24497 2 08:20:01 AM 67637 1216 24533 2 08:30:01 AM 67630 1152 24497 2 08:40:01 AM 91215 1216 48189 2 Average: 73524 1184 30429 2

#Number of entries that dentunusd does not use in buffered directory entries

#File NR number of file handles used by the system

#Number of indexes used by inode NR

#Number of ptys used in pty NR

The index and file handle values in this are not the values viewed by ulimit-a, but sysctl.conf The kernel related values defined in max file represent the number of file handles that can be opened at the system level, while ulimit-n controls the number of file handles that can be opened at the process level, which can be viewed using sysctl-a | grep inode and sysctl-a | grep file.

File Max specifies the system wide limit on the number of file handles that can be opened by all processes (system level, kernel level). (The value in file-max denotes the maximum number of file handles that the Linux kernel will allocate). This value should have been added when the error message "Too many open files in system" is received. # cat /proc/sys/fs/file-max 4096 # echo 100000 > /proc/sys/fs/file-max perhaps # echo ""fs.file-max=65535" >> /etc/sysctl.conf # sysctl -p File NR can view the number of currently open file handles in the system. It includes three numbers: the first represents the number of allocated file descriptors, the second represents the number of idle file handles, and the third represents the maximum value (consistent with file max) that can open file handles. The kernel will dynamically allocate file handles, but it will not release them again (this may not be applicable to the latest kernel. In my file NR, the second column is always 0, and the first column is increased or decreased) man bash, find the section describing ulimit: provide control over the available resources (including file handle, number of processes, core file size, etc.) of the shell and the processes it starts. This is process level, that is, a session in the system and its start

8. Statistical network information

The sar-n option uses six different switches : DEV, EDEV, NFS, NFSD, SOCK, IP, EIP, ICMP, EICMP, TCP, ETCP, UDP, SOCK6, IP6, EIP6, ICMP6, EICMP6 and UDP6, DEV displays network interface information, EDEV displays statistics about network errors, NFS statistics about active NFS clients, NFSD statistics about NFS servers, SOCK displays socket information, ALL displays ALL five switches. They can be used alone or together.

Sar-n dev 11: Statistics once every 1 second, once in total. The following average is the average value after multiple statistics

#Name of IFACE local network interface

#rxpck/s packets accepted per second

#txpck/s database sent per second

#rxKB/S packet size accepted per second, in kilobytes

#txKB/S the size of packets sent per second, in kilobytes

#rxcmp/s compressed packets per second

#txcmp/s compressed packets per second

#rxmcst/s received multicast packets per second

sar -n EDEV 11 ා statistics network equipment communication failure information:

#IFACE network card name

#rxerr/s received corrupted packets per second

#txerr/s number of packet errors sent per second

#coll/s when sending packets, the number of collisions per second is only available in half duplex mode

#rxdrop/s the number of network packets lost per second by the receiving end of the network card device when the buffer is full

#txdrop/s the number of network packets lost per second by the network device sender when the buffer is full

#txcarr/s the number of carrier errors per second when sending packets

#rxfram the number of frame to frame errors per second when receiving packets

#rxfifo number of buffer overflow errors per second while receiving packets

#txfifo the number of buffer overflow errors per second when a packet occurs

Sar-n sock 1 1 Statistics socket connection information

#Total number of socket s currently used by totsche

#The total number of TCP socket s currently being used by tcpsck

#The total number of UDP socket s currently being used by udpsck

#Total number of skickets that rawsck is currently using for RAW

#If frag current number of IP slices

#Number of connections in TIME-WAIT state in TCP TW TCP socket

########If you use the FULL keyword, it is equivalent to the combination of DEV, EDEV and SOCK

Statistics of sar-n TCP 13 × TCP connections

#active/s new active connection

#passive/s new passive connection

#Segment accepted by iseg/s

#Segment of oseg/s output

Summary of sar-n use

-n DEV: network interface statistics. -n EDEV: network interface error. -n IP: IP datagram statistics. -n EIP: IP error statistics. -n TCP: TCP statistics. -n ETCP: TCP error statistics. -n SOCK: socket usage. sar command summary

(1) sar -b 5 5 // IO transfer rate (2) sar -B 5 5 // Page exchange rate (3) sar -c 5 5 // Rate of process creation (4) sar -d 5 5 // Active information for block devices (5) sar -n DEV 5 5 // Status information of network devices (6) sar -n SOCK 5 5 // Use of SOCK (7) sar -n ALL 5 5 // All network status information (8) sar -P ALL 5 5 // Usage status information and IOWAIT statistics status of each CPU (9) sar -q 5 5 // The length of the queue (the number of processes waiting to run) and the state of the load (10) sar -r 5 5 // Memory and swap space usage (11) sar -R 5 5 // Memory statistics (allocation and release of memory pages, memory pages used by the system as BUFFER per second, memory pages cache d to per second) (12) sar -u 5 5 // CPU usage and IOWAIT information (same as default monitoring) (13) sar -v 5 5 // inode, file and other kernel tablesd status information (14) sar -w 5 5 // Number of context exchanges per second (15) sar -W 5 5 // Statistical information exchanged by SWAP (the monitoring status is the same as that of iostat's si so) (16) sar -x 2906 5 5 // Display statistics of the specified process (2906), including errors caused by the process, CPU usage of user level and system level users, and which CPU is running on (17) sar -y 5 5 // TTY device activity status (18) Export to file(-o)And read record information(-f)