Linux input and output: redirection, process pipeline technology

1. Redirection overview

1. What is redirection

Redirect the data information originally to be output to the screen to a specified file. For example, backup data regularly every morning. You want to save the results of the backup data to a file. In this way, you can know whether the data backed up yesterday was successful or failed by viewing the contents of the file the next day.

2. Why use redirection

1. When the information output from the screen is very important and you want to save important information;

2. When the program being executed in the background does not want it to interfere with the normal output results of the screen;

3. Routine commands of the system, such as the execution results of scheduled tasks, are expected to be saved;

4. When we know that there may be error messages in some execution orders and want to discard them directly;

5. When the error log and the correct log need to be output to different files for saving;

3. Learn the preliminary knowledge of redirection, standard input and output

When running a program, it usually automatically opens three standard files: standard input, standard output and error output

nameFile descriptoreffect
Standard input (STDIN)0The default is the keyboard, which can also be the output of files or other commands.
Standard output (STDOUT)1Default output to screen.
Error output (STDERR)2Default output to screen.
File name3+

The process will get the data from the standard input, print the normal output to the screen terminal, and print the wrong output information to the screen terminal.

PS: the process uses file descriptors to manage open files

Taking cat command as an example, the function of cat command is to read data from the file given by the command line and send these data directly to the standard output. If you use the following command:

#The contents of the file / etc/passwd will be output and displayed on the screen
[root@lqz ~]# cat /etc/passwd

However, if the cat command does not keep up with the input file name, the cat command will read the data from the command line standard input and send it to the standard output.

[root@lqz ~]# cat
hello   #Standard input
hello   #standard output 
^C
#Each line entered by the user is immediately output to the screen by the cat command.

Let's take a look at the standard I / O process

#Continuously track and view file content
[root@lqz ~]# tail -f /etc/passwd
ctrl+z Move process to background

#View running processes
[root@lqz ~]# ps
PID TTY          TIME CMD
5848 pts/1    00:00:00 bash
6885 pts/1    00:00:00 tail
6888 pts/1    00:00:00 ps

#Check the pid of the tail command and the file descriptor under the 6885 process
[root@lqz ~]# ls -l /proc/6885/fd
total 0
lrwx------ 1 root root 64 Dec  3 06:57 0 -> /dev/pts/1
lrwx------ 1 root root 64 Dec  3 06:57 1 -> /dev/pts/1
lrwx------ 1 root root 64 Dec  3 06:56 2 -> /dev/pts/1
lr-x------ 1 root root 64 Dec  3 06:57 3 -> /etc/passwd
lr-x------ 1 root root 64 Dec  3 06:57 4 -> inotify

#Linux viewing standard I / O devices
[root@lqz ~]# ls -l /dev/std
lrwxrwxrwx 1 root root 15 Dec  2 22:30 /dev/stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Dec  2 22:30 /dev/stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Dec  2 22:30 /dev/stdout -> /proc/self/fd/1

2. Output redirection

Output redirection, changing the position of output content. Output redirection can be done in the following ways, as shown in the table

typeOperatorpurpose
Standard override output redirection>Outputting the correct result of the program output to the specified file will overwrite the original content of the file
Standard append output redirection>>The correct results output by the program are output to the specified file in the form of addition, and the original file will not be overwritten
Error overwriting output redirection2>Outputting the error result of the program to the executed file will overwrite the original content of the file
Error: append output redirection2>>The error result output by the program is output to the specified file in the form of append, and the original file will not be overwritten
Standard input redirection<<Change the way of receiving input in the command from the default keyboard to the specified file or command

Case 1: standard output redirection (overwriting files every time)

#Standard output redirection, which is emptied first and then written. If the file does not exist, it is created
[root@lqz ~]# ifconfig eth0 > abc

Case 2: standard output redirection (content will be added at the end of the file)

#Standard output redirection, appending content to the end of the configuration file
[lqz@lqz ~]$ echo "This is network conf" >> if

Case 3: error output redirection

#Correct output and error output are redirected to a file
[root@lqz ~]# useradd lqz
[root@lqz ~]# su - lqz

#Redirect standard output and standard error output to different files
[lqz@lqz ~]$ find /etc -name ".conf" 1>a 2>b

Case 4: correct and error are entered in the same location

#Redirect standard output and standard error output to the same file and mix output
[lqz@lqz ~]$ find /etc -name ".conf" &>ab

#Merge the contents of two files into one file
[lqz@lqz ~]$ cat a b > c

Case 5: correct and error are entered in the same location

#Redirect to the same location
[root@lqz ~]# ls /root /error >ab  2>&1

Case 6: redirect to empty device / dev/null

#Any data generated into the black hole device is considered discarded.
[root@lqz ~]# ls /root /error >ab 2>/dev/null
[root@lqz ~]# ls /root /error >ab &>/dev/null

Case 7: using redirection in script (just understand)

[root@lqz ~]# vim ping.sh 
ping -c1 10.0.0.1
if [ $? -eq 0 ];then
    echo "10.0.0.1 is up." 
else
    echo "10.0.0.1 is down." 
fi
[root@lqz ~]# chmod +x ping.sh 
[root@lqz ~]# ./ping.sh

#Improved version
[root@lqz ~]# vim ping.sh
ping -c1 10.0.0.1 &>/dev/null
if [ $? -eq 0 ];then
    echo "10.0.0.1 is up." 
else
    echo "10.0.0.1 is down." 
fi

Case 8: using redirection in script (just understand)

[root@lqz ~]# vim ping2.sh 
ping -c1 10.0.0.1 &>/dev/null 
if [ $? -eq 0 ];then
    echo "10.0.0.1 is up." >>up.txt 
else
    echo "10.0.0.1 is down." >>down.txt 
fi
[root@lqz ~]# chmod +x ping2.sh 
[root@lqz ~]# ./ping2.sh

3. Enter redirection

Input redirection, that is, the input information originally obtained from the keyboard, is input by the output of the command< Equivalent 0<
Case 1: read input from file

#The default keyboard does not change the direction of input
[root@lqz ~]# mail alice 
Subject: hello
1111 
2222
3333
.   #end
EOT

#Check whether the mail is received
[root@lqz ~]# su - alice
[root@lqz ~]# mail

#Enter redirection from file
[root@lqz ~]# mail -s "test01" alice < /etc/hosts

Case 2: unspeakable case, please see the actual operation

#If the input direction is not changed, the default keyboard is used. At this time, wait for input
[root@lqz ~]# grep 'root' 
xxx
xxx

[root@lqz ~]# grep 'root' < /etc/passwd
root:x:0:0:root:/root:/bin/bash

Case 3: unspeakable case, please see the actual operation

[root@lqz ~]# dd if=/dev/zero of=/file1.txt bs=1M count=20
[root@lqz ~]# dd </dev/zero >/file2.txt bs=1M count=20

Case 4: you can understand how MySQL restores backups without paying attention.

[root@lqz ~]# mysql -uroot -p123 < bbs.sql

Case 5: using redirection to create a file with multiple lines of data

#Execute shell commands manually
[root@lqz ~]# echo "111" > file1.txt 
[root@lqz ~]# cat file1.txt
111
[root@lqz ~]# cat >file2.txt
111
222
333
^D

[root@lqz ~]# cat >>file3.txt
aaa
bbb
ccc
^D

Case 6: a method of using the print menu in the script.

[root@lqz ~]# vim vm.sh
cat <<-EOF
+------------------- --- ---- --- ---- --- --- ---- --- --+ ||
| ====================== | 
| Virtual machine basic management v5.0 |
| by lqz |
| ====================== | 
| 1. install KVM |
| 2. Install or reset CentOS-6.9 | 
| 3. Install or reset CentOS-7.4 | 
| 5. Install or reset Windows-7  | 
| 6. Delete all virtual machines |
| q. Exit management program |
+------------------- --- ---- --- ---- --- --- ---- --- --+ 
EOF

Case 7: simultaneous redirection of two commands

[root@lqz ~]# ls; date &>/dev/null
[root@lqz ~]# ls &>/dev/null; date &>/dev/null
[root@lqz ~]# (ls; date) &>/dev/null

#Background execution
[root@lqz ~]# (while :; do date; sleep 2; done) &
[1] 6378
[root@lqz ~]# (while :; do date; sleep 2; done) &>date.txt &
[root@lqz ~]# jobs
[1]+ In operation ( while :; do date; sleep 2;
done ) &>/date.txt &

Extension point: subshell can understand it

[root@lqz ~]# cd /boot; ls

//Execute in subshell
[root@lqz ~]# (cd /boot; ls)

#If you do not want the execution of some commands to affect the current shell environment, execute them in the subshell

4. Process pipeline technology

1. What is pipeline

The pipe operation symbol "|" is mainly used to connect the left and right commands, and hand over the standard output of the command on the left to the standard input of the command on the right

PS: unable to pass standard error output to the latter command

2. Pipeline flow diagram

Format: cmd1 | cmd2 [...|cmdn]

3. Pipeline use cases

Case 1: sort users in / etc/passwd by UID size

[root@lqz ~]# sort -t":" -k3 -n /etc/passwd
[root@lqz ~]# sort -t":" -k3 -n /etc/passwd -r
[root@lqz ~]# sort -t":" -k3 -n /etc/passwd |head

Case 2: count the shell types used by users in the current / etc/passwd

#Idea: take out the seventh column (shell) | sort (classify the same) | remove duplication
[root@lqz ~]# awk -F: '{print $7}' /etc/passwd
[root@lqz ~]# awk -F: '{print $7}' /etc/passwd |sort
[root@lqz ~]# awk -F: '{print $7}' /etc/passwd |sort |uniq
[root@lqz ~]# awk -F: '{print $7}' /etc/passwd |sort |uniq -c

Case 4: Statistics of website visits top 20

#Idea: print all the accessed connections | filter the connections to the website | print the user's IP | sort | de duplicate

[root@lqz ~]# yum -y install httpd
[root@lqz ~]# systemctl start httpd
[root@lqz ~]# systemctl stop firewalld

[root@lqz ~]# ss -an |grep :80 |awk -F":" '{print $8}' |sort |uniq -c
[root@lqz ~]# ss -an |grep :80 |awk -F":" '{print $8}' |sort |uniq -c |sort -k1 -rn |head -n 20

Case 5: print all current IP addresses

[root@lqz ~]# ip addr |grep 'inet ' |awk '{print $2}' |awk -F"/" '{print $1}'
127.0.0.1
192.168.69.112

Case 6: print the percentage of space used in the root partition (print numbers only)

[root@lqz ~]# df |grep '/$' |awk '{print $5}' |awk -F"%" '{print $1}'

PS: the pipeline command allows you to further master the collocation between commands and further improve the processing efficiency of command output value.

4. tee technology in pipeline

#Option: - a append
[root@lqz ~]# ip addr |grep 'inet ' |tee ip.txt |awk -F"/" '{print $1}' |awk '{print $2}'
127.0.0.1
10.0.0.100

[root@lqz ~]# cat ip.txt
inet 127.0.0.1/8 scope host lo
inet 10.0.0.100/24 brd 192.168.69.255 scope global ens32

What is the difference between redirection and tee

[root@lqz ~]# date > date.txt    #Write the contents directly to the date.txt file
[root@lqz ~]# date |tee date.txt #The command execution will be output to the screen, but a copy will be saved to the date.txt file at the same time

5.xargs parameter passing mainly enables some commands that do not support pipeline to use pipeline technology

# which cat|xargs ls- l
# ls |xargs rm -fv
# ls |xargs cp -rvt /tmp / - or - > LS | xargs - I {} CP - RV {} / tmp/
# ls |xargs mv -t /tmp / - or - > LS | xargs - I {} MV {} / tmp

ate.txt # directly writes the contents to the date.txt file
[ root@lqz ~]#The date | tee date.txt # command will be output to the screen, but a copy will be saved to the date.txt file at the same time

5.xargs Parameter passing mainly enables some commands that do not support pipeline to use pipeline technology
```python
# which cat|xargs ls- l
# ls |xargs rm -fv
# ls |xargs cp -rvt /tmp / - or - > LS | xargs - I {} CP - RV {} / tmp/
# ls |xargs mv -t /tmp / - or - > LS | xargs - I {} MV {} / tmp

Tags: Python Linux Operation & Maintenance

Posted on Mon, 06 Sep 2021 15:40:46 -0400 by mr_zhang