01 "shell programming prelude" widget

1, Text processing tools

1. grep tool

grep is a row filter tool; it is used to filter rows according to keywords

Syntax and options

Syntax:

#grep [options] 'keyword' filename

Common options:

OPTIONS:
    -i: Case insensitive
    -v: Find rows that do not contain the specified content,Reverse selection
    -w: Search by word
    -o: Print matching keywords
    -c: Count the number of rows matched
    -n: set number
    -r: Traverse directory lookup layer by layer
    -A: Show matching lines and how many lines follow	
    -B: Show matching lines and how many lines ahead
    -C: Show how many lines before and after matching lines
    -l: List only matching filenames
    -L: List mismatched file names
    -e: Use regular matching
    -E:Use extended regular matching
    ^key:Start with keyword
    key$:End with keyword
    ^$:Match blank line
    --color=auto : You can add color to the key words you find

Color display (alias settings):

Temporary settings:
#alias grep='grep --color=auto '/ / only valid for current terminal and current user

Permanent settings:
1) Global (effective for all users)
vim /etc/bashrc
alias grep='grep --color=auto'
source /etc/bashrc

2) Local (for a specific user)
vim ~/.bashrc
alias grep='grep --color=auto'
source ~/.bashrc

For example:

Note: do not directly use / etc/passwd file, copy it to / tmp for experiment!

#grep -i root passwd ignore case matching row with root
 #Grep-w ftp passwd exactly matches ftp words
 #grep -w hello passwd exactly matches Hello words; add lines containing hello to the file
 #grep -wo ftp passwd print the matching keyword ftp
 #grep -n root passwd print the line that matches the root keyword
 #grep -ni root passwd ignore case matching statistics row containing key root
 #grep -nic root passwd ignore case matching count the number of rows containing the keyword root
 #grep -i ^root passwd ignore case matching lines beginning with root
 #Grep bash $passwd matches lines ending in Bash
 #Grep - n ^ $passwd matches empty lines and prints line numbers
 #grep ^# /etc/vsftpd/vsftpd.conf matches lines beginning with a #
#Grep - V ^ / etc / vsftpd / vsftpd.conf matches lines that do not begin with a #
#Grep-a 5 mail passwd match contains the mail keyword and its last 5 lines
 #Grep-b 5 mail passwd match contains the mail keyword and its first 5 lines
 #Grep-c 5 mail passwd match contains the mail keyword and 5 lines before and after it

2. cut tools

cut is the * * column * * interception tool, which is used for column interception

Syntax and options

Syntax:

#cut options file name

Common options:

-c: Split and intercept characters
 -d: Custom separator, default is tab \ t
 -f: Use with - d to specify which area to intercept

For example:

#cut -d: -f1 1.txt to: colon to intercept column 1
 #cut -d: -f1,6,7 1.txt split with: colon, intercept the contents of the first, sixth and seventh columns
 #cut -c4 1.txt intercepts the fourth character in each line of the file
 #cut -c1-4 1.txt intercepts 1-4 characters per line in the file
 #cut -c4-10 1.txt intercepts 4-10 characters per line in the file
 #cut -c5- 1.txt starts from the 5th character and intercepts all the following characters

Classroom exercises:
Use gadgets to list the level of operation of your system. 5/3

  1. How to view system operation level
    • Command runlevel
    • File / etc/inittab
  2. How to filter run level
runlevel |cut -c3
runlevel | cut -d ' ' -f2
grep -v '^#' /etc/inittab | cut -d: -f2
grep '^id' /etc/inittab |cut -d: -f2
grep "initdefault:$" /etc/inittab | cut -c4
grep -v ^# /etc/inittab |cut -c4
grep 'id:' /etc/inittab |cut -d: -f2
cut -d':' -f2 /etc/inittab |grep -v ^#
cut -c4 /etc/inittab |tail -1
cut -d: -f2 /etc/inittab |tail -1

3. sort tool

The sort tool is used for sorting; it takes each line of a file as a unit, compares them by ASCII value from the first character back, and finally outputs them in ascending order.

Syntax and options

-u: Remove duplicate lines
 -r: Descending, default is ascending
 -o: output the sorting results to a file, similar to the redirection symbol >
-n: Sort by number, the default is sort by character
 -t: Separator
 -k: Column N
 -b: Ignore leading spaces.
-R: Random sorting, the results of each run are different

Illustrate with examples

# sort -n -t: -k3 1.txt
# sort -nr -t: -k3 1.txt
# sort -n 2.txt sort by number
# sort -nu 2.txt sort by number and de duplicate
# sort -nr 2.txt 
# sort -nru 2.txt 
# sort -nru 2.txt 
# sort -n 2.txt -o 3.txt sort by number and redirect results to a file
# sort -R 2.txt 
# sort -u 2.txt 

##4.uniq tools

uniq is used to remove * * consecutive duplicate * * lines

Common options:
-i: Ignore case
 -c: Count the number of duplicate lines
 -d: Show only duplicate lines

For example:
# uniq 2.txt 
# uniq -d 2.txt 
# uniq -dc 2.txt 

5.tee tools

The tee tool reads and writes from standard input to standard output and files, that is, bidirectional override redirection (screen output | text input)

Options:
-a Two way append redirection

# echo hello world
# echo hello world|tee file1
# cat file1 
# echo 999|tee -a file1
# cat file1 

6.diff tools

diff tool is used to compare file differences line by line

Note: diff describes the difference between two files in a way that tells us how to change the first file to match the second.

Syntax and options

Syntax:

diff [options] file1file2

Common options:

option Meaning Remarks
-b Do not check spaces
-B Do not check blank lines
-i Do not check case
-w Ignore all spaces
–normal Normal format display (default)
-c Context format display
-u Merge format display

For example:

  • Compare the similarities and differences between the two common documents, document preparation:
[root@MissHou ~]# cat file1
aaaa
111
hello world
222
333
bbb
[root@MissHou ~]#
[root@MissHou ~]# cat file2
aaa
hello
111
222
bbb
333
world

1) Normal display

diffObjective: file1 How to change talents and file2 matching
[root@MissHou ~]# diff file1 file2
1c1,2					Line 1 of the first file needs to be changed(c=change)To match lines 1 to 2 of the second file			
< aaaa				Less than"<"Represents the file on the left(file1)Document content
---					---Represents a separator
> aaa					Greater than">"Represents the file on the right(file2)Document content
> hello
3d3					Delete line 3 of the first file(d=delete)To match the third line of the second file
< hello world
5d4					Line 5 of the first file can't match line 4 of the second file until it's deleted
< 333
6a6,7					Add line 6 of the first file(a=add)Content before matching lines 6 to 7 of the second file
> 333					What needs to be added in the second file is 333 and world
> world

2) Context format display

[root@MissHou ~]# diff -c file1 file2
 The first two lines mainly list the file name to be compared and the time stamp of the file; the symbol before the file name * * * indicates file1, --- indicates file2
*** file1       2019-04-16 16:26:05.748650262 +0800
--- file2       2019-04-16 16:26:30.470646030 +0800
 ***************I'm the separator
 ***1,6 * * * * starts with * * * for file1 file, 1,6 for lines 1 to 6
 ! aaaa! Indicates that the line needs to be modified to match the second file
  111
 -hello world - indicates that the line needs to be deleted to match the second file
  222
 -333 - indicates that the line needs to be deleted to match the second file
  bbb
 ---1,7 -- file2 file starts with ---, 1,7 lines from 1 to 7
 ! aaa means the first file needs to be modified to match the second
 ! hello means the first file needs to be modified to match the second
  111
  222
  bbb
 +333 indicates that the first file needs to add this line to match the second file
 +world means that the first file needs to add this line to match the second file

3) Merge format display

[root@MissHou ~]# diff -u file1 file2
 The first two lines mainly list the file name to be compared and the time stamp of the file; the symbol in front of the file name --- indicates file1, + + + indicates file2
--- file1       2019-04-16 16:26:05.748650262 +0800
+++ file2       2019-04-16 16:26:30.470646030 +0800
@@ -1,6 +1,7 @@
-aaaa
+aaa
+hello
 111
-hello world
 222
-333
 bbb
+333
+world
  • Compare two different directories
By default, the contents of the same files in the two directories are also compared
[root@MissHou  tmp]# diff dir1 dir2
diff dir1/file1 dir2/file1
0a1
> hello
Only in dir1: file3
Only in dir2: test1
//If you only need to compare the differences between the files in the two directories, you do not need to further compare the contents of the files-q option
[root@MissHou  tmp]# diff -q dir1 dir2
Files dir1/file1 and dir2/file1 differ
Only in dir1: file3
Only in dir2: test1

Other tips:

Sometimes we need to use one file as the standard to modify other files. When there are many modifications, we can patch them.

1)First find out the differences between files, and then output to a file
[root@MissHou ~]# diff -uN file1 file2 > file.patch
-u:Context mode
-N:Treat nonexistent files as empty files
2)Patch different content to file
[root@MissHou ~]# patch file1 file.patch
patching file file1
3)Test verification
[root@MissHou ~]# diff file1 file2
[root@MissHou ~]#

7. paste tool

paste tool is used to merge file lines

Common options:
-d: Custom spacer, default is tab
 -s: Serial processing, non parallel

##8. tr tools

tr is used for character conversion, replacement and deletion; mainly used for deleting control characters in files or character conversion

Syntax and options

Syntax:

Usage 1: the execution result of the command is handed over to tr for processing, in which string1 is used for query and string2 is used for transformation
# commands|tr  'string1'  'string2'
Usage 2: tr processes content from a file, remember to use "<" standard input
# tr  'string1'  'string2' < filename

Usage 3: match string1 for corresponding operation, such as deletion
# tr options 'string1' < filename

Common options:

-d delete all input characters in string 1.
-s delete all repeated character sequences and keep only the first one; that is, compress the repeated character strings into one string

Constant matching string:

Character string Meaning Remarks
a-z or [: lower:] Match all lowercase letters [a-zA-Z0-9]
A-Z or [: upper:] Match all uppercase letters
0-9 or [: Digital:] Match all numbers
[:alnum:] Match all letters and numbers
[:alpha:] Match all letters
[:blank:] All horizontal blanks
[:punct:] Match all punctuation
[:space:] All horizontal or vertical spaces
[:cntrl:] All control characters \f Ctrl-L line feed
\n Ctrl-J newline

\r Ctrl-M enter
\t Ctrl-I tab|

For example:

[root@MissHou  shell01]# cat 3.txt create the file for testing
ROOT:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
boss02:x:516:511::/home/boss02:/bin/bash
vip:x:517:517::/home/vip:/bin/bash
stu1:x:518:518::/home/stu1:/bin/bash
mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin
smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin
aaaaaaaaaaaaaaaaaaaa
bbbbbb111111122222222222233333333cccccccc
hello world 888
666
777
999


# TR - D '[: /]' < 3.txt delete: and/
# Cat 3.txt|tr - D '[: /]' delete: and/
# TR '[0-9]' @ '< 3.txt replace the number in the file with the @ symbol
# TR '[A-Z]' [A-Z] '< 3.txt replace lowercase letters with uppercase letters in the file
# TR - s' [A-Z] '< 3.txt match lowercase letters and compress duplicates into one
# TR - s' [a-z0-9] '< 3.txt match lowercase letters and numbers and compress duplicates into one
# TR - D '[: digit:]' < 3.txt delete the number in the file
# TR - D '[: Blank:]' < 3.txt delete horizontal blank
# TR - D '[: Space:]' < 3.txt delete all horizontal and vertical blanks

Small test knife

  1. Use the gadget to intercept the IP address of the current host; intercept NETMASK; intercept broadcast address; intercept MAC address
# ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4
10.1.1.1:10.1.1.255:255.255.255.0
# ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4|tr ':' '\n'
10.1.1.1
10.1.1.255
255.255.255.0
# ifconfig eth0|grep 'HWaddr'|cut -d: -f2-|cut -d' ' -f4
00:0C:29:25:AE:54
# ifconfig eth0|grep 'HW'|tr -s ' '|cut -d' ' -f5
00:0C:29:B4:9E:4E

# ifconfig eth1|grep Bcast|cut -d: -f2|cut -d' ' -f1
# ifconfig eth1|grep Bcast|cut -d: -f2|tr -d '[ a-zA-Z]'
# ifconfig eth1|grep Bcast|tr -d '[:a-zA-Z]'|tr ' ' '@'|tr -s '@'|tr '@' '\n'|grep -v ^$
# ifconfig eth0|grep 'Bcast'|tr -d [:alpha:]|tr '[ :]' '\n'|grep -v ^$
# ifconfig eth1|grep HWaddr|cut -d ' ' -f11
# ifconfig eth0|grep HWaddr|tr -s ' '|cut -d' ' -f5
# ifconfig eth1|grep HWaddr|tr -s ' '|cut -d' ' -f5
# ifconfig eth0|grep 'Bcast'|tr -d 'a-zA-Z:'|tr ' ' '\n'|grep -v '^$'
  1. Save the user name, password and default shell of all ordinary users in the system to a file, which requires the user name, password and default shell to be separated by tab key
# grep 'bash$' passwd |grep -v 'root'|cut -d: -f1,2,7|tr ':' '\t' |tee abc.txt

2, Characteristics of bash

##1. Command and file auto completion

Tab can only complete commands and files (RHEL6/Centos6)

##2. Common shortcut keys

^c. terminate the program running at the front desk
 ^z. suspend programs running in the foreground to the background
 ^d. exit equivalent
 ^l. clear screen 
^Move the a |home cursor to the front of the command line
 ^Move the e |end cursor to the back of the command line
 ^u delete all characters before cursor
 ^k delete all characters after cursor
 ^r search history command

##3. Common wildcards (key)

*:	Match 0 or more arbitrary characters
?:	Match any single character
[list]:	matching[list]Any single character in,Or a set of single characters   [a-z]
[!list]: Matching elimination list Any single character in
{string1,string2,...}: matching string1,string2 Or more strings


# rm -f file*
# cp *.conf  /dir1
# touch file{1..5}

##4. Quotes in bash (emphasis)

  • Double quotation mark "": the contents of quotation marks will be regarded as a whole, and other variable values can be referenced through the $symbol
  • Single quotation mark '': the contents of the quotation mark will be regarded as a whole. It is forbidden to refer to other variable values. Special symbols in the shell are regarded as common characters
  • Apostrophe ` `: like $(), apostrophes take precedence over commands in quotation marks or brackets. If nesting exists, apostrophes cannot be used
[root@MissHou  dir1]# echo "$(hostname)"
server
[root@MissHou  dir1]# echo '$(hostname)'
$(hostname)
[root@MissHou  dir1]# echo "hello world"
hello world
[root@MissHou  dir1]# echo 'hello world'
hello world

[root@MissHou  dir1]# echo $(date +%F)
2018-11-22
[root@MissHou  dir1]# echo `echo $(date +%F)`
2018-11-22
[root@MissHou  dir1]# echo `date +%F`
2018-11-22
[root@MissHou  dir1]# echo `echo `date +%F``
date +%F
[root@MissHou  dir1]# echo $(echo `date +%F`)
2018-11-22
Published 115 original articles, won praise 3, visited 4421
Private letter follow

Tags: ftp vsftpd shell vim

Posted on Fri, 13 Mar 2020 00:18:24 -0400 by liquidmind