12, Regular expression
1. Introduction to regular expressions
Regular expressions are a form of text pattern matching, including ordinary characters (for example, letters between a and z) and special characters (called "metacharacters"). It is a pattern of string matching, which can be used to check whether a string contains a substring, replace the matching substring, or take the substring of a condition from a string.
Regular expressions are like mathematical formulas. We can generate a formula matching the corresponding string through some special characters provided by regular expressions, so as to match the data we want from massive data.
Regular expression is a third-party product, which is widely used by common computer languages, such as shell, PHP, python, java, js, etc! The shell also supports regular expressions, but not all commands support regular expressions. Among common commands, only grep, sed and awk commands support regular expressions.
Test data: #cat file ac ab abbc abcc aabbcc abbbc abbbbbc acc abc asb aa bb a_c aZc aAAAAc a c ABC ccc dddd http://www abababab c c d 123 a3c e*f
2. Special characters
Locator
Techniques for using locators: anchor the beginning and end at the same time to make accurate matching; Single anchor the beginning and end for fuzzy matching.
Locatorexplain^Anchor start ^ a starts with a. the default anchor is one character$Anchor end a $ends with A. anchor one character by defaultExamples are as follows:
1)Exact match to a start c a null-terminated string ,among egrep==grep -E [root@www ~]# egrep "^ac$" file ac 2)Fuzzy matching to a start [root@www ~]# egrep "^a" file ac ab abbc abcc aabbcc abbbc abbbbbc acc abc asb aa a_c aZc aAAAAc a c abababab a3c 3)Fuzzy matching to c a null-terminated string [root@www ~]# egrep "c$" file ac abbc abcc aabbcc abbbc abbbbbc acc abc a_c aZc aAAAAc a c ccc a3c
Match character
Match characterexplain.Matches any character except carriage return()String grouping[]Defines a character class that matches a character in parentheses[^]Indicates that the character in the character class in the negative bracket is reversed.\Escape character|orTest case:
1)Exact match to a start c A string with an arbitrary length of three bytes at the end [root@www ~]# egrep "^a.c$" file acc abc a_c aZc a c a3c 2)Fuzzy matching to cc Ending string because $Only a single character can be anchored. If it is a string, it needs to be used()To define [root@www ~]# egrep "(cc)$" file abcc aabbcc acc ccc 3)Exact match to a start c In the middle of the end is a-z,0-9 A three byte string [root@www ~]# egrep "^a[a-z0-9]c$" file acc abc a3c 4)Exact match to a start c The middle of the end does not contain a-z,0-9 A three byte string [root@www ~]# egrep "^a[^a-z0-9]c$" file a_c aZc a c 5)Exact match to e start f In the middle of the end is*A string with a length of three bytes e*f [root@www ~]# egrep "^e\*f$" file e*f 6)Exact match to a start b or c In the middle of the end is a string of any length of three bytes [root@www ~]# egrep "^a.(b|c)$" file acc abc asb a_c aZc a c a3c
qualifier
qualifierexplain*An asterisk after a character indicates that the character does not appear or appears more than once？Similar to the asterisk, but slightly changed, indicating that the character appears once or does not appear+Similar to the asterisk, it indicates that the character before it appears one or more times, but it must appear onceAppears after a character, indicating that the character has at least N times and at most m times, [n,m]It happened m timesTest case
1)Exact match to a start c In the middle of the end is b Or not b Unlimited length string [root@www ~]# egrep "^ab*c$" file ac abbc abbbc abbbbbc abc 2)Exact match to a start c Only once in the middle of the end b Or not b String of [root@www ~]# egrep "^ab?c$" file ac abc 3)Exact match to a start c In the middle of the end is b And an unlimited length string appears at least once [root@www ~]# egrep "^ab+c$" file abbc abbbc abbbbbc abc 4)Exact match to a start c In the middle of the end is b And at least twice and up to four times of unlimited length string [root@www ~]# egrep "^abc$" file abbc abbbc 5)Exact match to a start c In the middle of the end is b A string that appears exactly three times [root@www ~]# egrep "^abc$" file abbbc 6) Exact match to a start c In the middle of the end is b A string that appears at least once [root@www ~]# egrep "^abc$" file abbc abbbc abbbbbc abc
3.POSIX special characters
POSIX special charactersexplain[:alnum:]Match any alphabetic character 0-9 a-z A-Z[:alpha:]Matches any letter, uppercase or lowercase[:digit:]Number 0-9[:graph:]Non empty character (non space control character)[:lower:]Lowercase characters a-z[:upper:]Uppercase characters A-Z[:cntrl:]Control character[:print:]Non empty characters (including spaces)[:punct:]punctuation[:blank:]Spaces and TAB characters[:xdigit:]Hexadecimal digit[:space:]All white space characters (new lines, spaces, tabs)Test case
be careful[[ ]] Meaning of double brackets: The first bracket is a match[] Match any character in brackets, the second[]Yes format such as[:digit:] 1)Exact match to a start c End middle a-zA-Z0-9 Any character string with a length of three bytes [root@www ~]# egrep "^a[[:alnum:]]c$" file acc abc aZc a3c 2)Exact match to a start c In the middle of the end is a-zA-Z Any character string with a length of three bytes [root@www ~]# egrep "^a[[:alpha:]]c$" file acc abc aZc 3)Exact match to a start c End with 0 in the middle-9 Any character string with a length of three bytes [root@www ~]# egrep "^a[[:digit:]]c$" file a3c 4)Exact match to a start c In the middle of the end is a-z Any character string with a length of three bytes [root@www ~]# egrep "^a[[:lower:]]c$" file acc abc 4)Exact match to a start c In the middle of the end is A-Z Any character string with a length of three bytes [root@www ~]# egrep "^a[[:upper:]]c$" file aZc 5)Exact match to a start c In the middle of the end is a string of non empty arbitrary characters with a length of three bytes [root@www ~]# egrep "^a[[:print:]]c$" file acc abc a_c aZc a c a3c 6)Exact match to a start c In the middle of the end is a string with a symbolic character length of three bytes [root@www ~]# egrep "^a[[:punct:]]c$" file a_c 7)Exact match to a start c Ending with a space or TAB Character string with a length of three bytes [root@www ~]# egrep "^a[[:blank:]]c$" file a c similar [root@www ~]# egrep "^a[[:space:]]c$" file a c 8)Exact match to a start c In the middle of the end is a hexadecimal character string with a length of three bytes [root@www ~]# egrep "^a[[:xdigit:]]c$" file acc abc a3c
Note: special characters and POSIX characters are two sets of characters, which can complete the required matching.
3. Cases
Case 1: matching legal IP addresses
grep '^((25[0-5]|2[0-4][[:digit:]]|[01]?[[:digit:]][[:digit:]]?).)(25[0-5]|2[0-4][[:digit:]]|[01]?[[:digit:]][[:digit:]]?)$' —color ip_base
Case 2: matching landline telephone number
egrep "^[[:graph:]]$" number |egrep "^(0[1-9][0-9][0-9]?)-[1-9][0-9]$"
13, The shell operates on files
1.sed description
sed is an external command provided in linux. It is a line (stream) editor. It is non interactive to add, delete, modify and query the file content. Users can only enter the editing command on the command line, specify the file name, and then view the output on the screen. It is essentially different from text editor.
The difference is: text editor: The editing object is a file Line editor: editing objects are lines in a file
That is, the former processes one text at a time, while the latter processes one line of text at a time. This is something we should understand and keep in mind. Otherwise, we may not understand the operation principle and essence of sed.
2.sed command
sed command
Syntax:
sed [options] '[flags]' [filename]
Command options -e script Add the command specified in the script to the command executed when processing input. There should be multiple operations in one line -f script Adds the command specified in the file to the command executed when processing input -n Suppress automatic output -i Edit file content -i.bak Create at the same time when modifying.bak Backup files. -r Use extended regular expressions ! Reverse (following the mode condition and shell (different) sed Common internal commands a Add after match i Add before match p Print d delete s Find replace c change y transformation N D P flags number Represents the pattern of new text replacement g: Represents replacing all instances of existing text with new text p: Indicates that the original content is printed w filename: Write the replacement result to the file2.1 sed internal command description
Demo instance document
[root@www ~]# cat data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog.
After adding the file content and appending the data to a certain location, use the command a
stay data1 A new line of data content is added after each line of: append data "haha" [root@www ~]# sed 'a\append data "haha"' data1 1 the quick brown fox jumps over the lazy dog. append data "haha" 2 the quick brown fox jumps over the lazy dog. append data "haha" 3 the quick brown fox jumps over the lazy dog. append data "haha" 4 the quick brown fox jumps over the lazy dog. append data "haha" 5 the quick brown fox jumps over the lazy dog. append data "haha" Create a new row after the second row to add data: append data "haha" [root@www ~]# sed '2a\append data "haha"' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. append data "haha" 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. After the second to fourth rows, a new row is opened to add data: append data "haha" [root@www ~]# sed '2,4a\append data "haha"' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. append data "haha" 3 the quick brown fox jumps over the lazy dog. append data "haha" 4 the quick brown fox jumps over the lazy dog. append data "haha" 5 the quick brown fox jumps over the lazy dog. Match string append: Found contains"3 the"Add a new line after the new line: append data "haha" [root@www ~]# sed '/3 the/a\append data "haha"' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. append data "haha" 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. //Turn on matching mode / string to match/
To add file contents, use the command i before inserting data into a location
stay data1 Insert a new row of data content before each row of: insert data "haha" [root@www ~]# sed 'i\insert data "haha"' data1 insert data "haha" 1 the quick brown fox jumps over the lazy dog. insert data "haha" 2 the quick brown fox jumps over the lazy dog. insert data "haha" 3 the quick brown fox jumps over the lazy dog. insert data "haha" 4 the quick brown fox jumps over the lazy dog. insert data "haha" 5 the quick brown fox jumps over the lazy dog. Insert data in a new row before the second row: insert data "haha" [root@www ~]# sed '2i\insert data "haha"' data1 1 the quick brown fox jumps over the lazy dog. insert data "haha" 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. Insert data in a new row before each of the second to fourth rows: insert data "haha" [root@www ~]# sed '2,4i\insert data "haha"' data1 1 the quick brown fox jumps over the lazy dog. insert data "haha" 2 the quick brown fox jumps over the lazy dog. insert data "haha" 3 the quick brown fox jumps over the lazy dog. insert data "haha" 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. Match string insert: Found contains"3 the"Insert a new line before it: insert data "haha" [root@www ~]# sed '/3 the/i\insert data "haha"' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. insert data "haha" 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog.
File content modification - replace, replace the matching content in a line with new data, and use the command s
Replace from standard output stream test Replace with text [root@www ~]# echo "this is a test" |sed 's/test/text/' this is a text take data1 Of each row in the dog Replace with cat [root@www ~]# sed 's/dog/cat/' data1 1 the quick brown fox jumps over the lazy cat. 2 the quick brown fox jumps over the lazy cat. 3 the quick brown fox jumps over the lazy cat. 4 the quick brown fox jumps over the lazy cat. 5 the quick brown fox jumps over the lazy cat. take data1 In the second line dog Replace with cat [root@www ~]# sed '2s/dog/cat/' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy cat. 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. take data1 In the second to fourth lines dog Replace with cat [root@www ~]# sed '2,4s/dog/cat/' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy cat. 3 the quick brown fox jumps over the lazy cat. 4 the quick brown fox jumps over the lazy cat. 5 the quick brown fox jumps over the lazy dog. Match string replacement:Will contain string"3 the"In rows of dog Replace with cat [root@www ~]# sed '/3 the/s/dog/cat/' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy cat. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog.
File content modification operation - change, replace the matching content in a line with new data, and use the command c
take data1 The contents of all lines in the file are changed to: change data "data" [root@www ~]# sed 'c\change data "haha"' data1 change data "haha" change data "haha" change data "haha" change data "haha" change data "haha" take data1 Change the content of the second line of the file to: change data "haha" [root@www ~]# sed '2c\change data "haha"' data1 1 the quick brown fox jumps over the lazy dog. change data "haha" 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. take data1 The contents of the second, third and fourth lines in the document are changed to: change data "haha" [root@www ~]# sed '2,4c\change data "haha"' data1 1 the quick brown fox jumps over the lazy dog. change data "haha" 5 the quick brown fox jumps over the lazy dog. take data1 File contains"3 the"Change the line content of to: change data "haha" [root@www ~]# sed '/3 the/c\change data "data"' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. change data "data" 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog.
File content modification - character conversion, replace the matching content in a line with new data, and use the command y
take data1 Medium a b c Convert character to corresponding A B C character [root@www ~]# sed 'y/abc/ABC/' data1 1 the quiCk Brown fox jumps over the lAzy dog. 2 the quiCk Brown fox jumps over the lAzy dog. 3 the quiCk Brown fox jumps over the lAzy dog. 4 the quiCk Brown fox jumps over the lAzy dog. 5 the quiCk Brown fox jumps over the lAzy dog.
Delete the file content, delete the specified data in the file, and use the command d
Delete file data1 All data in [root@www ~]# sed 'd' data1 Delete file data1 The third row of data in [root@www ~]# sed '3d' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. Delete file data1 Data in the third to fourth rows [root@www ~]# sed '3,4d' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. Delete file data1 Contains a string"3 the"Line of [root@www ~]# sed '/3 the/d' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog.
View the file content, output the file content to the screen, and use the command p
Print data1 File content [root@www ~]# sed 'p' data1 1 the quick brown fox jumps over the lazy dog. 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. Print data1 Contents of the third line of the file [root@www ~]# sed '3p' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. Print data1 The second, third and fourth lines of the document [root@www ~]# sed '2,4p' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. Print data1 The file contains a string"3 the"Line of [root@www ~]# sed '/3 the/p' data1 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog. It can be seen that the printed content is a duplicate line, because the specified file content is printed once and all the data read into the cache is printed once, so this effect will be seen, If you don't want to see such a result, you can add the command option-n Just suppress the memory output.2.2 description of command options
Use multiple commands on the command line - e
take brown Replace with green dog Replace with cat [root@www ~]# sed -e 's/brown/green/;s/dog/cat/' data1 1 the quick green fox jumps over the lazy cat. 2 the quick green fox jumps over the lazy cat. 3 the quick green fox jumps over the lazy cat. 4 the quick green fox jumps over the lazy cat. 5 the quick green fox jumps over the lazy cat.
The read editor command - f from a file is suitable for daily repeated execution scenarios
1)Write command to file [root@www ~]# vim abc s/brown/green/ s/dog/cat/ s/fox/elephant/ 2)use-f Command options call command files [root@www ~]# sed -f abc data1 1 the quick green elephant jumps over the lazy cat. 2 the quick green elephant jumps over the lazy cat. 3 the quick green elephant jumps over the lazy cat. 4 the quick green elephant jumps over the lazy cat. 5 the quick green elephant jumps over the lazy cat.
Suppress memory output - n
Print data1 The second to last lines of the file $Last meaning [root@www ~]# sed -n '2,$p' data1 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog.
Use regular expression - r
Print data1 String in"3 the"Line content at the beginning [root@www ~]# sed -n -r '/^(3 the)/p' data1 3 the quick brown fox jumps over the lazy dog.
From the above demonstration, you can see that the data processing is only completed in the cache without actually modifying the file content. If you need to modify the file content, you can directly use the - I command option. What I need to explain here is that - I is an irreversible operation. Once modified, it is very difficult and almost impossible to restore. Therefore, it is recommended that you back up the source file during operation- The I command option provides the backup function. For example, if the parameter is - i.bak, a source file ending in. Bak will be backed up before modifying the source file.
1)Check the file list and find no data1.bak [root@www ~]# ls abc apache data1 Dobby file node-v10.14.1 Python-3.7.1 soft1 vimset 2)Execute the replace command and modify the file-i.bak Turn into-i.txt [root@www ~]# sed -i.bak 's/brown/green/' data1 3)Found one more folder data1.bak file [root@www ~]# ls abc data1 Dobby node-v10.14.1 soft1 apache data1.bak file Python-3.7.1 vimset 4)Print and compare, and find data1 Has been modified, data1.bak Is a backup of the source file. [root@www ~]# cat data1 1 the quick green fox jumps over the lazy dog. 2 the quick green fox jumps over the lazy dog. 3 the quick green fox jumps over the lazy dog. 4 the quick green fox jumps over the lazy dog. 5 the quick green fox jumps over the lazy dog. [root@www ~]# cat data1.bak 1 the quick brown fox jumps over the lazy dog. 2 the quick brown fox jumps over the lazy dog. 3 the quick brown fox jumps over the lazy dog. 4 the quick brown fox jumps over the lazy dog. 5 the quick brown fox jumps over the lazy dog.2.3 signs
Presentation document [root@www ~]# cat data2 1 the quick brown fox jumps over the lazy dog . dog 2 the quick brown fox jumps over the lazy dog . dog 3 the quick brown fox jumps over the lazy dog . dog 4 the quick brown fox jumps over the lazy dog . dog 5 the quick brown fox jumps over the lazy dog . dog
Digital sign
This flag is a non-zero positive number. By default, when performing replacement, if there are multiple matching strings in a line, if there is no flag bit definition, only the first string will be replaced, and the others will be ignored. In order to accurately replace, you can use the number bit for definition.
Replace the second place in a line dog by cat [root@www ~]# sed 's/dog/cat/2' data2 1 the quick brown fox jumps over the lazy dog . cat 2 the quick brown fox jumps over the lazy dog . cat 3 the quick brown fox jumps over the lazy dog . cat 4 the quick brown fox jumps over the lazy dog . cat 5 the quick brown fox jumps over the lazy dog . cat
g mark
Replace all matching strings in a line
take data1 All in the file dog Replace with cat [root@www ~]# sed 's/dog/cat/g' data2 1 the quick brown fox jumps over the lazy cat . cat 2 the quick brown fox jumps over the lazy cat . cat 3 the quick brown fox jumps over the lazy cat . cat 4 the quick brown fox jumps over the lazy cat . cat 5 the quick brown fox jumps over the lazy cat . cat
p flag
Print text content, similar to the - p command option
[root@www ~]# sed '3s/dog/cat/p' data2 1 the quick brown fox jumps over the lazy dog . dog 2 the quick brown fox jumps over the lazy dog . dog 3 the quick brown fox jumps over the lazy cat . dog 3 the quick brown fox jumps over the lazy cat . dog 4 the quick brown fox jumps over the lazy dog . dog 5 the quick brown fox jumps over the lazy dog . dog
w filename flag
Save the modified contents into the filename file
[root@www ~]# sed '3s/dog/cat/w text' data2 1 the quick brown fox jumps over the lazy dog . dog 2 the quick brown fox jumps over the lazy dog . dog 3 the quick brown fox jumps over the lazy cat . dog 4 the quick brown fox jumps over the lazy dog . dog 5 the quick brown fox jumps over the lazy dog . dog It can be seen that the third line to be modified already exists text In the file [root@www ~]# cat text 3 the quick brown fox jumps over the lazy cat . dog
3.sed tips
$= how many lines of statistical text
Statistics data2 How many lines are there [root@www ~]# sed -n '$=' data2 5 Print data2 Add line number to content [root@www ~]# sed '=' data2 1 1 the quick brown fox jumps over the lazy dog . dog 2 2 the quick brown fox jumps over the lazy dog . dog 3 3 the quick brown fox jumps over the lazy dog . dog 4 4 the quick brown fox jumps over the lazy dog . dog 5 5 the quick brown fox jumps over the lazy dog . dog
Note: the above cases are from the UP main Python community of station B. It is better to eat them together with blogs and videos
shell script video link: Shell script