38, SHELL programming

1, introduction

SHELL is a software program developed based on C language. It is placed in the outer layer of the Linux kernel by default. After the Linux system is started, a layer of SHELL will be loaded in the outer layer of the Linux kernel. This SHELL is called SHELL. SHELL shell is located between the user and the Linux kernel. It is mainly used to receive the instructions input by the user, parse the commands, and finally send the parsed instructions to the Linux kernel. After the Linux kernel finishes processing, it will return the processed results to SHELL, which will interpret (translate) the returned data of the Linux kernel, and finally return the translated information For users. After the Linux operating system starts, log in with the user and password. By default, it will log in to the SHELL terminal. All the user's operations are performed in the SHELL terminal.
The common SHELL interpreter software is as follows:
Bourne Shell (/ usr/bin/sh or / bin/sh)
Bourne Again Shell(/bin/bash)
C Shell(/usr/bin/csh)
K Shell(/usr/bin/ksh)
Shell for Root(/sbin/sh)

2, variables

Shell programming is a non type interpreted language, unlike C + +, JAVA programming, which requires variables to be declared in advance. When shell assigns a value to a variable, it actually defines the variable. In all shells supported by Linux, you can assign values to variables with the assignment symbol (=). Shell variable is of weak type. Defining variable does not need to Declare type, but the type of variable needs to be specified when using. You can use Declare to specify the type. Common parameters of Declare are:

+/-"-" can be used to specify the properties of the variable, "+" is the property set to cancel the variable
 -f show functions only
 r set variable to read-only
 The variables specified by x become environment variables, which can be used by programs other than the shell
 i specifies that the type is numeric, string, or expression

There are three kinds of variables in Shell programming: system variable, environment variable and user variable. The system variable is used for parameter judgment and command return value judgment, while the environment variable is mainly set when the program is running. The user variable is also called local variable, which is mostly used in Shell script or temporary local use. When defining a Shell variable name, the first character must be a letter (A-Z, A-Z), can't start with a number, can't have a space in the middle, can use underscore (_), can't use (-), can't use punctuation, etc. After the variable is defined, you can add $in front of it for variable reference.

SHELL programming common system variables:

$0 name of current script
 $n the nth parameter of the current script, n=1,2 Nine
 $* all parameters of the current script (excluding the program itself)
Number of parameters of the current script (excluding the program itself)
$? The status of a command or program after execution. A return of 0 indicates successful execution
 PID number of the $$program itself

SHELL programming common environment variables:

PATH command, divided by colons
 HOME print user HOME directory
 Shell displays the current Shell type
 USER print the current USER name
 id print current user id information
 PWD displays the current path
 TERM prints the current terminal type
 HOSTNAME displays the current HOSTNAME

3. Conditional flow control statement

3.1 If condition judgment statement

Usually starts with if and ends with fi. You can also add else or elif to judge multiple conditions. The if expression is as follows:

if (expression) 
Statement 1

If common judgment logic operators:

-F. judge whether the file exists eg: if [- f filename]
-d. determine whether the directory has eg: if [- d dir]
-eq equals, applies to numeric comparison equal
 -ne is not equal to, apply to numerical comparison not equal
 -lt less than, applied to integer comparison letter
 -gt greater than, applied to integer comparator
 -le less than or equal to, applied to integer comparison
 -ge greater than or equal to, applied to integer comparison
 -A. both sides hold (and) logical expression - a logical expression
 -o logical expression of unilateral (or) - o logical expression
 -z empty string
 ||Unilateral establishment
 &&Expression of both parties
 ==Equality, applied to string comparison

If judge the difference between brackets:

() for multiple command groups, command substitution, initialization arrays;
() integer extension, operator, redefining variable value, arithmetic operation comparison;
[] bash internal command, [is the same as test, regular character range, reference array element number, does not support + - * / mathematical operator, logical test uses - a, - o.
The keyword of [[]] bash program language is not a command. The [[]] structure is more general than the [] structure. It does not support + - * / mathematical operators. The logical test uses & &, |.
{} is mainly used for command sets or scopes, such as MKDIR - P / data / 201 {7,8}/

3.2 for loop statement

For loop statement is mainly used to read and traverse a data field. It is usually used to loop a file or list. Its syntax is in the form of for Do begins, do ends. The syntax format is as follows:

For VaR in (expression)
	Statement 1

3.3 While loop statement

The while loop statement is similar to the for loop. It is mainly used to read and traverse a data field. It is usually used to loop a file or list. If the loop condition is met, the loop will continue. If the condition is not met, the loop will exit. Its syntax format is while Do begins, do ends. The syntax format is as follows:

while (expression)
 Statement 1

3.4 Case selection statement

Case selection statement is mainly used for matching and outputting multiple selection conditions. Similar to if elif statement, it is usually used for script to pass input parameters and print output results and contents. Its syntax format is case in, esac. The syntax format is as follows:

case  $1  in  
    Statement 1 
    Statement 2
    Statement 3

3.5 Select statement

The select statement is generally used for selection. It is often used to create a selection menu. It can cooperate with PS3 to print the output information of the menu. Its syntax format is select Start in do, end in done. The syntax format is as follows:

Select I in (expression) 

#Note: parameter passing needs to be added during script execution. It cannot be added during selection

3.6 method

Shell allows a set of command sets or statements to form an available block. These blocks are called shell functions. Shell functions only need to be defined once, and can be used at any time in the later stage. There is no need to add duplicate statement blocks in shell scripts. Their syntax format starts with function name() {and ends with}.
The syntax format is as follows:

function name (){
#Note: the parameters in the method can only accept the parameters in the script. For receiving, add {} to receive the parameters, such as ${i}


function install_nginx(){ 
        echo ${1} 
PS3="Please select Install Menu:" 
select i in install_nginx install_mysql install_php 
        echo $i $1 


[root@localhost ~]# sh select_test.sh hello_world
1) install_nginx
2) install_mysql
3) install_php
Please select Install Menu:1
install_nginx hello_world
Please select Install Menu:

3.7 read

3.7.1 basic usage

echo -n "please input your name:" 
read name 
echo "Hello $name,welcome!" 

[root@localhost ~]# sh read.sh 
please input your name:cluo
Hello cluo,welcome!

3.7.2 read-p prompt

read -p  "please input your name:" name 
echo "Hello $name,welcome!" 

[root@localhost ~]# sh read.sh
please input your name:cluo
Hello cluo,welcome!                     

3.7.3 read-p prompt

read -p  "please input your name:" name1 name2 
echo "Hello $name1,$name2 welcome!"

[root@localhost ~]# sh read.sh
please input your name:cluo meixi
Hello cluo,meixi welcome!

3.7.4 without specifying variables, the received data will be stored in the special environment variable REPLY

read -p  "please input your name:" 
select i in $REPLY 
        echo $i 

[root@localhost ~]# sh read.sh
please input your name:cluo meixi luoben
1) cluo
2) meixi
3) luoben

3.7.5 read-t timeout

if read -t 5 -p "please input your name:" name 
        echo "Hello $name ,Welcome!" 
        echo "timeout!" 

[root@localhost ~]# sh read.sh
please input your name:cluo
Hello cluo ,Welcome!
#Five seconds no input
[root@localhost ~]# sh read.sh
please input your name:

3.7.6 number of read-n characters

read -n1 -p "please input your choose[Y/N]:" opt  
case $opt in 
        Y | y ) 
        echo "I love cluo" 
        N | n ) 
        echo "I love meixi" 

 [root@localhost ~]# sh read.sh
please input your choose[Y/N]:y
I love cluo

3.7.7 hide

read -s -p "please input your passwd:" passwd 
        echo "your passwd is $passwd"  

[root@localhost ~]# sh read.sh
please input your passwd:
your passwd is 123

3.7.8 read text

cat test.txt | while read line 
        echo "This is line$COUNT:$line" 
        COUNT=$[ $COUNT + 1] 

[root@localhost ~]# sh read.sh 
This is line1:I
This is line2:LOVE
This is line3:CHINA

4. Four Swordsmen

4.1 Find

The Find tool is mainly used to Find operating system files and directories. Its syntax parameter format is:

find path -option [ -print ] [ -exec -ok command ] { } ;

Details of common parameters of option:

-name   filename    			#Find the file named filename;
-type    b/d/c/p/l/f			    #Find block devices, directories, character devices, pipes, symbolic links, and common files;
-size      n[c]     			#Check the file with n blocks [or N bytes] in length;
-perm               			#Search by execution authority;
-user    username   			#Search by file owner;
-group   groupname  			#Search by group;
-mtime    -n +n     			#Find files by file change time, - N refers to within n days, + n refers to before n days;
-atime    -n +n     			    #Find files by file access time;
-ctime    -n +n     			    #Find files by file creation time;
-mmin     -n +n     			    #Find files by file change time, - N refers to within n minutes, + n refers to before n minutes;
-amin     -n +n     			    #Find files by file access time;
-cmin     -n +n     			    #Find files by file creation time;
-maxdepth						#Find directory level depth.


find   /data/    -name   "*.txt"     		#Find the file whose / data / directory ends in. txt;
find   /data/    -name   "[A-Z]*"    		#Find the file whose / data / directory starts with an uppercase letter;
find   /data/    -name   "test*"     		#Find the file whose / data / directory starts with test;

find   /data/    -type d   				#Find the folder under the / data / directory;
find   /data/    !   -type   d    		#Find the non folder under the / data / directory;
find   /data/    -type  l   			#Find the linked file in the / data / directory.
find  /data/ -type d|xargs chmod 755 -R 	#Check the directory type and set the permission to 755;
find  /data/ -type f|xargs chmod 644 -R 	#Check the file type and set the permission to 644;

find   /data/    -size   +1M              #Check the files whose file size is larger than 1Mb;
find   /data/    -size   10M            	#Check the file size of 10M;
find   /data/    -size   -1M            	#Check the files whose file size is less than 1Mb;

atime,access time  	Time the file was read or executed;
ctime,change time  	File status change time;
mtime,modify time  	When the content of the file was modified;
find /data/ -mtime +30 	-name 	"*.log"  #Find the log file 30 days ago;
find /data/ -mtime -30 	-name 	"*.txt"  	#Search for log files within 30 days;
find /data/ -mtime 30 	-name  	"*.txt"	#Find the log file of the 30th day;
find /data/ -mmin  +30	-name  	"*.log"  #Find the log file modified 30 minutes ago;
find /data/ -amin  -30 	-name  	"*.txt"  	#Find the log file accessed within 30 minutes;
find /data/ -cmin  30 	-name  	"*.txt"	#Find the log file changed in the 30th minute.

#Find the file whose / data directory ends with. log and whose file is larger than 10k, and cp to / tmp directory at the same time;
find /data/  -name "*.log"  -type f  -size +10k -exec cp {} /tmp/ \;
#Find the file whose / data directory ends in. txt and whose file is greater than 10k, and whose permission is 644, and delete the file;
find /data/ -name "*.log"  –type f  -size +10k  -perm 644 -exec rm –rf {} \;
#Find the file whose size is larger than 10M 30 days ago, and move it to the directory / tmp;
find /data/ -name "*.log"  –type f  -mtime +30 –size +10M -exec mv {} /tmp/ \;

4.2 SED

Sed is a non interactive text editor, which can edit text files and standard input. Standard input can come from keyboard input, text redirection, strings, variables, and even text from pipes. Similar to VIM editor, it processes one line of content at a time. Sed can edit one or more files, simplify repeated operation of files, and write conversion programs And so on. When processing text, the currently processed lines are stored in the temporary buffer, which is called "pattern space". Then, the contents of the buffer are processed with sed command. After processing, the contents of the buffer are output to the screen or written to the file. If you output to the screen, the actual file contents do not change unless you use redirection to store the output or write to the file. Its syntax parameter format is:

sed [-options] ['Commands'] filename;
The sed tool processes the text by default. The output screen of the text content has been modified, but the file content has not been modified in fact. You need to add the - i parameter to completely modify the file;

Details of common parameters of option:

x                   		#x is the designated line number;
x,y                 		    #Specifies the range of line numbers from x to y;
/pattern/           		    #Query the row containing the pattern;
/pattern/pattern/   		        #The query contains two patterns of rows;
/pattern/,x         		    #From the matching line with pattern to the line with No. x;
x,/pattern/         		    #From line x to the matching line with pattern;
x,y!                		    #The query does not include rows with x and y line numbers;
r                			#Reading a file from another file;
w                			#Write text to a file;
y                			#Changing characters;
q             				#Exit after the first pattern matching;
l                			#Display control characters equivalent to octal ASCII code;
{}              			    #The command group executed on the location line;
p                			#Print matching lines;
=                			#Print file line number;
a\              			    #The text information is added after the positioning line number;
i\              			    #Insert text information before positioning line number;
d                			#Delete location line;
c\              			    #Replace positioning text with new text;
s                			#Replace the corresponding mode with the replacement mode;
n                			#Read the next input line and process the new line with the next command;
N                           #Reads the next row of the current read in row into the current schema space.


#Replace old with new in cluo.txt text
sed 's/old/new/g' cluo.txt

#Print the first line to the third line of cluo.txt text
sed -n '1,3p' cluo.txt

#Print the first and last lines of cluo.txt text
sed -n '1p;$p' cluo.txt

#Delete the first line to the third line of cluo.txt
sed '1,3d' cluo.txt

#Delete the matching line in cluo.txt to the last line
sed '/hello/,$d' cluo.txt

#Delete the last line of cluo.txt
sed '$d' cluo.txt

#Delete the last 6 lines of cluo.txt
for i in `seq 1 6`; do sed -i '$d' cluo.txt; done

#Find the line of cluo in cluo.txt and add word character in the next line. a means to add string in the next line
sed '/cluo/aword' cluo.txt

#Find the line of cluo in cluo.txt, and add word character on the previous line. i means to add string on the previous line
sed '/cluo/iword' cluo.txt

#In cluo.txt, find the end of the line ending with test and add the string word, $for the end ID, & in Sed, add
sed 's/test$/&word/g' cluo.txt

#Find the row of www in cluo.txt, add the string word at the beginning of the row, ^ for the start ID, & for add in Sed
sed '/www/s/^/&word/' cluo.txt

#Multiple sed command combinations, using the - e parameter
sed -e  '/www.jd.com/s/^/&1./'  -e  's/www.jd.com$/&./g'  cluo.txt

#Multiple sed command combinations, split with semicolon ";"
sed  -e  '/www.jd.com/s/^/&1./;s/www.jd.com$/&./g'  cluo.txt

#Sed read system variable, variable replacement
sed  "s/www.jd.com/$WEBSITE/g" cluo.txt

#Modify SELINUX policy enforcer to disabled, find / SELINUX / line, and then change its enforcer value to disabled,! s means SELINUX line is not included
sed  -i   '/SELINUX/s/enforcing/disabled/g' /etc/selinux/config
sed  -i   '/SELINUX/!s/enforcing/disabled/g' /etc/selinux/config

#Merge two lines
sed 'N;s/\n//' cluo.txt

4.3 AWK

AWK is an excellent text processing tool. It is one of the most powerful data processing engines available in Linux and Unix environments. It is named AWK with the initials of three inventors, Aho, Weinberger and Kernighan. AWK is a line level text efficient processing tool. The new version that AWK has been improved to generate includes Nawk and gawk. Generally, gawk is gawk by default for Linux. Gawk is GNU open source free version of AWK.
The basic principle of AWK is to process the data in the file line by line, find the pattern matching the given content in the command line, if the matching content is found, carry out the next programming step, if the matching content is not found, continue to process the next line. Its syntax parameter format is:

awk -option 'pattern + {action}' file

Details of AWK basic syntax parameters:
Single quotes' are used to distinguish them from shell commands;
Braces {} denote a command group;
Pattern is a filter, which means that only rows matching pattern conditions can be processed with Action;
Action is a processing action. The common action is Print;
Using ා as a comment, pattern and action can have only one, but not both.

Details of AWK built-in variables:
FS separator, the default is space;
OFS output separator;
NR current number of rows, starting from 1;
NF number of current record fields;
$0 current record;
1 1 ~ 1 n the nth field (column) of the current record.

Details of AWK built-in functions:
gsub(r,s): replace r with s in $0;
index(s,t): returns the first position of T in S;
length(s): the length of S;
match(s,r): whether s matches r;
split(s,a,fs): divide s into sequence a on fs;
substr(s,p): returns the substring of s starting from p.

AWK common operators, operators and judges:
*++– increase and decrease (pre or post);
^* index (right combination);
! + - non, unary plus sign, unary minus sign;
+- * /% addition, subtraction, multiplication, division and remainder;
< = = =! = > = number comparison;
&&Logic and;
||Logic or;
=+ = - = * = / =% = ^ = assignment.

AWK and process control statement:
if(condition) { } else { };
while { };
do{ }while(condition);
for(init;condition;step){ };


#AWK prints the hard disk device name, which is divided by space by default
df -h|awk  '{print  $1}'

#AWK is divided by spaces, colons, semicolons \ t
awk -F '[:\t;]' '{print $1}' cluo.txt

#AWK is divided by colons, printing the first column, and appending the content to / tmp/awk.log
awk -F: '{print $1 >> "/tmp/awk.log"}' cluo.txt

#Print lines 3 to 5 in the file, NR for print lines, $0 for all fields of text
awk 'NR==3,NR==5 {print}' cluo.txt
awk 'NR==3,NR==5 {print $0}' cluo.txt

#Print the first and last columns of lines 3 to 5 in the file
awk 'NR==3,NR==5 {print $1,$NF}' cluo.txt

#Line numbers greater than 80 in the printed file
awk 'length($0)>80 {print NR}' cluo.txt

#AWK refers to Shell variables. Use - v or double quotation mark + single quotation mark
awk -v STR=hello '{print STR,$NF}' cluo.txt
STR="hello";echo|awk '{print "'${STR}'";}'

#AWK cut with colon, print the first column and only display the first 5 lines at the same time
cat /etc/passwd|head -5|awk -F: '{print $1}'
awk -F: 'NR>=1&&NR<=5 {print $1}' /etc/passwd

#awk specifies the sum of the first column of the file jfedu.txt
cat cluo.txt |awk '{sum+=$1}END{print sum}'

#If NR line number is divided by 2 and the remainder is 0, the line will be skipped and the next line will continue to be printed on the screen
awk -F: 'NR%2==0{next}{print NR,$1}' /etc/passwd

#Add custom character
ifconfig  eth0|grep "Bcast"|awk '{print "ip_"$2}'

#Combination of AWK and if, digital comparison
echo 3 2 1 |awk '{if(($1>$2)||($1>$3)) {print $2} else {print $1}}'

#Analyze the status code 404, 502 and other error information pages of the Nginx access log, and count the IP addresses with more than 20 times
awk '{if ($9~/502|499|500|503|404/) print $1,$9}' access.log|sort|uniq –c|sort –nr | awk '{if($1>20) print $2}'

#Count server state connections
netstat -an | awk '/tcp/ {print $NF}' | sort | uniq -c

4.4 GREP

Global search regular expression(RE) is a powerful text search tool, which can use regular expression to search text and print out matching lines.
The grep family of Unix/Linux includes grep, egrep and fgrep. The commands of egrep and fgrep are slightly different from grep. Egrep is an extension of grep, supporting more re metacharacters. Fgrep is fixed grep or fast grep shorthand, they regard all letters as words. The metacharacters in regular expressions represent their own literal meaning, without any other special meaning, and are seldom used.
At present, the GNU version of grep is used by default for Linux operating system. It is more powerful and can use egrep and fgrep functions through the - G, - E, - F command line options. Its syntax format is as follows:

grep -opiton 'word' Filename

Parameter details:

-a. search by text file;
-c. calculate the number of times the line is found;
-i. ignore case;
-n output the line number by the way;
-v. reverse selection, i.e. display all lines without matching text;
-h. do not display the file name when querying multiple files;
-l. when querying multiple files, only the file name containing matching characters is output;
-s does not display the error message of no existing or no matching text;
-E. allow egrep extended pattern matching.

Wildcard type details:

*0 or more characters and numbers;
Match any character;
#Indicate comments;
|Pipe symbols;
; multiple commands are executed continuously;
&Background operation instruction;
! logical not;
[] content range, matching the content in brackets;
{} command block, multiple commands matching.

Regular expression details:

*The previous character matches 0 or more times;
. match any character except line break;
. * represents any character (more than 1 +);
^Match the beginning of a line, i.e. start with a certain character;
$matches the end of a line, i.e. ends with a character;
\(.. \) mark matching characters;
[] matches any specified character in brackets, but only one character;
[^] matches any character except the bracket;
\Escape sign, cancel special meaning;
\< anchor the beginning of the word;
\>Anchor the end of the word;
{n} Matching characters appear n times;
The occurrence of {n,} matching characters is greater than or equal to N times;
{n,m} matching characters appear at least N times and at most m times;
\w. match text and number characters, not symbols;
\The reverse form of w \ w, matching one or more non word characters, matching symbols;
\b. word lock;
\s matches any blank characters;
\d matches a numeric character, equivalent to [0-9].


Grep - C "test" cluo.txt counts the total number of test characters;
Grep - I "TEST" cluo.txt finds all TEST rows case insensitive;
Grep - n "test" cluo.txt print the line and line number of test;
Grep - V "test" cluo.txt does not print the line of test;
Grep "test [53]" cluo.txt starts with the character test, followed by lines 5 or 3;
Grep "^ [^ test]" cluo.txt displays the row whose output line is not test at the beginning;
Grep "[M M] ay" cluo.txt "matches lines beginning with m or m;
Grep "K... D "cluo. TXT" matches K, three arbitrary characters, followed by the line of D;
Grep "[A-Z] [9] d" cluo.txt "matches upper case letters, followed by 9D character lines;
Grep "T \ {2, \}" cluo.txt print characters T characters appear more than two consecutive lines;
Grep "T \ {4,6 \}" cluo.txt prints the lines with T characters appearing four or six times in succession;
Grep - n "^ $" cluo.txt print the line number of the blank line;
Grep - ve "#^ $" cluo.txt does not matchාand blank lines in the file;
Grep -- color - ra - e "db|config|sql" * matches files containing dB or config or SQL;
Grep -- color - e "\ < ([0-9] {1,3} \) {3} ([0-9] {1,3}) \ > cluo.txt matches the IPV4 address

5, array

An array is a set of elements of the same data type arranged in a certain order. A finite number of variables of the same type are named by one name, and then their variable sets are distinguished by numbers. This name is called the array name and the number is called the subscript. One dimensional array is often used in Linux Shell programming.
The definition of an array is generally defined in parentheses. The values of an array can be specified randomly, as follows: definition, statistics, reference and deletion of a one-dimensional array:

(1) One only array definition and creation:

LAMP=(httpd  php  php-devel php-mysql mysql mysql-server)

(2) Array subscript generally starts from 0, as follows is the method of referencing array:

echo	${ARRTEST[0]}    	    Reference the first array variable and print the result test1;
echo 	${ARRTEST[1]}		    Reference to the second array variable;
echo  	${ARRTEST[@]}   		Display all parameters of the array;
echo  	${#ARRTEST [@]} shows the number of array parameters;
echo    ${#ARRTEST[0]} displays the test1 character length;
echo    ${ARRTEST[@]:0}   	Print all values of the array;
echo    ${ARRTEST[@]:1}   	Print all values from the second value;
echo    ${ARRTEST[@]:0:2} 	Print from first and second values;
echo    ${ARRTEST[@]:1:2} 	Prints from the second and third values.

(3) Array replace operation:

Arrtest = ([0] = www1 [1] = www2 [2] = www3) array assignment;

Echo ${arrtest [@] / www1 / www0} replaces the array value www1 with www0;
Newarrtest = ` echo ${arrtest [@] / www1 / www0} ` assigns the result to a new array.

(4) Array delete operation:

unset array[0] delete the first value of the array;
unset array[1] deletes the second value of the array;
unset array deletes the entire array.

Published 40 original articles, won praise 3, visited 3035
Private letter follow

Tags: shell Linux Programming SELinux

Posted on Sun, 08 Mar 2020 23:29:24 -0400 by toovey