AWK

Difference between AWK and sed:

• AWK is more like a scripting language
• AWK is used for text processing of "comparison specification" and for counting quantity and outputting specified fields
• Use sed to process non-standard text into "more standard" text

Process control of AWK script

• Routine BEGIN {}
• Main input loop {} (usually only the main input loop is written, and in a few cases the begin and end routines are written)
• All file read completion routine END {}

Field reference and separation of AWK

Records and fields

• Each line is called an AWK record
• Words separated by spaces and tabs are called fields
• You can specify delimited fields yourself

Field reference

• Each field is represented by $1, $2... $n in awk
  • awk '{print , ,}' filename ({} called main input loop)
• awk who can change the field separator with the - F option
  • Awk - F ',' {print $1, $2, $3} 'filename (- F specifies that the delimiter is comma)
  • Delimiters can use regular expressions

awk '/^menu/{ print $0 }' /boot/grub2/grub.cfg

awk -F "'" '/^menu/{ print $2 }' /boot/grub2/grub.cfg // Take '' as the delimiter, take the second field and take out the kernel

awk -F "'" '/^menu/{ print x++,$2 }' /boot/grub2/grub.cfg // Set number through x + +

Expression for AWK

• Assignment operator
• Arithmetic operator
• System variable
• Relational operator
• Boolean operator

Assignment operator

= Is the most common assignment operator
var1 = "name"
var2 = "hello" "world" // Splicing of two strings
var3 = $1 // Separated first field (different from shell)

Other assignment operators
++ -- += -= *= /= %= ^=

arithmetic operator

+ - * / % ^

System variable

• FS and OFS, field separator. FS represents the input field separator (what symbol is used as the division symbol for the input file), OFS represents the output field separator.
• RS, record separator (a record represents a line. For example, if two lines are combined into one line, we can process the RS newline character into other symbols) (the default RS is \ n)
• NR and FNR, number of lines (NR, the file will not be rearranged when it changes; FNR, the serial number will be rearranged when it changes)
• NF, the number of fields. The content of the last field can be retrieved with $NF

head -5 /etc/passwd | awk -F ":" '{print $1}' // : separate, output the first field

head -5 /etc/passwd | awk 'BEGIN{FS=":"}{print $1}' // Set BEGIN to output the first field

head -5 /etc/passwd | awk 'BEGIN{FS=":'}{print $1,$2}' // Output the first and second fields. The default separator is a space

head -5 /etc/passwd | awk 'BEGIN{FS=":';OFS="-"}{print $1,$2}' // Set the output separator to '-'

head -5 /etc/passwd | awk 'BEGIN{RS=":"}{print $0}' // Set ':' as a newline character

head -5 /etc/passwd | awk '{print NR}' // set number 

head -5 /etc/passwd | awk '{print NR,$0}' // Display line number and file content

awk '{print FNR,$0}' /etc/hosts /etc/hosts // The serial number will be rearranged

awk '{print NR,$0}' /etc/hosts /etc/hosts // Serial numbers are not rearranged

head -5 /etc/passwd | awk 'BEGIN{FS=":'}{print NF}' // Number of output fields

head -5 /etc/passwd | awk 'BEGIN{FS=":'}{print $NF}' // Output last field

AWK conditions and cycles

Conditional statement

* Conditional statement use if At the beginning, judge which statement to execute according to the result of the expression
if(expression) // If the expression holds, the return value is 1; if the expression does not hold, the return value is 0 (the difference between awk expression and shell)
  awk Statement 1
[else
  awk Statement 2
]
* If more than one statement needs to be executed, you can use{}Enclose multiple statements

[external chain picture transfer failed. The source station may have anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-p1wputzf-163478143823)( https://note.youdao.com/yws/res/14439/F58C4EC9DAE944AB9FA1F6FEBD052837 )]

awk '{if($2>=80) print $1}' kpi.txt // User with KPI > = 80, output

awk '{if($2>=80) {print $1; print$2}}' kpi.txt  // Both the user and the kpi will be output (in parentheses)

Circular statement

* while loop
    while((expression)
        awk Statement 1
* do loop
    do {
        awk Statement 1
    }while(expression)
* for loop
    for(Initial value; cyclic judgment condition; accumulation)
        awk Statement 1
* Other statements that affect control
    break
    continue
    

head -1 kpi.txt

head -1 kpi.txt | awk '{for(c=2;c<=NF;c++) print c}'

head -1 kpi.txt | awk '{for(c=2;c<=NF;c++) print $c}'

head -1 kpi.txt | awk '{for(c=2;c<=NF;c++) sum+=$c ; print sum}' // Total performance

head -1 kpi.txt | awk '{for(c=2;c<=NF;c++) sum+=$c; print sum/(NF-1)}' // Calculate the average performance. / / note that sum will not be destroyed, and the previous value is still there

heada -1 kpi.txt | awk '{sum=0; for(c=2;c<=NF;c++) sum+=$c ; print sum/(NF-1)}' kpi.txt // Average all

Array of AWK

• Definition of array
• Traversal of array
• Delete array
• Command line parameter array

array define

Array name[subscript]=value

Subscripts can be numbers or strings

Traversal of array

for(variable in Array name)

Use array name[variable]Operate on the elements of each array in turn

Delete array

delete array // You don't want the entire array

delete array[subscript] // Do not want an element in the array

awk '{ sum=0; for(column=2;column<=NF;column++) sum+=$column; average[$1]=sum/(NF-1)}END{ for( user in average) print user,average[user]}' kpi.txt  // Take the average value of each person

awk '{ sum=0; for(column=2;column<=NF;column++) sum+=$comumn; average[$1]=sum/(NF-1)}END{ for(user in average) sum2+=average[user] ;print sum2/NR}' kpi.txt // Take out the average score of everyone

vim avg.awk  // Save awk script file

awk -f avg.awk kpi.txt  // Load awk script

Command line parameter array

ARGC  // Number of command line arguments followed by awk

ARGV // Specific content of each parameter

vim arg.awk

BEGIN{
    for(x=0;x<ARGC;x++)
        print ARGV[x]
    print ARGC
}

awk -f arg.awk 11 22 33

vim result.awk

{
sum = 0
for(column = 2; column <=NF ;column++)
    sum += $column
average[$1] = sum/(NF-1)

if( average[$1] >=80) 
    letter = "S"
else if(average[$1]>=70)
    letter = "A"
else if(average[$1] >= 60)
    letter = "B"
else
    letter = "C"
    
print $1,average[$1]
letter_all[letter]++
}
END{
for(user in average)
    sum_all += average[user]
avg_all=sum_all/NR
print "average all:",avg_all
for( user in average) 
    if(average[user] > avg_all)
        above++
    else 
        below++
print "above",above
print "below",below
print "S:",letter_all["S"]
print "A:",letter_all["A"]
print "A:",letter_all["B"]
print "A:",letter_all["C"]
}

awk -f result.awk kpi.txt

Function of AWK

• Arithmetic function
• String function
• Custom function

Arithmetic function

int() // Take integer
awk 'BEGIN{pi=3.14 ;print int(pi) }'

rand() // Take random tree, between 0-1
awk 'BEGIN{print rand()}' // Pseudo random number
awk 'BEGIN{srand();print rand()}' // random number

String function

gsub(r,s,t)

index(s,t)

length(s)

match(s,r)

split(s,a,sep)

sub(r,s,t)

substr(s,p,n)

man awk
/gsub

Custom function

function Function name (parameter){
    awk sentence
    return awk variable
}

awk 'function a(){return 0} BEGIN{print a()}' // Custom functions are written outside the main input loop

awk 'function double(str) {return str str} BEGIN {print double("hello awk")}'