Nginx log analysis script

Operation and maintenance work is a complex work. Sometimes, in the face of tens of thousands of logs, how to analyze them? Is it an analysis of one by one?
Smart people will choose scripts. That's why we advocate automatic operation and maintenance now. I don't need to talk much about nonsense, just go to scripts.

vim /data/scripts/log_analysis.sh
#!/bin/bash
###############################################
#    Desc        : nginx Log analysis script           #
#    Author      : Bertram                    #
#    Date        : 2019-12-21                 #
#    Copyright   : Personal belongs           #
###############################################
public(){
    echo ""
    read -p "Please enter the access log to analyze: " log_file
    echo ""   
    if [ ! -f $log_file ];then
        echo "not found: ${log_file}"
        exit 1
    fi  

    if [ ! -s $log_file ];then
        echo "${log_file}Empty file"
        exit 1
    fi

    #Output the top num data of log access ranking, which can be customized 
    top_num=5
    input_file=`echo $log_file | awk -F '/' '{print $(NF)}'`
    analyze_dir=/home/Bertram/`date +%F`
    top_ip_file=$analyze_dir/ngx_log_top_ip_${input_file}.txt
    top_src_url_file=$analyze_dir/ngx_log_top_src_url_${input_file}.txt
    top_dest_url_file=$analyze_dir/ngx_log_top_dest_url_${input_file}.txt
    top_code_file=$analyze_dir/ngx_log_top_code_${input_file}.txt
    top_terminal_file=$analyze_dir/ngx_log_top_terminal_${input_file}.txt

    mkdir -p $analyze_dir
    start_time=`head -1 $log_file | awk '{print $4}'|cut -d "[" -f2`
    end_time=`tail -1 $log_file | awk '{print $4}'|cut -d "[" -f2`
    total_nums=`wc -l $log_file | awk '{print $1}'`
    size=`du -sh $log_file | awk '{print $1}'`

    #Get start and end times
    echo "Access start time: $start_time ; Deadline: $end_time"
    #Get total row number and size
    echo  "Common access $total_nums second ; Log size: $size"
    #Get the most active IP
    cat $log_file | awk '{print $1}' | sort | uniq -c | sort -rn | head -${top_num} > $top_ip_file
    #Get the url with the most access sources
    cat $log_file | awk '{print $13}' | sort | uniq -c | sort -rn | head -${top_num} > $top_src_url_file
    #Get the most requested url
    cat $log_file | awk '{print $8}' | sort | uniq -c | sort -rn | head -${top_num} > $top_dest_url_file
    #Get the most returned status codes
    cat $log_file | awk '{print $11}'| sort | uniq -c | sort -rn | head -${top_num} > $top_code_file
    #Get the most returned terminal types
    cat $log_file | awk '{print $14}'| sort | uniq -c | sort -rn | head -${top_num} > $top_terminal_file
    }

    simple(){
    echo "+-+-+-+-+-+- Here is the analysis +-+-+-+-+-+-"
    #Get the most active IP
    printf "The most active front ${top_num}One visit IP: \n"
    cat $top_ip_file
    echo ""
    #Get the url with the most access sources
    printf "Before the most visited sources ${top_num}individual url: \n"
    cat $top_src_url_file
    echo ""
    #Get the most requested url
    printf "Before most requests ${top_num}individual url: \n"
    cat $top_dest_url_file
    echo ""
    #Get the most returned status codes
    printf "Back to the top ${top_num}Status codes: \n"
    cat $top_code_file
    echo ""
    printf ""
    #Get the terminal number that returns the most
    printf "Back to the top ${top_num}Terminal number: \n"
    cat $top_terminal_file
    echo ""
    printf ""   
    printf "Back to the top ${top_num}individual IP City (query time is a little slow, wait patiently!): \n"
    echo ''
    printf "%-15s %-15s %-30s\n" "Number of visits" "  IP address" "      Place of ownership"
    echo '-----------------------------------------------'
    a=0
    cat $analyze_dir/ngx_log_top_ip_${input_file}.txt | while read line
    do
    ip=$(echo $line | cut -d '"' -f2)
    count=$(echo $line | cut -d '"' -f1)
        printf "%-10s %-15s %-30s\n" $count $ip $(curl -s "http://freeapi.ipip.net/$(echo $line | cut -d '"' -f2)" | awk -F '\"' {'print $2"--"$4"--"$6'})
    echo '-----------------------------------------------'
    let a=a+1
    done
    echo ""
    printf ""
}

case $1 in
    help)
        echo ""
        echo -e $"Usage: $0 enter a log file \n"               
        ;;
    *)
     public
     simple
        ;;
esac
exit 0

Implementation function:
1. Analyze the ip address of the top N;
2. Analyze the top N url of the visit;
3. Analyze the target url of top N;
4. Analyze the top N terminal types;
5. Automatically match the ip home of the top N.
Note: the log file and analysis script can be placed in one directory; enter the absolute path of the log file.

Usage:
Nginx

Tags: Linux vim Nginx curl

Posted on Sun, 12 Jan 2020 09:42:25 -0500 by jllydgnt