Batch Linux server patrol through Shell script

1. Batch inspection process

Generally, a script consists of three parts, including:
(1) Patrol script: collect and return performance information or indicators;
(2) Distribute the patrol script: the main server connects the corresponding host through the exception tool according to the account password in the information list of the server to be patrol, distributes the script to the corresponding host, and then executes the patrol script on the target host;
(3) Information list of servers to be inspected: including management IP and login password of servers to be inspected.
Because of the need to obtain the server software and hardware and performance information, you need to obtain the root permission of each server;
Because patrol information needs to be returned to the main server, SCP command needs to be supported;
Because the implementation requires the interaction between hosts, the Expect tool needs to be supported between hosts.

2. Preparation of environmental parameters

2.1 related server information

(1) IP address of the log upload target server: input as a parameter or preset in the script, for example, 192.168.52.3;
(2) Log upload target server root password: input as a parameter or preset in the script, such as 123456;
(3) List of server information to be inspected: file format, as parameter input or preset in the script, each behavior is the IP address and password of a server, for example:

(4) The server supports the Expect tool:

2.2 log storage path

Log upload storage path / TMP / checklog "directory
Local log storage path / TMP / local? Serverlog? Directory
Patrol script file storage path: / tmp/check_script

3. Expect tool

Expect is a tool for automatic control and testing based on tcl.
It mainly solves the non interactive problem in shell script. Usually used in scenarios where data needs to be entered manually, expect can be used in scripts to automate.
In general, when logging in to Linux server remotely, when running commands, scripts or programs, these commands, scripts or programs need to input some instructions to continue running from the terminal again after the server echoes some prompts. Because the requirements for echo prompts are not fixed, these inputs need to be done manually.
For example, the response steps of the normal remote login Linux server and the first remote login server are not the same. At this time, you need to use expect tool to simulate standard input and provide it to the program according to the prompts of the program, so as to realize automatic interactive execution.

3.1, command

The Expect tool works by judging the echo content and then executing the next instruction. Its main commands include:

command Explain
spawn Start a new interactive process, followed by a command or a specified program
expect Receive information from the process. If the match is successful, perform the action after expect
send Send string to process
send exp_send Used to send the specified string information
exp_continue In expect, multiple matches are needed
send_user Used to print out the echo in the shell
interact Allow user interaction
exit Exit expect script
eof expect end of execution, exit
set Defining variables
puts Output variables
set timeout Set timeout

3.2, installation

(1) Install tcl: Yum install - y tcl tclx tcl devel


(2) Install expect: yum – y install exec

3.3, test

(1) Test script

(2) Test results

4. Implementation of patrol script

4.1 script implementation

The file name is: check_command.sh

#!/bin/bash
#

#Check whether it is root permission, otherwise exit
[ $(id -u) -gt 0 ] && echo "please used the root or sudo " && exit 1

#Setting parameters
main_ipaddr=$1 #Log upload server IP
main_passwd=$2 #Log upload server password
main_log_path="/tmp/checklog_directory" #Log upload path
local_log_path="/tmp/local_serverlog_directory" #Local log storage path
LogName=`ifconfig | grep "inet"|sed '2,4d' |awk -F" " '{print $2}' ``echo "-"``date +%y%m%d%H%M%S` #Construction log name

#Get system information
function get_status(){
echo -e "#Basic system information:“
OS=` head -n 1 /etc/centos-release ` #System version
Kernel=` uname -r ` #system kernel
Hostname=` hostname ` #Host name
Default_lang=` echo $LANG ` #default language
Time=$(date +'%Y-%m-%d %H:%M:%S') #current time
Logintime=$(who -b |awk '{print $3,$4}') #Login time of this user
Uptime=$(uptime|awk '{print $3}'|awk -F "," '{print $1}') #System run time
Ipaddr=$(ifconfig |grep 'inet ' |awk {'print$2'}|grep -v '127.0.0.1') #IP address
Netmask=$(ifconfig|grep "inet"|sed '2,4d'|awk -F " " '{print $4}') #Mask
cpu_info=$(cat /proc/cpuinfo|grep "name"|cut  -d: -f2|awk '{print "*"$1,$2,$3,$4}'|uniq -c|sed 's/^[ \t]*//G ') CPU Information
cpu_type=$(grep 'model name' /proc/cpuinfo | uniq | awk -F":" '{print $2}') #CPU model
cpu_arch=$(uname -m) #CPU architecture
cpu_number_physical=$(grep 'physical id' /proc/cpuinfo | sort | uniq | wc -l) #Physical CPU number
cpu_number_vir=$(grep 'processor' /proc/cpuinfo | wc -l) #Logical CPU number
cpu_cores=$(grep 'cores' /proc/cpuinfo | uniq | awk -F":" '{print $NF}' ) #Cores per CPU
Sigle_memory_capacity=$(dmidecode|grep -P -A5 "Memory\s+Device"|grep "Size"|grep -v "Range"|grep '[0-9]'|awk -F ":" '{print $2}'|sed 's/^[ \t]*//g ') single memory capacity
Maximum_memory_capacity=$(dmidecode -t 16|grep "Maximum Capacity"|awk -F ":" '{print $2}'|sed 's/^[ \t]*//g ') maximum storage capacity
Number_of_memory_slots=$(dmidecode -t 16|grep "Number Of Devices"|awk -F ":" '{print $2}'|sed 's/^[ \t]*//g ') number of memory card slots
Memory_total=$(cat /proc/meminfo|grep "MemTotal"|awk '{printf("MemTotal:%1.0fGB\n",$2/1024/1024)}'|awk -F ":" '{print $2}') #Theoretical total memory
Product_name=$(dmidecode|grep -A10 "System Information"|grep "Product Name"|awk -F ":" '{print $2}'|sed 's/^[ \t]*//g ') memory brand
Process_numbers=$(top -d 2 -n 1 -b|grep "Tasks" |awk -F':' '{print $2}'|awk -F" " '{print $1}') #Total process number
Process_running=$(top -d 2 -n 1 -b|grep "Tasks"|awk -F "[: ,]" '{print $8}') #Number of active processes
Process_sleeping=$(top -d 2 -n 1 -b|grep "Tasks"|awk -F "[: ,]" '{print $11}') #Number of sleep processes
Process_stoping=$(top -d 2 -n 1 -b|grep "Tasks"|awk -F "[: ,]" '{print $16}') #Number of stopped processes
Process_zombie=$(top -d 2 -n 1 -b|grep "Tasks"|awk -F "[: ,]" '{print $21}') #Number of zombie processes
cpu_load=$(uptime | awk '{for(i=6;i<=NF;i++) printf $i""FS;print ""}') #System load
cpu_load_number=$(uptime | awk -F"load average:" '{print $2}' | awk -F"," '{print $1}' | awk -F"." '{print $1}' |sed 's/^[ \t]*//g ') get the number of CPU cores occupied in the past 1 minute
cpu_user_space=$(top -d 2 -n 1 -b|grep 'C[Pp][Uu]'|head -1|awk -F "[:,]" '{print $2}'|awk -F" " '{print $1}') #Percentage of CPU occupied by user space
cpu_system_space=$(top -d 2 -n 1 -b|grep 'C[Pp][Uu]'|head -1|awk -F "[:,]" '{print $3}'|awk -F" " '{print $1}') #Percentage of CPU occupied by kernel space
cpu_idle=$(top -d 2 -n 1 -b|grep C[Pp][Uu]|grep id|awk -F"," '{print $4}'|awk -F" " '{print $1}') #Idle CPU
cpu_wait=$(top -d 2 -n 1 -b|grep 'C[Pp][Uu]'|head -1|awk -F "[:,]" '{print $6}'|awk -F" " '{print $1}') #Percentage of CPU waiting
Pem_totle_m=$(free -m |grep "Mem" |awk -F" " '{print $2"M"}') #Actual total physical memory M
Mem_free_m=$(free -m |grep "Mem" |awk -F" " '{print $4"M"}') #Actual available memory M
Mem_used=$(top -d 2 -n 1 -b|grep "Mem"|head -1|sed 's/[:,]/ /g'|awk -F" " '{print $7}'|awk '{printf("%dM\n",$1/1024)}') #User cache capacity
Mem_free=$( top -d 2 -n 1 -b|grep "Mem"|head -1|sed 's/[:,]/ /g'|awk -F" " '{print $5}'|awk '{printf("%dM\n",$1/1024)}') #Idle cache capacity
Mem_buffer=$(top -d 2 -n 1 -b|grep "Mem"|head -1|sed 's/[:,]/ /g'|awk -F" " '{print $9}'|awk '{printf("%dM\n",$1/1024)}') #Buffer cache capacity
Swap_total=$(top -d 2 -n 1 -b|grep "Swap"|awk -F" " '{print $3}'|awk '{printf("%dM\n",$1/1024)}') #Total capacity of switching partition
Swap_free=$(top -d 2 -n 1 -b|grep "Swap"|awk -F" " '{print $5}'|awk '{printf("%dM\n",$1/1024)}') #Free capacity of switch partition
Swap_used=$(top -d 2 -n 1 -b|grep "Swap"|awk -F" " '{print $7}'|awk '{printf("%dM\n",$1/1024)}') #Capacity occupied by switching partition
Swap_avail=$(top -d 2 -n 1 -b|grep "Swap"|awk -F" " '{print $9}'|awk '{printf("%dM\n",$1/1024)}') #Usable capacity of swap partition
rx_pck=$(sar -n DEV 1 1 |sed '1,4d'|sed '2,$d'|awk -F" " '{print $3}') #Number of packets received per second by network card 1
tx_pck=$(sar -n DEV 1 1 |sed '1,4d'|sed '2,$d'|awk -F" " '{print $4}') #Number of packets sent per second by network card 1
rx_kB=$(sar -n DEV 1 1 |sed '1,4d'|sed '2,$d'|awk -F" " '{print $5}') #Bytes received per second by network card 1
tx_kB=$(sar -n DEV 1 1 |sed '1,4d'|sed '2,$d'|awk -F" " '{print $6}') #Bytes sent per second by network card 1
echo -e "System version|$OS
//System Kernel
//Hostname
//Default language
//Current Time
//Login time of this user $Logintime
//System Uptime
IP address|$Ipaddr
//Mask | $Netmask
CPU information|$cpu_info
CPU Model|$cpu_type
CPU Framework|$cpu_arch
//Physical CPU number physical
//Logical CPU number vir
//Cores per CPU
//Single memory capacity
//Maximum storage capacity
//Number of memory slots
//Theoretical total memory
//Memory brand
//Total number of processes
//Number of active processes
//Number of sleep processes
//Process stop
//Number of zombie processes
//System load
//Get the number of CPU cores occupied in the past 1 minute
//Percentage of CPU occupied by user space
//Percentage of CPU occupied by kernel space
//Idle CPU
//Waiting percentage of CPU $CPU | wait
//Actual total physical memory
//Actual available memory
//User cache capacity
//Free cache capacity
//Buffer buffer capacity
//Swap total
//Swap free
//Swap used
//Swap available capacity
//Number of packets received per second by network card
//Number of packets sent per second by network card
//Bytes received per second by network card 1
//Bytes sent per second by network card 1“
}

#Get disk status information
function get_disk_status(){
echo -e "#Disk usage:“
    IFS="
        "
    for i in ` df -iP | sed 1d | awk '{print $(NF-1)"\t"$NF"\t"$(NF-2)}'`;do
    DISK_UTILIZ=$( echo $i |awk '{print $1}') #Utilization ratio
    MOUNT_DISK=$(echo $i |awk '{print $2}') #Mount point
    DISK_FREE=$(echo $i |awk '{print $3}') #Partition size
    if [[ $(echo $MOUNT_DISK | sed s/%//g) -gt 70 ]];then
    echo "abnormal""("$MOUNT_DISK"Usage rate is"$DISK_UTILIZ",More than 70%)"
    else
        continue
    fi
    done
df -hP |sed 1d |awk '{print $NF" ""partition"" ""Surplus space"" "$(NF-2),"Utilization rate"" "$(NF-1)}'
}

#Get CPU status
function get_cup_status(){
echo -e "#CPU status "
cpu_number=$(cat /proc/cpuinfo |grep name |cut -d: -f2 |uniq -c |awk '{print $1}') #Extract CPU cores
cpu_load_number=$(uptime | awk -F"load average:" '{print $2}' | awk -F"," '{print $1}' | awk -F"." '{print $1}' |sed 's/^[ \t]*//g ') get the number of CPU cores occupied in the past 1 minute
if [[ $cpu_load_number -lt $cpu_number ]];then
    CPU_STATUS=normal
else
    CPU_STATUS=abnormal
fi
}

#Get memory status
function get_memory_status(){
echo -e "#Memory status:“
MEM_TOTLE=$(free -m |grep "Mem" |awk -F" " '{print $2}') #Total physical memory
MEM_FREE=$(free -m |grep "Mem" |awk -F" " '{print $4}') #Available memory
MEM_TOTLE_M=$(free -m |grep "Mem" |awk -F" " '{print $2"M"}') #Total physical memory M
MEM_FREE_M=$(free -m |grep "Mem" |awk -F" " '{print $4"M"}') #Available memory M
MEM_USED=$(echo $(($MEM_TOTLE-$MEM_FREE))) #Total memory
PERCENT=$(printf "%d%%" $(($MEM_USED*100/$MEM_TOTLE))) #Memory usage
PERCENT_N=$(echo $PERCENT|sed s/%//g) ා memory usage value
if [[ $PERCENT_N -lt 80 ]];then
    MEM_STATUS=normal
else
    MEM_STATUS=abnormal
fi
echo -e "$MEM_STATUS\n"Total memory size:"$MEM_TOTLE_M,"Remaining memory size:"$MEM_FREE_M,"Memory usage"$PERCENT"
}

#Check for violent password cracking
function get_ssh_deny(){
echo -e "#Whether there is violent password cracking:“
SYSTEM_TYPE=$(head -n 1 /etc/os-release |awk -F"=" '{print $2}'|awk -F" " '{print $1}'|sed 's/"//g ') get system type
if [[ $SYSTEM_TYPE = 'CentOS' ]];then
    SSH_SUM=$(cat /var/log/secure |grep "authentication failure" |wc -l)
    SSH_DIY=10
    if [[ $SSH_SUM -gt $SSH_DIY ]];then
        echo ""Please note that the number of password attempts is"$SSH_SUM"
    else
        echo "normal"
    fi
else
    echo "System type not supported"
fi
}

#Check the startup task
function get_auto_start_status() {
echo -e "#Startup task“
conf=$(grep -v "^#" /etc/rc.d/rc.local | sed '/^$/d')
echo "$conf"
}

#Check users who can log in and users without password
function get_user(){
echo -e "#Users who can log in and users without password:“
/usr/bin/w #Current online user information
user=$(cat /etc/passwd|awk -F":" '$7 ~"/bin/bash"{print $1}') #Login user
echo -e "Users who can log in:\n$user"
echo "Password user not set:"
for i in $user ;do
cat /etc/shadow|grep $i|awk -F":" '$2 ~"!!"{print $1,$2}' #User with no password set
done
}

#Check scheduled tasks
function get_corn(){
echo -e "#Scheduled tasks:“
user=` cat /etc/passwd |awk -F":" '$7 ~"/bin/bash"{print $1}' ` #Get user list
for cronuser in $user;do
    crontab -l -u $cronuser > /dev/null 2>&1
    if [ $? -eq 0 ];then
        echo "$cronuser"
        echo "######"
        crontab -l -u $cronuser | grep -vE "^#|^$"
        echo "######"
    else
        echo "user $cronuser No scheduled tasks exist"
    fi
done
}

#Check process status
function get_process(){
echo -e "#Is there a zombie process“
if  [[ $(ps -aux | grep Zs |grep -v grep | wc -l ) -ge 1 ]];then #Check if zombie process exists
    echo "Zombie process exists"
    ps -aux |grep Zs |grep -v grep #Output zombie process information
else
    echo "There is no zombie process"
fi
echo -e "Memory usage TOP10 To list:"
ps aux | awk '{print $2, $4, $6, $11}' | sort -k3rn | head -n 10 #Get the list of top 10 memory utilization
echo -e "CPU Utilization ratio TOP10 To list:"
top b -n1 | head -17 | tail -11 #Get the list of top 10 CPU utilization
}

#Obtain system Patrol information and write it to local patrol log
function check_gather_log(){
if [ ! -x "$local_log_path" ];then #Determine whether the local log storage path exists
    mkdir "$local_log_path" #Create local log storage path
fi
if [ -f "$local_log_path/$LogName.log" ];then #Determine whether the log file exists
    rm -rf $local_log_path/$LogName.log #Delete existing log file
fi
touch $local_log_path/$LogName.log #Create log file
get_status>>$local_log_path/$LogName.log #Call the corresponding method and write the patrol result to the log file
get_disk_status>>$local_log_path/$LogName.log
get_cup_status>>$local_log_path/$LogName.log
get_memory_status>>$local_log_path/$LogName.log
get_ssh_deny>>$local_log_path/$LogName.log
get_auto_start_status>>$local_log_path/$LogName.log
get_user>>$local_log_path/$LogName.log
get_corn>>$local_log_path/$LogName.log
get_process>>$local_log_path/$LogName.log
}

#Perform inspection tour
check_gather_log

#Perform log retrieval
scp_log_to_main_server="scp $local_log_path/$LogName.log root@$main_ipaddr:$main_log_path"

#Calling the password verification process of Expect tool auxiliary log return
/usr/bin/expect<<EOF
    set timeout 20
    spawn $scp_log_to_main_server
    expect {
        "*yes/no)?" 
        { 
            send "yes\n"
            "*password:*" {send "main_passwd\n"}
        } 
        "*password:"         
        {
            send "$main_passwd\n"
        }
    }
    expect "*password:"  { send "$main_passwd\n" }
    expect "100%"
    expect eof
EOF

4.2 script preparation

Set the executable permission of the file: chmod +x check_command.sh
Copy file to target path: cp check_command.sh /tmp/check_script/

5. Issuing script implementation

5.1. Create the script storage path on the target server

File name: check? Expect? Mkdir.sh

#!/usr/bin/expect
set timeout 30
set server_ip [lindex $argv 0]
set server_passwd [lindex $argv 1]
set CheckScriptPath [lindex $argv 2]

spawn ssh -o StrictHostKeyChecking=no root@$server_ip
    expect {
        "*yes/no)?" 
        { 
            send "yes\n"
            "*password:*" {send "server_passwd\n"}
        } 
        "*password:*"         
        {
            send "$server_passwd\n"
        }
    }
    expect "*password:*"  { send "$server_passwd\n" }
    expect "*#" { send "mkdir $CheckScriptPath\n"}
    expect eof
exit

5.2 issue patrol script to target server

File name: check menu expect menu upload.sh

#!/usr/bin/expect
set timeout 30
set server_ip [lindex $argv 0]
set server_passwd [lindex $argv 1]
set check_script_path [lindex $argv 2]

spawn scp $check_script_path/check_command.sh root@$server_ip:$check_script_path
    expect {
        "*yes/no)?" 
        { 
            send "yes\n"
            "*password:*" {send "server_passwd\n"}
        } 
        "*password:*"         
        {
            send "$server_passwd\n"
        }
    }
    expect "*password:*"  { send "$server_passwd\n" }
    expect "100%"
    expect eof
exit

5.3 executing patrol script on target server

File name: check menu expect menu do.sh

#!/usr/bin/expect    
set timeout 20
set server_ip [lindex $argv 0]
set server_passwd [lindex $argv 1]
set check_script_path [lindex $argv 2]
set main_ipaddr [lindex $argv 3]
set main_passwd [lindex $argv 4]

spawn  ssh -o StrictHostKeyChecking=no root@$server_ip
    expect {
        "*yes/no)?" 
        { 
            send "yes\n"
            "*password:*" {send "server_passwd\n"}
        } 
        "*password:"         
        {
            send "$server_passwd\n"
        }
    }
    expect "*password:"  { send "$server_passwd\n" }
    expect "*#" { send "cd $check_script_path;./check_command.sh $main_ipaddr $main_passwd\n" }
    expect eof
exit

5.4 control script distribution and execution

File name: check? Ctrl 2.sh

#!/bin/bash
#
#Loading parameter
login_info=$1 #Parameter 1, server file to be patrolled
main_ipaddr=$2 #Parameter 2: IP address of the target server for log upload
main_passwd=$3 #Parameter 3: log upload target server login password 
main_log_path="/tmp/checklog_directory" #Log upload target server log storage path
check_script_path="/tmp/check_script" #Patrol script file storage path

#Detect the number of parameters. If the number of parameters is abnormal, the prompt will be output
if [ $# -ne 3 ];then
    echo -e "parameters if fault!\n"
    exit;
fi

#Analyze login list
cat $login_info |while read line
do
server_ip=` echo $line |awk '{print $1}' ` #Read target server IP
server_passwd=` echo $line |awk '{print $2}' ` #Read target server password
echo -e "\n######Current server: $server_ip######\n"
./check_expect_mkdir.sh $server_ip $server_passwd $check_script_path #Create script store path
./check_expect_upload.sh $server_ip $server_passwd $check_script_path #Issue test script
./check_expect_do.sh $server_ip $server_passwd $check_script_path $main_ipaddr $main_passwd #Execute patrol script

done

6. Test execution

Script execution:. / check ﹣ Ctrl 2.sh server ﹣ info ﹣ file.txt 192.168.52.3 123456


Execution result:

Log content:

Tags: Linux network shell yum

Posted on Tue, 18 Feb 2020 05:31:39 -0500 by nicdp