Daily monitoring log generation

Daily monitoring log generation

I. Purpose

In order to fully automate data collection on large data platforms, daily monitoring logs need to be generated for operation and maintenance personnel to view.

2. Subjects

Daily monitoring log file generation

3. Procedures and Methods

[Task 1]

Task Description: Write shell scripts to automate daily monitoring log generation

Achieving results:

1) Executing scripts that automate data collection will generate the following data [each execution date is updated]

The date of file 2021-09-24 20:56:16 INFO >> JSON was successfully replaced by 20200901 with 20200902!

2021-09-2420:56:56 INFO >>Data collected from mysql to hive successfully! 62 successful data collected

Screenshot of log file contents

IV. Instruments, Equipment and Tools

Equipment InstrumentsComputer room
operating systemWindows Operating System
toolConnection tools: sqlyog, navicat
network environmentCapable of networking

5. Think Questions

1)$? What is the function of linux and what are the special symbols besides this one?

0 : on yes this b a s h writing piece name , individual position number Of , can straight meet send use number word , but two position number with upper , be have to must send use symbol Number come Include live , as 0: This is the bash file name. For digits, you can use numbers directly, but for more than two digits, you must enclose them with the {} symbol, such as 0: This is the bash file name. For digits, you can use numbers directly, but for digits above two, you must enclose them with symbols, such as {10}.

$?: is the return value from the previous instruction. Success is 0 and failure is 1. Generally speaking, a process of a UNIX(linux) system ends by executing a system call exit(). This return value is the status value. It is passed back to the parent process to check the execution status of the child process. A normal instruction program returns a value of 0 if execution is successful; failure is 1.

$*: The content of all script parameters: that is, the parameters calling this bash shell.

@ : base book upper and upper Noodles mutually with . only No too yes " @: basically the same as above. It's just " @: Basically the same as above. Only'*'returns a string with multiple extra spaces in it.'$@' returns multiple strings.

$#: Returns the number of all script parameters.

$$: The PID (ProcessID) of the Shell itself, that is, the PID of the current process.

$!:PID of the last running background Process for Shell

$-: Flag overview set with Set command

6. Implementation Process

Turn on hadoop1,hadoop2,hadoop3

Connect to the database of hadoop1

Import Three Tables

Open Cluster

Go into hive to create the test table and create four tables

create database test;

use test;

create table `order_status_log20200901`(`id` string,`order_id` string,`order_status` string,`operate_time` string) row format delimited fields terminated by ',';

create table `order_status_log20200902`(`id` string,`order_id` string,`order_status` string,`operate_time` string) row format delimited fields terminated by ',';

create table `order_status_log20200903`(`id` string,`order_id` string,`order_status` string,`operate_time` string) row format delimited fields terminated by ',';

create table `order_status_log20200904`(`id` string,`order_id` string,`order_status` string,`operate_time` string) row format delimited fields terminated by ','; 

Upload log.json file to / root/datax/job directory using winscp

log.json weighted limit

chmod +x log.json

Then modify log.json

{
    "job": {
        "setting": {
            "speed": {
                 "channel": 3
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0.02
            }
        },
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "root",
                        "password": "123456",
                        "column": [
                            "id",
                            "order_id",
                            "order_status",
                            "operate_time"
                        ],
                        "splitPk": "id",
                        "connection": [
                            {
                                "table": [
                                    "order_status_log20200901"
                                ],
                                "jdbcUrl": [
     "jdbc:mysql://192.168.174.10:3306/test"
                                ]
                            }
                        ]
                    }
                },
               "writer": {
                    "name": "hdfswriter",
                    "parameter": {
                        "defaultFS": "hdfs://192.168.174.11:8020",
                        "fileType": "text",
                        "path": "/user/hive/warehouse/test.db/order_status_log20200901",
                        "fileName": "order_status_log20200901",
                        "column": [
                            {
                                "name": "id",
                                "type": "STRING"
                            },
                            {
                                "name": "order_id",
                                "type": "STRING"
                            },
                            {
                                "name": "order_status",
                                "type": "STRING"
                            },
                            {
                                "name": "operate_time",
                                "type": "STRING"
                            }
                        ],
                        "writeMode": "append",
                        "fieldDelimiter": ",",
                        "compress":"GZIP"
                    }
                }
            }
        ]
    }
}

Upload auto1.sh file to / root/sh directory using winscp

Weighted Limit

chmod +x auto2.sh

Write auto2.sh

#!/bin/bash
log_dir=/root/log
log_prefix=order_status
function write_log(){
  log_format=`date '+%Y-%m-%d %H:%M:%S'`
  echo "${log_format} $1 >> ${2}" >> ${log_dir}/${log_prefix}.`date '+%Y-%m-%d'`.log
  [ $1 == "ERROR" ] && exit 1
}
date=`cat /root/sh/date`
afterday_timestamp=$[`date -d "${date}" +%s`+86400]
afterday=`date -d @${afterday_timestamp} +%Y%m%d`
sed -i "s/order_status_log${date}/order_status_log${afterday}/g" /root/datax/job/log.json
write_log INFO "log.json The date of the file is ${date}Successfully replaced by ${afterday}!!!"
sed -i "s/${date}/${afterday}/g" /root/sh/date
python /root/datax/bin/datax.py /root/datax/job/log.json

if [ $? -eq 0 ];then
  result=`hive -e "use test;select count(id) from order_status_log${afterday}"`
  write_log INFO "Data from mysql Successfully collected!!! Data collected are ${result}strip"
else
  write_log ERROR "Data collection failed!!!, please check now"
fi

Create date under sh

touch date
#And set 20200901

Create folder log log file in root directory

mkdir /root/log

Final Execution

sh auto2.sh
cat /root/log/order_status.2021-09-30.log

Tags: Linux Operation & Maintenance hive

Posted on Thu, 30 Sep 2021 13:08:09 -0400 by jtgraphic