Log analysis department ELK

1, ELK overview

-Elastic search: responsible for log retrieval and storage
-Logstash: responsible for the collection, analysis and processing of logs
-Kibana: responsible for the visualization of logs
-ELK is a complete set of solutions. It is the acronym of three software products, which are used by many companies, such as Sina, Ctrip, Huawei, meituan, etc
-These three kinds of software are open source software, which are usually used together, and belong to Elastic.co company, ELK for short

ELK components can be used to solve problems in the operation and maintenance of massive log system

-Centralized query and management of distributed log data
-System monitoring, including the monitoring of system hardware and application components
-Security information and event management
-Report function

2, Elasticsearch

1. Overview of elasticsearch

Elasticsearch is a Lucene based search server. It provides a distributed multi-user full-text search engine, elastic search, a Web interface based on restful API
It is an open source software developed in Java and using Apache license terms. It is a popular enterprise search engine. Designed for cloud computing, it can achieve real-time search, stable, reliable, fast, easy to install and use

main features

-Real time analysis, document oriented, distributed real-time file storage
-High availability, easy to expand, support cluster, Shards and replication
-Friendly interface and support JSON
-No typical transaction
-Is a document oriented database

Noun interpretation

Node: a node with an ES server installed
Cluster: a cluster composed of multiple nodes
Document: - basic information units that can be searched
Index: a collection of documents with similar characteristics
Type: one or more types can be defined in an index
File: is the smallest unit of ES, equivalent to a - column of data
Shards: index Shards. Each Shard is a Shard
Replicas: copy of index

Comparison with relational database

2. Preparation for deploying Elasticsearch
1) Configure yum warehouse on springboard machine
Copy the cloud disk RPM package / elk directory to the springboard machine

[root@ecs-proxy public]# ls elk/
elasticsearch-6.8.8.rpm    logs.jsonl.gz
filebeat-6.8.8-x86_64.rpm  logstash-6.8.8.rpm
head.tar.gz                metricbeat-6.8.8-x86_64.rpm
[root@ecs-proxy ~]# cp -a elk /var/ftp/localrepo/elk
[root@ecs-proxy ~]# cd /var/ftp/localrepo/
[root@ecs-proxy localrepo]# createrepo --update .

2) Purchase virtual machine

hostIP addressto configure
es-0001192.168.1.41Minimum configuration: 2-core 2G
es-0002192.168.1.42Minimum configuration: 2-core 2G
es-0003192.168.1.43Minimum configuration: 2-core 2G
es-0004192.168.1.44Minimum configuration: 2-core 2G
es-0005192.168.1.45Minimum configuration: 2-core 2G

3. Install a stand-alone board elasticsearch

[root@es-0001 ~]# vim /etc/hosts	es-0001
[root@es-0001 ~]# yum install -y java-1.8.0-openjdk elasticsearch
[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml
55:  network.host:
[root@es-0001 ~]# systemctl enable --now elasticsearch


[root@es-0001 ~]# curl
  "name" : "War Eagle",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.3.4",
    "build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
    "build_timestamp" : "2016-06-30T11:24:31Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  "tagline" : "You Know, for Search"

4. Install the cluster board elasticsearch

status: green
-Cluster status, green is normal
-Yellow indicates a problem but not very serious, and red indicates a serious fault
-Indicates the number of nodes in the cluster
-Number of nodes used to store data

es-0001... es-0005 all hosts must perform the following operations

[root@es-0001 ~]# vim /etc/hosts	es-0001	es-0002	es-0003	es-0004	es-0005
[root@es-0001 ~]# yum install -y java-1.8.0-openjdk elasticsearch
[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml
17:  cluster.name: my-es #Cluster name
23:  node.name: es-0001 # Native hostname
55:  network.host:  #Listening address, here are all
68:  discovery.zen.ping.unicast.hosts: ["es-0001", "es-0002"]          #founder
[root@es-0001 ~]# systemctl enable --now elasticsearch


[root@es-0001 ~]# curl
  "cluster_name" : "my-es",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 5,
  "number_of_data_nodes" : 5,
   ... ...

Note: the cluster is the same for users, so the ip accessed can be the ip of any node

3, Cluster profile

1. Overview of head plug-in

-It shows the topology of the ES cluster and can be used for index and node level operations
-It provides a - set of query API s for clusters and returns the results in json and tabular form
-It provides some shortcut menus to show various states of the cluster

2. Schematic diagram of plug-in
Purchase virtual machine

hostIP addressto configure
web192.168.1.48Minimum configuration: 1 core 1G

3. Install the plug-in
Install apache and deploy the head plug-in

[root@web ~]# yum install -y httpd
[root@web ~]# tar zxf head.tar.gz
[root@web ~]# mv elasticsearch-head /var/www/html/head
[root@web ~]# systemctl enable --now httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

4.es-0001 access authorization
[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml

#The configuration file is appended at the end
http.cors.enabled : true
http.cors.allow-origin : "*"
http.cors.allow-methods : OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length
[root@es-0001 ~]# systemctl restart elasticsearch.service 

5. Test
Use Huawei ELB to publish the 80 port of web service and the 9200 port of es-0001 to the Internet and access them through a browser
1) Purchase ELB and purchase ip and bind
! [insert picture description here]( https://img-blog.csdnimg.cn/b206534b800e4cafa38d686f9621cd01.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAa2FsaV95YW8=,size_20,color_FFFFFF,t_70,g_se,x_16

2) Add two monitors (809200)

3) Listen to the web and Head plug-ins respectively
4) Browser access connection plug-in

Note: * is the master, which is automatically assigned

4, Elasticsearch access request plugin (curl)

Simple management overview of cluster API s
1. Browser access
Elastic icsearch is accessed using the http protocol
The http request consists of three parts
-They are: request line, message header and request body
-Request line: method request URI HTTP -- version CRLF
http request method
-Common methods: GET, POST, HEAD

2. Command line access
Request method used by elasticseal Ch
Delete - delete
Change - POST
Query GET
The data interacting with elastic icsearch shall be in json format

3.curl command access
In linux, curl is a file transfer tool that uses URL rules to work on the command line. It can be said to be a very powerful http command line tool. It supports a variety of request modes, custom request headers and other powerful functions. It is a comprehensive tool
Use format:
curl -X request method http: / / request address
curI -H custom request header http: / / request address

4.curl usage
_cat keyword is used to query cluster status, node information, etc. - display detailed information (? v) and help information (? he lp)

#Query supported keywords
[root@es-0001 ~]# curl -XGET http://es-0001:9200/_cat/
#Check the specific information
[root@es-0001 ~]# curl -XGET http://es-0001:9200/_cat/master
#Show details? v
[root@es-0001 ~]# curl -XGET http://es-0001:9200/_cat/master?v
#Show help information? Help
[root@es-0001 ~]# curl -XGET http://es-0001:9200/_cat/master?help

5. Use PUT method to create index

Specify the name of the index, specify the number of slices, and specify the number of copies
The PUT method is used to create the index. After the index is created, it is verified by the head plug-in
[ root@es -0001 ~]# curl -XPUT -H "Content-Type: application/json" http://es-0001:9200/tedu -D \ '{"Settings": {"index": {"number_of_shards": 5, # data is divided into several pieces "number_of_replicas": 1 # several copies}}}'

6. Add data

[root@es-0001 ~]# curl -XPUT -H "Content-Type: application/json" http://es-0001:9200/tedu/teacher/1 -d \
  "occupation": "poet",
  "name": "Li Bai",
  "title": "Shixian",
  "years": "Tang"

7. Query data

[root@es-0001 ~]# curl -XGET http://es-0001:9200/tedu/teacher/1?pretty

8. Modify data

[root@es-0001 ~]# curl -XPOST -H "Content-Type: application/json" \
                  http://Es-0001:9200 / Tedu / teacher / 1 / _update - D '{"Doc": {"year": "701"}}'

9. Delete data

# Delete one
[root@es-0001 ~]# curl -XDELETE http://es-0001:9200/tedu/teacher/1
# Delete index
[root@es-0001 ~]# curl -XDELETE http://es-0001:9200/tedu

5, kibana

1. Introduction to kibana

What is kibana
-Data visualization platform tools


-Flexible analysis and visualization platform
-Charts summarizing traffic and data in real time
-Display an intuitive interface for different users
-Instant sharing and embedded dashboards

2.kibana installation configuration
1) # purchase virtual machine

hostIP addressto configure
kibana192.168.1.46Minimum configuration: 1 core 1G

2) Install kibana

[root@kibana ~]# vim /etc/hosts	es-0001	es-0002	es-0003	es-0004	es-0005	kibana
[root@kibana ~]# yum install -y kibana
[root@kibana ~]# vim /etc/kibana/kibana.yml
02  server.port: 5601             #port
07  server.host: ""        #Allow access to ip
28  elasticsearch.hosts: ["http://es-0002:9200", "http://es-0003:9200 "] # default web address 
37  kibana.index: ".kibana"      #Indexes
40  kibana.defaultAppId: "home"  #Default home menu
113 i18n.locale: "zh-CN"         #language
[root@kibana ~]# systemctl enable --now kibana

3. Bind the elastic public IP and verify through the WEB browser
ip has been bound in front
##6, kibana charting

1. Overview of importing data
Data import conditions:

-json format must be specified content - type: application I on / json
-Import Keywords: _bulk

  • HTTP method: POST

2. Import log data test
Copy the cloud disk / var/ftp/localrepo/elk/logs.jsonl.gz to the springboard

[root@ecs-proxy ~]# cd  /var/ftp/localrepo/elk/
[root@ecs-proxy ~]# cp logs.jsonl.gz /root/
[root@ecs-proxy ~]# gunzip logs.jsonl.gz 
[root@ecs-proxy ~]# curl -XPOST -H "Content-Type: application/json" --data-binary @logs.jsonl 

3. Insert picture description here to view
4. Draw flow chart
1) Choose to use Elasticsearch
These operations are not required if there are real-time logs
2) Another analysis diagram
Pie chart
You can also select subgraphs

Tags: Java Linux Operation & Maintenance ElasticSearch

Posted on Tue, 23 Nov 2021 19:21:37 -0500 by EnDee321