1, Background of ELK birth

1.1 before ELK analysis log

What are the pain points of operation and maintenance before there is no log analysis tool?

1. In case of production failure, the O & M needs to constantly view various logs for analysis? Is there no clue;
2. If there is an error when the project goes online, how to quickly locate the problem? What if there are too many backend nodes and logs are scattered;
3. Developers need to view the log in real time, but they don't want to give the login permission to the server. What should they do? Do you help developers get logs every day;
4. How to quickly extract the data we want from massive logs? For example: PV, UV, TOP10 URL? If the amount of log data analyzed is large, the query speed will be slow and the difficulty will increase, and finally we will not be able to quickly obtain the desired indicators.
5. CDN company needs to keep analyzing logs. What is the analysis? Mainly analyze the hit rate. Why? Because the hit rate we promise users is more than 90%. If it does not reach 90%, we have to analyze why the data is not hit and cached;

1.2 after using ELK to analyze logs

All the above pain points can be solved by using the log analysis system ELK. Through ELK, all discrete servers can be collected under one platform; Then extract the desired content, such as error information and warning information. When this information is filtered, an alarm will be given immediately. After the alarm, the operation and maintenance personnel can immediately locate which machine, which business system and what problems have occurred.

2, ELK technology stack

2.1 what is ELK

• ELK is not a single technology, but a set of technology combination, which is composed of elasticsearch, logstash and kibana.

• ELK is an open source, free and powerful log analysis and management system.

• ELK can collect, filter and clean our system logs, website logs, application service logs and other logs, and then store them centrally, which can be used for real-time retrieval and analysis.

    E: elasticsearch Data storage;
    L: logstash Data collection, data cleaning and data filtering;
    K: kibana Data analysis and data display;
  

2.2 what is EFK

Simply put, Logstash is replaced with filebeat. Why do you want to replace it?
Because logstash is developed based on JAVA, it will occupy a lot of business system resources when collecting logs, thus affecting the normal online business. Replacing filebeat, a lightweight log collection component, will make the operation of the business system more stable.

2.3 what is EFLK

Due to the limited data processing capacity of Filebeat, it can only process json format data. The data processing capacity of Logstash is relatively strong. The following architecture can be formed to allow Filebeat to collect data, further process Logstash, and then write it to Elasticsearch

2.4 what logs does efk collect

Agent: Haproxy,Nginx
web: Nginx,Tomcat,Httpd,PHP
db: mysql,redis,mongo,elasticsearch
 Storage: nfs,glusterfs,fastdfs
 System: message,security
 Business: app

3, Elasticsearch

Elasticsearch is a distributed and RESTful search and data analysis engine. Its main functions are data storage, data search and data analysis.

3.1 ES related terms

3.1.1 Document

Document document is some data that users exist in ES. It is the smallest unit stored in ES. (similar to a row of data in a table). Each document has a unique ID, which can be specified by itself. If it is not specified, es will be generated automatically.

3.1.2 Index

An index is actually a collection of documents. (it is similar to a table in a database)

3.1.3 field file

In ES, Document is a Json Object. A Json Object is actually composed of multiple fields. Each field has different data types.

Common field data types

character string: text,keyword. 
Numeric: long,integer,short,byte,double,float
 Boolean: boolean
 Date: date
 Binary: binary
 Range type:
integer_range,float_range,long_range,double_range,
date_range

3.1.4 summary

Many Document documents are stored in an index. A Document is a json object. A json object is composed of multiple different or the same filed fields;

3.2 ES operation mode

The operation of ES is different from our traditional database operation. It operates through restful API. In fact, it essentially changes our resource state through http; Specify the resources to be operated through URI, such as Index and Document; Specify the methods to be operated through Http Method, such as GET, POST, PUT and DELETE;
There are two common ways to operate ES: Curl and Kibana DevTools

3.2.1 curl command operation ES

# Save data
[root@es-node1 ~]# curl -XPOST  \
> 'http://127.0.0.1:9200/bertwu_index/_doc/1' \
> -H "Content-Type: application/json" \
> -d '{
> "name":"tom",
> "age":18,
> "salary": 1000000
> }'
{"_index":"bertwu_index","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
[root@es-node1 ~]# 

# get data
[root@es-node1 ~]# curl -XGET "http://127.0.0.1:9200/bertwu_index/_doc/1"
{"_index":"bertwu_index","_type":"_doc","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{
"name":"tom",
"age":18,
"salary": 1000000

3.2.2 Kibana operation ES

3.2.2.1 Es installation

[root@es-node3 ~]# rpm -ivh elasticsearch-7.8.1-x86_64.rpm 
[root@es-node3 ~]# vim /etc/elasticsearch/jvm.options # If the memory is small, you can modify the minimum heap memory and the maximum heap memory appropriately
-Xms512m
-Xmx521m
[root@es-node3 ~]# systemctl daemon-reload
[root@es-node3 ~]# systemctl enable elasticsearch.service 
[root@es-node3 ~]# systemctl start elasticsearch.service

[root@es-node3 ~]# netstat -lntp | grep java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      1408/java           # Elastic search is the communication port between clusters
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      1408/java           # Port for external communication

3.2.2.2 installation of kibana

# Install kibana
[root@es-node1 ~]# rpm -ivh kibana-7.8.1-x86_64.rpm

# Configure kibana
[root@kibana ~]# vim/etc/kibana/kibana.yml
server.port: 5601 #kibana default listening port
server.host: "0.0.0.0" #kibana listening address segment
server.name: "kibana.bertwu.net" # domain name

elasticsearch.hosts:["http://localhost:9200"] # kibana gets data from the coordinating node

# elasticsearch.hosts: ["http://172.16.1.161:9200","http://172.16.1.162:9200","http://172.16.1.163:9200 "] please change this form to form a cluster

i18n.locale: "zh-CN" #kibana Sinicization

# Start kibana
[root@kibana ~]# systemctl start kibana
[root@kibana ~]# systemctl enable kibana

3.3 ES index API

es has a special Index API for creating, updating and deleting index configurations
The api for creating indexes is as follows:

3.3.1 create index

# Create index
PUT /bertwu_index
#View all existing indexes
GET _cat/indices

3.3.2 delete index

DELETE / index name

3.4 ES documentation API

3.4.1 creating documents

POST /bertwu_index/_doc/1
{
  "username":"Di Renjie",
  "age":18,
  "role":"shooter"
}

3.4.2 query documents

Query the document, and specify the document id to query

Query all documents under the index with _search

Query specific documents under the index:

GET /bertwu_index/_search
{
  "query": {
    "match": {
      "role": "shooter"
    }
  }
  
}

3.4.3 batch creation of documents

POST _bulk
{"index":{"_index":"tt","_id":"1"}}  # Create index tt id 1
{"name":"oldxu","age":"18"} # Add with
{"create":{"_index":"tt","_id":"2"}} # Create a document with id 2 under tt index
{"name":"oldqiang","age":"30"}
{"delete":{"_index":"tt","_id":"2"}} # Delete the document with id 2 under tt index
{"update":{"_id":"1","_index":"tt"}} # Update the document with id 1 under index tt
{"doc":{"age":"20"}}

3.4.4 batch query documents

es allows you to query multiple documents at once through _mget.

GET _mget
{
  "docs": [
    {
     "_index": "tt",
      "_id": "1"
    },
    {
      "_index": "tt",
      "_id": "2"
    }
  ]
}

4, Elasticsearch cluster

es naturally supports cluster mode. The advantages of cluster mode are as follows:

1. It can increase the capacity of the system, such as memory and disk, so that the es cluster can support PB level data;
2. The system availability can be improved. Even if some nodes stop serving, the whole cluster can still serve normally

How to join:
ELasticsearch cluster is composed of multiple nodes. The cluster name is set through cluster.name and used to distinguish other nodes
Cluster, each node specifies the name of the node through node.name.

4.1 deployment environment

Host name	Internet IP WAN	Intranet IP LAN
es-node1	10.0.0.161	172.16.1.161
es-node2	10.0.0.162	172.16.1.162
es-node3	10.0.0.163	172.16.1.163

4.2 installation of ES and java

# yum install java -y
# rpm -ivh elasticsearch-7.8.1-x86_64.rpm

4.2.1 node1 node configuration

First, stop the es and delete the experimental test data in the es, otherwise it will affect the creation of the cluster

[root@es-node1 ~]# rm -rf /var/lib/elasticsearch/*

[root@es-node1 ~]# vim /etc/elasticsearch/elasticsearch.yml 
[root@es-node1 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml 
cluster.name: my-application # Cluster name. The cluster names of all nodes participating in the cluster should be consistent
node.name: es-node1 # Node name
path.data: /var/lib/elasticsearch # es data storage path
path.logs: /var/log/elasticsearch  # es log storage path

#bootstrap.memory_lock: true # Do not use swap partitions

network.host: 172.16.1.161 # Listen to local ip
http.port: 9200 # Listen on local port
discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"] #Cluster host list
cluster.initial_master_nodes: ["172.16.1.161", "172.16.1.162","172.16.1.163"] # Only hosts in the list are elected when the cluster is started for the first time

4.2.2 node2 node configuration

[root@es-node2 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml 
cluster.name: my-application
node.name: es-node2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.16.1.162
http.port: 9200
discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"]
cluster.initial_master_nodes: ["172.16.1.161", "172.16.1.162","172.16.1.163"]

4.2.3 node3 node configuration

[root@es-node3 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml 
cluster.name: my-application
node.name: es-node3
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.16.1.163
http.port: 9200
discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"]
cluster.initial_master_nodes: ["172.16.1.161", "172.16.1.162","172.16.1.163"]

4.3 ES cluster health check

Cluster Health obtains the health status of the cluster. The whole cluster status includes the following three types:

1. green health status means that all primary and secondary partitions are normally allocated
2. yellow means that all primary partitions are allocated normally, but some replica partitions are not allocated normally
3. red has an unallocated primary partition, which indicates that the index is incomplete and there may be a problem with writing. (but it does not mean that data cannot be stored and read)
   Check whether the ES cluster is running normally through curl and Cerebro;

4.3.1 curl check can be performed on any host in curl cluster

[root@es-node1 elasticsearch]# curl http://172.16.1.161:9200/_cluster/health?pretty=true
{
  "cluster_name" : "my-application",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}


# User defined monitoring items can be passed to zabbix monitoring
curl  -s http://172.16.1.161:9200/_cluster/health?pretty=true | grep "status" | awk -F '"' '{print $4}'

4.3.2 Cerebor checking cluster status

Viewing the cluster status through curl command is not intuitive, but can be viewed with the visual graphical tool Cerebor
Cerebor Download

[root@es-node1 ~]# rpm -ivh cerebro-0.9.4-1.noarch.rpm
[root@es-node1 ~]# vim/etc/cerebro/application.conf
data.path: "/var/lib/cerebro/cerebro.db"
#data.path = "./cerebro.db"
[root@es-node1 ~]# systemctl start cerebro
[root@es-node1 ~]# netstat -lntp | grep java
tcp6       0      0 :::9000                 :::*                    LISTEN      4646/java     

5, ES cluster node type

Cluster State
Master
Data
Coordinating

5.1 Cluster State

cluster state: cluster related data is called cluster state; Will be stored in each node, mainly including the following information:

1. Node information, such as node name, node connection address, etc
2. Index information, such as index name, index configuration information, etc

5.2 Master

1. There can only be one master node in the ES cluster. The master node is used to control the operation of the whole cluster;
2. The master mainly maintains the Cluster State. When new data is generated, the master will synchronize the data to other nodes;
3. The master node is elected and can be specified as the master node through node.master: true. (default true)
   When we create an index PUT /index through the API, the Cluster State will change, and the Master will synchronize to other nodes;

5.3 Data

1. The node storing data is the data node. By default, all nodes are of data type. Configure node.data: true (the default is true)
2. After creating an index, the data created by the index will be stored in a node. The node that can store data is called a data node

5.4 Coordinating

1. The node that processes the request is the coordinating node. This node is the default role of all nodes and cannot be cancelled
2. The coordinating node mainly routes the request to the correct node for processing. For example, the request to create an index will be routed by coordinating to the master node for processing; When node.master:false and node.data:false are configured, it is the coordinating node

Summary:

master: controls the status of the entire cluster; Maintain cluster state
Node: responsible for storing data. By default, all nodes are data nodes; node.data: false
coordinating: responsible for routing. All nodes have routing functions and cannot be cancelled: node. Master node. Data fasle
Master eligible nodes that can participate in the election;

6, ES cluster fragment replica

6.1 improving ES cluster availability

How to improve the availability of ES cluster system; There are two aspects as follows;

1. Service availability:
   In the case of 1.2 nodes, one node is allowed to stop service;
   2. In the case of multiple nodes, the bad nodes cannot exceed more than half of the cluster;
2. Data availability
   1. Solve the problem through replica replication, so that each node has complete data.
   2. As shown in the figure below, node2 is oldxu_ A complete copy of the index data.
   

6.2 increase the capacity of ES cluster

1. How to increase the capacity of ES cluster system; We need to find a way to evenly distribute the data on all nodes; Introducing shard to solve the problem;
2. What is sharding? A complete piece of data is scattered into multiple shards for storage;
   2.1. Fragmentation is the cornerstone of es supporting Pb level data
   2.2 partial data of the index is stored in slices and can be distributed on any node
   2.3 fragmentation exists on the main fragmentation and replica fragmentation, and replica fragmentation is mainly used to achieve high data availability
   2.4 the data of replica fragmentation is synchronized by the main fragmentation, and there can be multiple, which is used to improve the data reading performance
   Note: the number of primary partitions is specified during index creation and cannot be changed later; The default number of ES7 slices is 1
3. As shown in the following figure: create a cluster with 3 nodes
   oldxu_index index, specifying 3 slices and 1 copy

# Create an index and set master and replica shards 3 1 = 6
PUT /oldxu_index
{
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 1
}
}
}
# Dynamically modify replica fragmentation
PUT /oldxu_index/_settings
{
"number_of_replicas": 9
}

Click more - > create index

Result display

6.3 can adding nodes improve capacity

Question: at present, there are three ES nodes. Can adding a new node improve oldxu_index index data capacity?

Answer: No, because oldxu_ The index has only three partitions and has been distributed on three nodes, so the new fourth node is for oldxu_index is not available. Therefore, it can not improve the data capacity;

6.4 can adding replicas improve read performance

Question: at present, there are 3 ES nodes in total. Can you increase oldxu if you increase the number of replicas_ Read throughput of index;

Answer: No, because the new replicas will still be distributed on node1, node2 and node3, and the same resources are used, which means that when there are read requests, these requests will still be allocated to
Processing on node1, node2 and node3 means that the same hardware resources are used, so the read performance will not be improved

6.5 what should I do if I need to increase the read throughput performance

Answer: to increase the read throughput, you still need to add nodes. For example, when adding three nodes node4, node5 and node6, migrate the original R0, R1 and R2 to the three new nodes respectively. When a read request comes, node4, node5 and node6 will be allocated, which means that there are new CPU, memory and IO, so that the hardware resources of node1, node2 and node3 will not be occupied, Then the read throughput will be really improved at this time;

6.6 copy and slice summary

The number of slices and replica settings are very important and need to be planned in advance

1. If it is too small, the subsequent horizontal expansion cannot be realized by adding nodes;
2. Setting too many shards will result in too many shards distributed on a node, resulting in a waste of resources. Too many fragments will also affect the query performance;
3. Once the number of slices is set, it cannot be modified

7, ES cluster failover and recovery

7.1 what is failover

The so-called failover refers to how the cluster automatically repairs when a node in the cluster fails. The ES cluster is currently composed of three nodes, as shown in the figure below. At this time, the cluster
The status is green

7.2 simulated node failure

Suppose: the service is terminated due to the downtime of the node1 machine. How will the cluster handle it; There are three steps:
1. Re election
2. Main slice adjustment
3. Copy slice adjustment

7.2.1 re-election

Node2 and node3 find that node1 cannot respond; A master election will be initiated after a period of time. For example, node2 is selected as the master node here; At this time, the cluster status changes to Red;

7.2.2 main slice adjustment

node2 found that the main partition P0 was not allocated, and promoted R0 on node3 to the main partition; At this time, all primary partitions are allocated normally, and the cluster status changes to Yellow status;

7.2.3 copy slice adjustment

node2 regenerates the primary partitions P0 and P1 into new replica partitions R0 and R1. At this time, the cluster state changes to Green;

8, ES document routing principle

Es document distributed storage. When a document is stored in the ES cluster, what is the storage principle?
As shown in the figure, how is Document1 stored in fragment P1 when we want to save documents in a cluster? What is the basis for selecting P1?

In fact, there is a document to fragment mapping algorithm, which aims to make all documents evenly distributed on all fragments. What algorithm is it? Random or polling? This is not desirable, because the data needs to be read after storage. In this case, how to read it?
In fact, in ES, the node to which the partition corresponding to the document is stored is calculated by the following formula:

shard = hash(routing) % number_of_primary_shards
# hash algorithm ensures that the data is evenly distributed in the partition
# routing is a key parameter. The default is the document id, which can also be customized.
# number_of_primary_shards primary shards
# Note: this algorithm is related to the number of primary partitions, but once it is determined, the primary partition cannot be changed.
# Because once the master partition is modified, the calculation of Share is completely different

The document data in the index will be evenly distributed in different slices.

8.1 document creation process

8.2 document reading process

8.3 document batch creation process

8.4 document batch reading process

9, ES extended cluster node, extended data node and routing node method

9.1 environmental preparation

Host name	Internet IP WAN	Intranet IP LAN
es-node4	10.0.0.164	172.16.1.164
es-node5	10.0.0.165	172.16.1.165

Expansion node 4 is configured as a storage node

[root@es-node4 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml 
cluster.name: my-application
node.name: es-node-4
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.16.1.164
http.port: 9200
discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"] # From these hosts, it is found that the cluster joins the organization, and multiple can be written
node.data: true  # Storage node
node.master: false # Do not participate in master election

The extension node 5 is configured to only act as a routing node Coordinating

[root@es-node5 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml 
cluster.name: my-application
node.name: es-node5
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.16.1.165
http.port: 9200
discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"]
node.data: false # Do not store
node.master: false # Not running

9.2 extension node inspection

Check the status after cluster expansion through cerebor; If the cluster cannot be joined or is rejected, try to delete the file under / var/lib/elasticsearch, and then restart es;

If the existing data node is modified to the Coordinating node; Data cleaning is required, otherwise it cannot be started;

[root@es-node5 ~]# /usr/share/elasticsearch/bin/elasticsearch-node repurpose

10, ES cluster tuning

10.1 kernel parameter optimization

fs.file-max=655360 # Set the maximum number of open file descriptors in the system. It is recommended to modify it to 655360 or higher,
vm.max_map_count = 262144 # It is used to limit the size of virtual memory that a process can have. It is recommended to modify it to 262144 or higher.
net.core.somaxconn = 32768 # Full connection pool size
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1000 65535
net.ipv4.tcp_max_tw_buckets = 400000

[root@node1 ~]# sysctl -p

# Adjust the maximum number of user processes (nproc) and the maximum open file descriptor (nofile)
[root@node ~]# rm -f /etc/security/limits.d/20-nproc.conf # Delete default nproc settings file
[root@node ~]# vim /etc/security/limits.conf
* soft nproc 20480
* hard nproc 20480
* soft nofile 65536
* hard nofile 65536

10.2 es configuration parameter optimization

# 1. Lock the physical memory address and avoid es using swap to exchange partitions. Frequent exchange will lead to high IOPS.
[root@es-node ~]# vim /etc/elasticsearch/elasticsearch.yml
bootstrap.memory_lock: true
#2. Configure elasticsearch startup parameters
[root@es-node ~]# sed -i '/\[Service\]/a LimitMEMLOCK=infinity' /usr/lib/systemd/system/elasticsearch.service
[root@es-node ~]# systemctl daemon-reload
[root@es-node ~]# systemctl restart elasticsearch

10.3 JVM optimization

jvm memory should be estimated according to the amount of data to be stored in node. In order to ensure performance, there is a recommended ratio between memory and data: like general log files, 1G memory can store 48G~96GB data; The maximum jvm heap memory should not exceed 31GB;

The second is the storage capacity of the main partition, which is controlled at 30-50GB;

It is assumed that the total amount of data is 1TB, 3 node nodes and 1 replica; Then the actual storage size is 2TB, because there is a replica; 2TB / 3 = 700GB, and then each node needs to reserve 20% empty space
It means that each node should store about 700/0.8=875GB of data;

Calculated according to the ratio of memory to stored data:
875GB/48=18GB, less than 31GB. Because 31*48=1.4TB and each Node can store 1.4TB data, three nodes are enough;
875GB/30=29/2=15 main segments. Because we should try to control the size of the main partition to 30GB

Assuming that the total amount of data is 2TB, three node nodes and one replica, 4tb / 3 = 1365gb needs to be stored, and then 20% of the space needs to be reserved for each node, which means that each node needs to store about
1365GB/0.8=1706GB data.
Calculated according to the ratio of memory to stored data:
1706GB/48G=35 is greater than 31GB. Because 31*48=1.4TB and each node can store 1.4TB data, at least 4 nodes are required
1706 / 30 = 56 / 2 = 28 main segments

[root@es-node ~-]# vim /etc/elasticsearch/jvm.options
-Xms 31g #Minimum heap memory
-Xmx31g #Maximum heap memory

#Modify to an appropriate value according to the memory size of the server. It is generally set to half of the physical memory of the server, but the maximum can not exceed 32G
#Server configuration with data volume of about 1TB per day
16C  64G   6T   3 platform ECS