1, Background of ELK birth
1.1 before ELK analysis log
What are the pain points of operation and maintenance before there is no log analysis tool?
- In case of production failure, the O & M needs to constantly view various logs for analysis? Is there no clue;
- If there is an error when the project goes online, how to quickly locate the problem? What if there are too many backend nodes and logs are scattered;
- Developers need to view the log in real time, but they don't want to give the login permission to the server. What should they do? Do you help developers get logs every day;
- How to quickly extract the data we want from massive logs? For example: PV, UV, TOP10 URL? If the amount of log data analyzed is large, the query speed will be slow and the difficulty will increase, and finally we will not be able to quickly obtain the desired indicators.
- CDN company needs to keep analyzing logs. What is the analysis? Mainly analyze the hit rate. Why? Because the hit rate we promise users is more than 90%. If it does not reach 90%, we have to analyze why the data is not hit and cached;
1.2 after using ELK to analyze logs
All the above pain points can be solved by using the log analysis system ELK. Through ELK, all discrete servers can be collected under one platform; Then extract the desired content, such as error information and warning information. When this information is filtered, an alarm will be given immediately. After the alarm, the operation and maintenance personnel can immediately locate which machine, which business system and what problems have occurred.
2, ELK technology stack
2.1 what is ELK
-
ELK is not a single technology, but a set of technology combination, which is composed of elasticsearch, logstash and kibana.
-
ELK is an open source, free and powerful log analysis and management system.
-
ELK can collect, filter and clean our system logs, website logs, application service logs and other logs, and then store them centrally, which can be used for real-time retrieval and analysis.
E: elasticsearch Data storage; L: logstash Data collection, data cleaning and data filtering; K: kibana Data analysis and data display;
2.2 what is EFK
Simply put, Logstash is replaced with filebeat. Why do you want to replace it?
Because logstash is developed based on JAVA, it will occupy a lot of business system resources when collecting logs, thus affecting the normal online business. Replacing filebeat, a lightweight log collection component, will make the operation of the business system more stable.
2.3 what is EFLK
Due to the limited data processing capacity of Filebeat, it can only process json format data. The data processing capacity of Logstash is relatively strong. The following architecture can be formed to allow Filebeat to collect data, further process Logstash, and then write it to Elasticsearch
2.4 what logs does efk collect
Agent: Haproxy,Nginx web: Nginx,Tomcat,Httpd,PHP db: mysql,redis,mongo,elasticsearch Storage: nfs,glusterfs,fastdfs System: message,security Business: app
3, Elasticsearch
Elasticsearch is a distributed and RESTful search and data analysis engine. Its main functions are data storage, data search and data analysis.
3.1 ES related terms
3.1.1 Document
Document document is some data that users exist in ES. It is the smallest unit stored in ES. (similar to a row of data in a table). Each document has a unique ID, which can be specified by itself. If it is not specified, es will be generated automatically.
3.1.2 Index
An index is actually a collection of documents. (it is similar to a table in a database)
3.1.3 field file
In ES, Document is a Json Object. A Json Object is actually composed of multiple fields. Each field has different data types.
Common field data types
character string: text,keyword. Numeric: long,integer,short,byte,double,float Boolean: boolean Date: date Binary: binary Range type: integer_range,float_range,long_range,double_range, date_range
3.1.4 summary
Many Document documents are stored in an index. A Document is a json object. A json object is composed of multiple different or the same filed fields;
3.2 ES operation mode
The operation of ES is different from our traditional database operation. It operates through restful API. In fact, it essentially changes our resource state through http; Specify the resources to be operated through URI, such as Index and Document; Specify the methods to be operated through Http Method, such as GET, POST, PUT and DELETE;
There are two common ways to operate ES: Curl and Kibana DevTools
3.2.1 curl command operation ES
# Save data [root@es-node1 ~]# curl -XPOST \ > 'http://127.0.0.1:9200/bertwu_index/_doc/1' \ > -H "Content-Type: application/json" \ > -d '{ > "name":"tom", > "age":18, > "salary": 1000000 > }' {"_index":"bertwu_index","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1} [root@es-node1 ~]# # get data [root@es-node1 ~]# curl -XGET "http://127.0.0.1:9200/bertwu_index/_doc/1" {"_index":"bertwu_index","_type":"_doc","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{ "name":"tom", "age":18, "salary": 1000000
3.2.2 Kibana operation ES
3.2.2.1 Es installation
[root@es-node3 ~]# rpm -ivh elasticsearch-7.8.1-x86_64.rpm [root@es-node3 ~]# vim /etc/elasticsearch/jvm.options # If the memory is small, you can modify the minimum heap memory and the maximum heap memory appropriately -Xms512m -Xmx521m [root@es-node3 ~]# systemctl daemon-reload [root@es-node3 ~]# systemctl enable elasticsearch.service [root@es-node3 ~]# systemctl start elasticsearch.service [root@es-node3 ~]# netstat -lntp | grep java tcp6 0 0 127.0.0.1:9300 :::* LISTEN 1408/java # Elastic search is the communication port between clusters tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1408/java # Port for external communication
3.2.2.2 installation of kibana
# Install kibana [root@es-node1 ~]# rpm -ivh kibana-7.8.1-x86_64.rpm # Configure kibana [root@kibana ~]# vim/etc/kibana/kibana.yml server.port: 5601 #kibana default listening port server.host: "0.0.0.0" #kibana listening address segment server.name: "kibana.bertwu.net" # domain name elasticsearch.hosts:["http://localhost:9200"] # kibana gets data from the coordinating node # elasticsearch.hosts: ["http://172.16.1.161:9200","http://172.16.1.162:9200","http://172.16.1.163:9200 "] please change this form to form a cluster i18n.locale: "zh-CN" #kibana Sinicization # Start kibana [root@kibana ~]# systemctl start kibana [root@kibana ~]# systemctl enable kibana
3.3 ES index API
es has a special Index API for creating, updating and deleting index configurations
The api for creating indexes is as follows:
3.3.1 create index
# Create index PUT /bertwu_index #View all existing indexes GET _cat/indices
3.3.2 delete index
DELETE / index name
3.4 ES documentation API
3.4.1 creating documents
POST /bertwu_index/_doc/1 { "username":"Di Renjie", "age":18, "role":"shooter" }
3.4.2 query documents
Query the document, and specify the document id to query
Query all documents under the index with _search
Query specific documents under the index:
GET /bertwu_index/_search { "query": { "match": { "role": "shooter" } } }
3.4.3 batch creation of documents
POST _bulk {"index":{"_index":"tt","_id":"1"}} # Create index tt id 1 {"name":"oldxu","age":"18"} # Add with {"create":{"_index":"tt","_id":"2"}} # Create a document with id 2 under tt index {"name":"oldqiang","age":"30"} {"delete":{"_index":"tt","_id":"2"}} # Delete the document with id 2 under tt index {"update":{"_id":"1","_index":"tt"}} # Update the document with id 1 under index tt {"doc":{"age":"20"}}
3.4.4 batch query documents
es allows you to query multiple documents at once through _mget.
GET _mget { "docs": [ { "_index": "tt", "_id": "1" }, { "_index": "tt", "_id": "2" } ] }
4, Elasticsearch cluster
es naturally supports cluster mode. The advantages of cluster mode are as follows:
- It can increase the capacity of the system, such as memory and disk, so that the es cluster can support PB level data;
- The system availability can be improved. Even if some nodes stop serving, the whole cluster can still serve normally
How to join:
ELasticsearch cluster is composed of multiple nodes. The cluster name is set through cluster.name and used to distinguish other nodes
Cluster, each node specifies the name of the node through node.name.
4.1 deployment environment
Host name | Internet IP WAN | Intranet IP LAN |
---|---|---|
es-node1 | 10.0.0.161 | 172.16.1.161 |
es-node2 | 10.0.0.162 | 172.16.1.162 |
es-node3 | 10.0.0.163 | 172.16.1.163 |
4.2 installation of ES and java
# yum install java -y # rpm -ivh elasticsearch-7.8.1-x86_64.rpm
4.2.1 node1 node configuration
First, stop the es and delete the experimental test data in the es, otherwise it will affect the creation of the cluster
[root@es-node1 ~]# rm -rf /var/lib/elasticsearch/*
[root@es-node1 ~]# vim /etc/elasticsearch/elasticsearch.yml [root@es-node1 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml cluster.name: my-application # Cluster name. The cluster names of all nodes participating in the cluster should be consistent node.name: es-node1 # Node name path.data: /var/lib/elasticsearch # es data storage path path.logs: /var/log/elasticsearch # es log storage path #bootstrap.memory_lock: true # Do not use swap partitions network.host: 172.16.1.161 # Listen to local ip http.port: 9200 # Listen on local port discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"] #Cluster host list cluster.initial_master_nodes: ["172.16.1.161", "172.16.1.162","172.16.1.163"] # Only hosts in the list are elected when the cluster is started for the first time
4.2.2 node2 node configuration
[root@es-node2 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml cluster.name: my-application node.name: es-node2 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 172.16.1.162 http.port: 9200 discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"] cluster.initial_master_nodes: ["172.16.1.161", "172.16.1.162","172.16.1.163"]
4.2.3 node3 node configuration
[root@es-node3 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml cluster.name: my-application node.name: es-node3 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 172.16.1.163 http.port: 9200 discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"] cluster.initial_master_nodes: ["172.16.1.161", "172.16.1.162","172.16.1.163"]
4.3 ES cluster health check
Cluster Health obtains the health status of the cluster. The whole cluster status includes the following three types:
- green health status means that all primary and secondary partitions are normally allocated
- yellow means that all primary partitions are allocated normally, but some replica partitions are not allocated normally
- red has an unallocated primary partition, which indicates that the index is incomplete and there may be a problem with writing. (but it does not mean that data cannot be stored and read)
Check whether the ES cluster is running normally through curl and Cerebro;
4.3.1 curl check can be performed on any host in curl cluster
[root@es-node1 elasticsearch]# curl http://172.16.1.161:9200/_cluster/health?pretty=true { "cluster_name" : "my-application", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 1, "active_shards" : 2, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } # User defined monitoring items can be passed to zabbix monitoring curl -s http://172.16.1.161:9200/_cluster/health?pretty=true | grep "status" | awk -F '"' '{print $4}'
4.3.2 Cerebor checking cluster status
Viewing the cluster status through curl command is not intuitive, but can be viewed with the visual graphical tool Cerebor
Cerebor Download
[root@es-node1 ~]# rpm -ivh cerebro-0.9.4-1.noarch.rpm [root@es-node1 ~]# vim/etc/cerebro/application.conf data.path: "/var/lib/cerebro/cerebro.db" #data.path = "./cerebro.db" [root@es-node1 ~]# systemctl start cerebro [root@es-node1 ~]# netstat -lntp | grep java tcp6 0 0 :::9000 :::* LISTEN 4646/java
5, ES cluster node type
Cluster State
Master
Data
Coordinating
5.1 Cluster State
cluster state: cluster related data is called cluster state; Will be stored in each node, mainly including the following information:
- Node information, such as node name, node connection address, etc
- Index information, such as index name, index configuration information, etc
5.2 Master
- There can only be one master node in the ES cluster. The master node is used to control the operation of the whole cluster;
- The master mainly maintains the Cluster State. When new data is generated, the master will synchronize the data to other nodes;
- The master node is elected and can be specified as the master node through node.master: true. (default true)
When we create an index PUT /index through the API, the Cluster State will change, and the Master will synchronize to other nodes;
5.3 Data
- The node storing data is the data node. By default, all nodes are of data type. Configure node.data: true (the default is true)
- After creating an index, the data created by the index will be stored in a node. The node that can store data is called a data node
5.4 Coordinating
- The node that processes the request is the coordinating node. This node is the default role of all nodes and cannot be cancelled
- The coordinating node mainly routes the request to the correct node for processing. For example, the request to create an index will be routed by coordinating to the master node for processing; When node.master:false and node.data:false are configured, it is the coordinating node
Summary:
master: controls the status of the entire cluster; Maintain cluster state
Node: responsible for storing data. By default, all nodes are data nodes; node.data: false
coordinating: responsible for routing. All nodes have routing functions and cannot be cancelled: node. Master node. Data fasle
Master eligible nodes that can participate in the election;
6, ES cluster fragment replica
6.1 improving ES cluster availability
How to improve the availability of ES cluster system; There are two aspects as follows;
- Service availability:
In the case of 1.2 nodes, one node is allowed to stop service;
2. In the case of multiple nodes, the bad nodes cannot exceed more than half of the cluster; - Data availability
1. Solve the problem through replica replication, so that each node has complete data.
2. As shown in the figure below, node2 is oldxu_ A complete copy of the index data.
6.2 increase the capacity of ES cluster
- How to increase the capacity of ES cluster system; We need to find a way to evenly distribute the data on all nodes; Introducing shard to solve the problem;
- What is sharding? A complete piece of data is scattered into multiple shards for storage;
2.1. Fragmentation is the cornerstone of es supporting Pb level data
2.2 partial data of the index is stored in slices and can be distributed on any node
2.3 fragmentation exists on the main fragmentation and replica fragmentation, and replica fragmentation is mainly used to achieve high data availability
2.4 the data of replica fragmentation is synchronized by the main fragmentation, and there can be multiple, which is used to improve the data reading performance
Note: the number of primary partitions is specified during index creation and cannot be changed later; The default number of ES7 slices is 1 - As shown in the following figure: create a cluster with 3 nodes
oldxu_index index, specifying 3 slices and 1 copy
# Create an index and set master and replica shards 3 1 = 6 PUT /oldxu_index { "settings": { "index": { "number_of_shards": 3, "number_of_replicas": 1 } } } # Dynamically modify replica fragmentation PUT /oldxu_index/_settings { "number_of_replicas": 9 }
Click more - > create index
Result display
6.3 can adding nodes improve capacity
Question: at present, there are three ES nodes. Can adding a new node improve oldxu_index index data capacity?
Answer: No, because oldxu_ The index has only three partitions and has been distributed on three nodes, so the new fourth node is for oldxu_index is not available. Therefore, it can not improve the data capacity;
6.4 can adding replicas improve read performance
Question: at present, there are 3 ES nodes in total. Can you increase oldxu if you increase the number of replicas_ Read throughput of index;
Answer: No, because the new replicas will still be distributed on node1, node2 and node3, and the same resources are used, which means that when there are read requests, these requests will still be allocated to
Processing on node1, node2 and node3 means that the same hardware resources are used, so the read performance will not be improved
6.5 what should I do if I need to increase the read throughput performance
Answer: to increase the read throughput, you still need to add nodes. For example, when adding three nodes node4, node5 and node6, migrate the original R0, R1 and R2 to the three new nodes respectively. When a read request comes, node4, node5 and node6 will be allocated, which means that there are new CPU, memory and IO, so that the hardware resources of node1, node2 and node3 will not be occupied, Then the read throughput will be really improved at this time;
6.6 copy and slice summary
The number of slices and replica settings are very important and need to be planned in advance
- If it is too small, the subsequent horizontal expansion cannot be realized by adding nodes;
- Setting too many shards will result in too many shards distributed on a node, resulting in a waste of resources. Too many fragments will also affect the query performance;
- Once the number of slices is set, it cannot be modified
7, ES cluster failover and recovery
7.1 what is failover
The so-called failover refers to how the cluster automatically repairs when a node in the cluster fails. The ES cluster is currently composed of three nodes, as shown in the figure below. At this time, the cluster
The status is green
7.2 simulated node failure
Suppose: the service is terminated due to the downtime of the node1 machine. How will the cluster handle it; There are three steps:
1. Re election
2. Main slice adjustment
3. Copy slice adjustment
7.2.1 re-election
Node2 and node3 find that node1 cannot respond; A master election will be initiated after a period of time. For example, node2 is selected as the master node here; At this time, the cluster status changes to Red;
7.2.2 main slice adjustment
node2 found that the main partition P0 was not allocated, and promoted R0 on node3 to the main partition; At this time, all primary partitions are allocated normally, and the cluster status changes to Yellow status;
7.2.3 copy slice adjustment
node2 regenerates the primary partitions P0 and P1 into new replica partitions R0 and R1. At this time, the cluster state changes to Green;
8, ES document routing principle
Es document distributed storage. When a document is stored in the ES cluster, what is the storage principle?
As shown in the figure, how is Document1 stored in fragment P1 when we want to save documents in a cluster? What is the basis for selecting P1?
In fact, there is a document to fragment mapping algorithm, which aims to make all documents evenly distributed on all fragments. What algorithm is it? Random or polling? This is not desirable, because the data needs to be read after storage. In this case, how to read it?
In fact, in ES, the node to which the partition corresponding to the document is stored is calculated by the following formula:
shard = hash(routing) % number_of_primary_shards # hash algorithm ensures that the data is evenly distributed in the partition # routing is a key parameter. The default is the document id, which can also be customized. # number_of_primary_shards primary shards # Note: this algorithm is related to the number of primary partitions, but once it is determined, the primary partition cannot be changed. # Because once the master partition is modified, the calculation of Share is completely different
The document data in the index will be evenly distributed in different slices.
8.1 document creation process
8.2 document reading process
8.3 document batch creation process
8.4 document batch reading process
9, ES extended cluster node, extended data node and routing node method
9.1 environmental preparation
Host name | Internet IP WAN | Intranet IP LAN |
---|---|---|
es-node4 | 10.0.0.164 | 172.16.1.164 |
es-node5 | 10.0.0.165 | 172.16.1.165 |
Expansion node 4 is configured as a storage node
[root@es-node4 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml cluster.name: my-application node.name: es-node-4 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 172.16.1.164 http.port: 9200 discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"] # From these hosts, it is found that the cluster joins the organization, and multiple can be written node.data: true # Storage node node.master: false # Do not participate in master election
The extension node 5 is configured to only act as a routing node Coordinating
[root@es-node5 ~]# grep "^[a-zA-Z]" /etc/elasticsearch/elasticsearch.yml cluster.name: my-application node.name: es-node5 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 172.16.1.165 http.port: 9200 discovery.seed_hosts: ["172.16.1.161", "172.16.1.162","172.16.1.163"] node.data: false # Do not store node.master: false # Not running
9.2 extension node inspection
Check the status after cluster expansion through cerebor; If the cluster cannot be joined or is rejected, try to delete the file under / var/lib/elasticsearch, and then restart es;
If the existing data node is modified to the Coordinating node; Data cleaning is required, otherwise it cannot be started;
[root@es-node5 ~]# /usr/share/elasticsearch/bin/elasticsearch-node repurpose
10, ES cluster tuning
10.1 kernel parameter optimization
fs.file-max=655360 # Set the maximum number of open file descriptors in the system. It is recommended to modify it to 655360 or higher, vm.max_map_count = 262144 # It is used to limit the size of virtual memory that a process can have. It is recommended to modify it to 262144 or higher. net.core.somaxconn = 32768 # Full connection pool size net.ipv4.tcp_tw_reuse = 1 net.ipv4.ip_local_port_range = 1000 65535 net.ipv4.tcp_max_tw_buckets = 400000
[root@node1 ~]# sysctl -p
# Adjust the maximum number of user processes (nproc) and the maximum open file descriptor (nofile) [root@node ~]# rm -f /etc/security/limits.d/20-nproc.conf # Delete default nproc settings file [root@node ~]# vim /etc/security/limits.conf * soft nproc 20480 * hard nproc 20480 * soft nofile 65536 * hard nofile 65536
10.2 es configuration parameter optimization
# 1. Lock the physical memory address and avoid es using swap to exchange partitions. Frequent exchange will lead to high IOPS. [root@es-node ~]# vim /etc/elasticsearch/elasticsearch.yml bootstrap.memory_lock: true #2. Configure elasticsearch startup parameters [root@es-node ~]# sed -i '/\[Service\]/a LimitMEMLOCK=infinity' /usr/lib/systemd/system/elasticsearch.service [root@es-node ~]# systemctl daemon-reload [root@es-node ~]# systemctl restart elasticsearch
10.3 JVM optimization
jvm memory should be estimated according to the amount of data to be stored in node. In order to ensure performance, there is a recommended ratio between memory and data: like general log files, 1G memory can store 48G~96GB data; The maximum jvm heap memory should not exceed 31GB;
The second is the storage capacity of the main partition, which is controlled at 30-50GB;
It is assumed that the total amount of data is 1TB, 3 node nodes and 1 replica; Then the actual storage size is 2TB, because there is a replica; 2TB / 3 = 700GB, and then each node needs to reserve 20% empty space
It means that each node should store about 700/0.8=875GB of data;
Calculated according to the ratio of memory to stored data:
875GB/48=18GB, less than 31GB. Because 31*48=1.4TB and each Node can store 1.4TB data, three nodes are enough;
875GB/30=29/2=15 main segments. Because we should try to control the size of the main partition to 30GB
Assuming that the total amount of data is 2TB, three node nodes and one replica, 4tb / 3 = 1365gb needs to be stored, and then 20% of the space needs to be reserved for each node, which means that each node needs to store about
1365GB/0.8=1706GB data.
Calculated according to the ratio of memory to stored data:
1706GB/48G=35 is greater than 31GB. Because 31*48=1.4TB and each node can store 1.4TB data, at least 4 nodes are required
1706 / 30 = 56 / 2 = 28 main segments
[root@es-node ~-]# vim /etc/elasticsearch/jvm.options -Xms 31g #Minimum heap memory -Xmx31g #Maximum heap memory #Modify to an appropriate value according to the memory size of the server. It is generally set to half of the physical memory of the server, but the maximum can not exceed 32G #Server configuration with data volume of about 1TB per day 16C 64G 6T 3 platform ECS