ELK + filebeat + kafka non cluster log management system

ELK + filebeat + kafka build log management system

Because the current project needs to manually go to the server to view the log when it encounters an error, which is time-consuming, the company decided to use the log management system and had the opportunity to learn

ELK introduction:

ELK = Elasticsearch, Logstash, Kibana    It is a set of solutions for real-time data collection, storage, indexing, retrieval, statistical analysis and visualization. The latest version has been renamed Elastic Stack,And added Beats project
Elasticsearch It is an open source distributed search engine, which provides three functions: collecting, analyzing and storing data. Its features include: distributed, zero configuration, automatic discovery, automatic index fragmentation, index copy mechanism, restful Style interface, multiple data sources, automatic search load, etc.
Logstash It is mainly used to collect, analyze and filter logs, and supports a large number of data acquisition methods. The general working mode is c/s framework, client The client is installed on the host that needs to collect logs, server The client is responsible for filtering and modifying the received node logs at one time elasticsearch up
Kibana It is also an open source and free tool, Kibana Can be Logstash and ElasticSearch The log analysis provided is friendly Web Interface, which can help summarize, analyze and search important data logs.

Introduction to filebeat:

Filebeat is a lightweight delivery tool for forwarding and centralizing log data. Filebeat monitors the log files or locations you specify, collects log events, and forwards them to Elasticsearch or Logstash for indexing.

kafka introduction:

Kafka was originally developed by Linkedin company. It is a distributed, partitioned, multi replica, multi subscriber distributed logging system based on zookeeper coordination (it can also be used as an MQ system). It can be commonly used for web/nginx logs, access logs, message services, etc

Article reference link: https://www.yangxinghui.com/1389.html (the basic configuration and optimization of the environment are all taken from this link and the relevant installed configurations)

Environmental preparation:   filebeat + kafka  nginx Log 4 core 4 G   elasticsearch + kibana  4 Nuclear 4 G   logstash  8 Nuclear 8 G

Software version:

Elasticsearch: elasticsearch-7.5.1-linux-x86_64.tar.gz

Kibana: kibana-7.5.1-linux-x86_64.tar.gz

Logstash: logstash-7.5.1.tar.gz

Filebeat: filebeat-7.5.1-linux-x86_64.tar.gz

JDK: jdk-11.0.1_linux-x64_bin.tar.gz

Zookeeper: zookeeper-3.4.10.tar.gz

Kafka: kafka_2.12-2.5.0.tgz

Install the JDK environment (I configured all three servers for convenience)

You can choose to download the local from the official website and upload it to the server or download it directly from the server

[root@localhost data]# wget https://mirrors.yangxingzhen.com/jdk/jdk-11.0.1_linux-x64_bin.tar.gz

[root@localhost data]# tar zxf jdk-11.0.1_linux-x64_bin.tar.gz -C /usr/local

#Configure / etc/profile and add the following

[root@localhost local]# vim /etc/profile

export JAVA_HOME=/usr/local/jdk-11.0.1
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH:$HOMR/bin

[root@localhost local]# source /etc/profile

[root@localhost local]# java -version

#See the following information. The java environment configuration is successful

java version "11.0.1" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.1+13-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode)

Create ELK user

[root@localhost ~]# useradd elk

Install in server zookeeper and kafka

Installing Zookeeper

1. Download Zookeeper package

[root@localhost data]# wget -c http://archive.apache.org/dist/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz

2. Unzip, install and configure Zookeeper

[root@localhost data]# tar zxf zookeeper-3.4.10.tar.gz

[root@localhost data]# mv zookeeper-3.4.10 /usr/local/zookeeper

[root@localhost data]# cd /usr/local/zookeeper/

3. Create a snapshot log storage directory:

[root@localhost zookeeper]# mkdir -p data

4. Create transaction log storage directory:

[root@localhost zookeeper]# mkdir -p logs

[[note]:If not configured dataLogDir,Then the transaction log will also be written in data In the directory. This will seriously affect zookeeper Performance. Because in zookeeper When the throughput is high, too many transaction logs and snapshot logs are generated.

[root@localhost zookeeper]# cd conf/

[root@localhost conf]# cp zoo_sample.cfg zoo.cfg

[root@localhost conf]# vim zoo.cfg

#Configuration content

#Single heartbeat detection interval between servers or between clients and servers, in milliseconds
#The maximum number of heartbeats (number of ticktimes) that can be tolerated during the initial connection between the follower server (F) and the leader server (L) in the cluster
#The maximum number of heartbeats that can be tolerated between the flower server (F) and the leader (L) server in the cluster
#The port on which the client connects to the zookeeper server. Zookeeper will listen to this port and accept the client's access requests
#Storing data files
#Store log files
#Zookeeper cluster, 2888 is the election port and 3888 is the heartbeat port
#Server number = server IP:LF data synchronization port: LF election port
[root@localhost conf]# echo "1" > /usr/local/zookeeper/data/myid

5. Start Zookeeper service

[root@localhost conf]# /usr/local/zookeeper/bin/zkServer.sh start

Install Kafka

1. Download Kafka package

[root@localhost data]# wget -c https://www.apache.org/dyn/closer.cgi?path=/kafka/2.5.0/kafka_2.12-2.5.0.tgz

2. Unzip Kafka software package

[root@localhost data]# tar xf kafka_2.12-2.5.0.tgz
[root@localhost data]# mv kafka_2.12-2.5.0 /usr/local/kafka

3. Configure Kafka

[root@localhost data]# vim /usr/local/kafka/config/server.properties
listeners=PLAINTEXT:// port # needs to be opened
zookeeper.connect=  #Port 2181 needs to be opened

4. Start Kafka service

Note: need to wait Zookeeper Server startup and restart Kafka,Otherwise, an error will be reported

[root@localhost data]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties

Install elasticsearch and kibana in

Install elasticsearch

1. Create a persistent directory and Logs directory

[root@localhost data]# mkdir -p /data/elasticsearch/{data,logs}

2. Download the elasticsearch package

[root@localhost data]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.1-linux-x86_64.tar.gz

3. Unzip and rename

[root@localhost data]# tar xf elasticsearch-7.5.1-linux-x86_64.tar.gz

[root@localhost data]# mv elasticsearch-7.5.1 /usr/local/elasticsearch

4. Modify the elasticsearch.yml configuration file as follows:

[root@localhost data]# vim /usr/local/elasticsearch/config/elasticsearch.yml
# Cluster name
cluster.name: es
# Node name
node.name: es-master
# Create a directory to store data
path.data: /data/elasticsearch/data
# Create a log directory first
path.logs: /data/elasticsearch/logs
# Node IP
# tcp port
transport.tcp.port: 9300
# http port
http.port: 9200 #Port 9200 needs to be opened
# List of primary qualified nodes. If there are multiple primary nodes, the primary node shall be configured accordingly
cluster.initial_master_nodes: [""]
# Allow as master node
node.master: true
# Save data
node.data: true
node.ingest: false
node.ml: false
cluster.remote.connect: false
# Cross domain
http.cors.enabled: true
http.cors.allow-origin: "*" 

5. ELK user authorization

[root@localhost data]# chown -R elk.elk /usr/local/elasticsearch/

[root@localhost data]# chown -R elk.elk /data/elasticsearch/*

6. Start elasticsearch service (test it for the first time, and then add -d background startup)

[root@localhost data]# su - elk

[elk@localhost data]$ /usr/local/elasticsearch/bin/elasticsearch

7. Start elasticsearch service in the background

[elk@localhost data]$ /usr/local/elasticsearch/bin/elasticsearch -d

8. Monitoring and detection

[elk@localhost ~]$ curl -X GET ''
   "cluster_name" : "es",
    "status" : "green",
    "timed_out" : false,
    "number_of_nodes" : 1,
    "number_of_data_nodes" : 1,
    "active_primary_shards" : 18,
    "active_shards" : 18,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 13,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 58.06451612903226
# status = green indicates that the service is normal

9. Configure SSL and enable X-Pack

9.1,X-pack What is it?

X-Pack yes Elastic Stack Extended functions to provide security, alarm, monitoring, reporting, machine learning and many other functions. ES7.0+After that, by default, when installed Elasticsearch When, it will be installed X-Pack,No separate installation is required.

X-Pack Four core steps of security configuration:

1)set up: xpack.security.enabled: true. 

2)generate TLS Certificate.

3)Configure encrypted communication.

4)Set the password.

9.2,Generate node certificate

[elk@master ~]$ cd /usr/local/elasticsearch/bin

[elk@master bin]$ ./elasticsearch-certutil ca -out /usr/local/elasticsearch/config/elastic-certificates.p12 -pass "123456"

9.3,Configure encrypted communication

[elk@master bin]$ vim ../config/elasticsearch.yml

# Configure X-Pack
http.cors.allow-headers: Authorization
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

9.4,The password is entered when creating the certificate, which can be set by the following method. (no password can be set during generation)

# Enter the password to generate the certificate

[elk@master bin]$ ./elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

[elk@master bin]$ ./elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

9.5,restart Elasticsearch

# Nodes configured to use TLS cannot communicate with nodes using unencrypted networks (and vice versa). After TLS is enabled, all nodes must be restarted to maintain communication between clusters.

# Obtain the PID number by querying the port

[elk@master bin]$ netstat -lntup |grep 9200

# Kill the PID number

[elk@master bin]$ kill -9 3337

[elk@master bin]$ /usr/local/elasticsearch/bin/elasticsearch -d

ElasticSearch configuration username and password

[elk@localhost ~]$ /usr/local/elasticsearch/bin/elasticsearch-setup-passwords interactive

Note: for the convenience of demonstration, the password is uniformly set to 123456

Note: after configuring the password, the command to obtain the cluster status is as follows

[elk@localhost ~]$ curl --user elastic:123456 -X GET ''

Elasticsearch common commands

curl -XDELETE ' - * 'delete index (followed by index name)

curl -XGET ' Cat / health? V & pretty 'view cluster status

curl -XGET ' Cat / indexes? V & pretty 'view indexes

Install Kibana

1. Download Kibana package

[root@localhost data]$ wget https://artifacts.elastic.co/downloads/kibana/kibana-7.5.1-linux-x86_64.tar.gz

2. Unzip the Kibana package and rename it

[root@localhost data]$ tar xf kibana-7.5.1-linux-x86_64.tar.gz

[root@localhost data]$ mv kibana-7.5.1-linux-x86_64 /usr/local/kibana

3. Configure Kibana profile

[root@localhost data]$ vim /usr/local/kibana/config/kibana.yml

# Configure kibana's port
server.port: 5601

# Configure listening ip
server.host: ""

# Configure the ip of the es server. If it is a cluster, configure the ip of the primary node in the cluster
elasticsearch.hosts: [""]
elasticsearch.username: "elastic"
elasticsearch.password: "123456"

# Configure kibana's log file path. Otherwise, the log is recorded in messages by default
logging.dest: /usr/local/kibana/logs/kibana.log

# Configure to Chinese 
i18n.locale: "zh-CN"

4. Create log directory and authorize

[root@localhost ~]# mkdir /usr/local/kibana/logs

[root@localhost ~]# chown -R elk.elk /usr/local/kibana/

5. Start Kibana service

[root@localhost ~]# su - elk

Foreground start

[elk@localhost ~]$ /usr/local/kibana/bin/kibana

Background start

[elk@localhost ~]$ /usr/local/kibana/bin/kibana &

Warm tip: you can start the front desk to view the log, and then start it in the background after it is normal.

Installing filebeat in server

Install filebeat

1. Download the filebeat package

[root@localhost data]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.1-linux-x86_64.tar.gz

2. Unzip and rename

[root@localhost data]# tar xf filebeat-7.5.1-linux-x86_64.tar.gz

[root@localhost data]# mv filebeat-7.5.1-linux-x86_64 /usr/local/filebeat

3. Edit the filebeat.yml configuration file. The configuration contents are as follows:

[root@localhost data]# vim /usr/local/filebeat/filebeat.yml
#========= Filebeat inputs ==========
- type: log
 enabled: true
   - /data/wwwlogs/nginx/nginx-access.log
   log_source: nginx-access
- type: log
 enabled: true
   - /data/wwwlogs/nginx/nginx-error.log
   log_source: nginx-error
  enabled: true
  hosts: [""]
  topic: "nginx-log"

4. Create the Filebeat log directory

[root@localhost ~]# mkdir /usr/local/filebeat/logs

[root@localhost ~]# chown -R elk.elk /usr/local/filebeat

5. Start the filebeat service

[root@localhost ~]# su - elk

[elk@localhost ~]# cd /usr/local/filebeat

Foreground start

[elk@localhost filebeat]$ ./filebeat -e -c filebeat.yml >>logs/filebeat.log

Background start

[elk@localhost filebeat]$ nohup ./filebeat -e -c filebeat.yml >>logs/filebeat.log >/dev/null 2>&1 &

Install logstash in server

Install logstash

1. Download software package

[root@localhost data]# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.5.1.tar.gz

2. Unzip and rename

[root@localhost data]# tar zxf logstash-7.5.1.tar.gz

[root@localhost data]# mv logstash-7.5.1 /usr/local/logstash

3. Create the log.conf file and add the following

[root@localhost ~]# vim /usr/local/logstash/config/log.conf
input {
  kafka {
     bootstrap_servers => ""
     topics => "nginx-log"
     codec => json
filter {
  if [fields][log_source]=="nginx-access"{
    grok {
      match => {
           "message" => '%{IP:clientip}\s*%{DATA}\s*%{DATA}\s*\[%{HTTPDATE:requesttime}\]\s*"%{WORD:requesttype}.*?"\s*%{NUMBER:status:int}\s*%{NUMBER:bytes_read:int}\s*"%{DATA:requesturl}"\s*%{QS:ua}'
      overwrite => ["message"]
   if [fields][log_source]=="nginx-error"{
     grok {
       match => {
         "message" => '(?<time>.*?)\s*\[%{LOGLEVEL:loglevel}\]\s*%{DATA}:\s*%{DATA:errorinfo},\s*%{WORD}:\s*%{IP:clientip},\s*%{WORD}:%{DATA:server},\s*%{WORD}:\s*%{QS:request},\s*%{WORD}:\s*%{QS:upstream},\s*%{WORD}:\s*"%{IP:hostip}",\s*%{WORD}:\s*%{QS:referrer}'
       overwrite => ["message"]

output {
  if [fields][log_source] == "nginx-access" {
      elasticsearch {
          hosts => [""]
          user => "elastic"
          password => "123456"
          action => "index"
          index => "nginx_access.log-%{+YYYY.MM.dd}"
  if [fields][log_source] == "nginx-error" {
      elasticsearch {
          hosts => [""]
          user => "elastic"
          password => "123456"
          action => "index"
          index => "nginx_error.log-%{+YYYY.MM.dd}"

4. Start logstash service

[root@localhost ~]# chown -R elk.elk /usr/local/logstash

[root@localhost ~]# su - elk

Foreground start

[elk@localhost ~]$ /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/elasticsearch.conf

Background start

[elk@localhost ~]$ cd /usr/local/logstash/bin && nohup ./logstash -f /usr/local/logstash/config/elasticsearch.conf >/dev/null 2>&1 &

Visit Kibana

#Browser access: , the following interface appears

Enter the user name: elastic and password: 123456 set previously. After logging in, the following interface appears

Select browse by yourself and the following interface will appear

Click the "Settings" icon = > index management, and you can see the index information of nginx

1. Create Nginx access log index

Index mode = > create index mode, enter the index mode name, and click next

2. Create nginx error log index

Index mode = > create index mode, enter the index mode name, and click next

Click Discover to see the log data, as shown below
So far, the collection of Nginx logs by ELK log analysis platform has been completed.

Special note:

This article mainly refers to: https://www.yangxinghui.com/1400.html (build ELK-7.5.1 distributed cluster and configure X-Pack for Linux),https://www.yangxinghui.com/1389.html (build ELK+Filebeat+Kafka distributed log management platform based on Linux) The main configurations of the two articles are basically the same, and then there are some errors in the configuration. After self query and solution, there will be no problems according to this configuration. This article is written after the problems encountered have been solved

Tags: ELK

Posted on Fri, 15 Oct 2021 05:08:08 -0400 by Casalen