Brief Filebeat configuration

Filebeat

brief introduction

Filebeat is a log file shipment tool that monitors the log directory or specified log files, tracks them for reading, and forwards the information to either elasticsearch or logstarsh when the client is installed on your server.

Architecture System

Beats is a lightweight log collector that takes up almost negligible CPU and memory on the system, with six members in the Beats family.

  • Packetbeat: Network data (collects network traffic data)
  • Metricbeat: Indicator (collecting system, process, and file system-level CPU and memory usage data, etc.)
  • Filebeat: Log file (collects file data)
  • Winlogbeat: Winows Event Log (collects Windows Event Log data)
  • Auditbeat: Audit data (collect audit logs)
  • Heartbeat: Runtime monitoring (collecting data about the system as it runs)

Working principle

  • When you open the filebeat program, it will start one or more probes (prospectors) to detect the log directory or file you specified
  • For each log file found by the detector, filebeat starts the harvester process
  • Each harvesting process reads the new contents of a log file and sends the new log data to the spooler.
  • The handler aggregates these events, and finally filebeat sends the aggregated data to the location you specified

Software Installation

Edition:

Software Version: Operating System (64bit) Filebeat 6.4.3

1. Online Installation

Redhat Series

Import Elastic Key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Create YUM source at/etc/yum.repos.d/create new filebeat.repo

[elastic-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Perform Installation

sudo yum install filebeat

Start and Stop Services

sudo systemctl enable filebeat.service
sudo systemctl start filebeat.service
sudo systemctl stop filebeat.service

Debian Series

Import Elastic Key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Create APT source, create filebeat-6.x.list at/etc/apt/sources.list.d

sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/filebeat-6.x.list

Perform Installation

sudo apt-get update && sudo apt-get install filebeat

Start and Stop Services

sudo systemctl enable filebeat.service
sudo systemctl start filebeat.service
sudo systemctl stop filebeat.service

2. Offline Installation

Redhat Series

Download installation package

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.3-x86_64.rpm
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.3-x86_64.rpm.sha512

Verify File

shasum -a 512 filebeat-6.4.3-x86_64.rpm -c filebeat-6.4.3-x86_64.rpm.sha512

Install rpm package

sudo rpm --install filebeat-6.4.3-x86_64.rpm

Debian Series

Download installation package

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.3-amd64.deb
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.3-amd64.deb.sha512

Verify File

shasum -a 512 filebeat-6.4.3-amd64.deb

Install deb package

sudo dpkg -i filebeat-6.4.3-amd64.deb

3. Binary Installation

Download installation package

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.3-linux-x86_64.tar.gz

decompression

tar -xvf filebeat-6.4.3.tar.gz

Start Services

sudo ./filebeat -e -c filebeat.yml

Basic Configuration

Main Profile/etc/filebeat/filebeat.yml

1. Data collection

Configure Collection Log Path

filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log

2. Data output

Configure Output Mode

Elasticsearch

output.elasticsearch:
hosts: ["myES:port"]

Logstash

output.logstash:
hosts: ["myLostash:port"]

Kafka

output.kafka:
enabled: true
hosts: ["myKafka:port"]
topic: "mytopic"

Redis

output.redis:
hosts: ["myRedis"]
port: 6379
key: "key"
password: "pwd" 
db: 2
timeout: 5
Two original articles have been published. Approved 0. Visits 42
Private letter follow

Tags: sudo RPM ElasticSearch yum

Posted on Mon, 24 Feb 2020 21:20:51 -0500 by vonnero