About kafka monitoring tool

Summary

Apache Kafka is a fast, scalable, high-throughput, fault-tolerant distributed "publish subscribe" message system, written in Scala and Java language, which can deliver messages from one endpoint to another.

Compared with traditional message middleware (such as ActiveMQ, RabbitMQ), Kafka has the characteristics of high throughput, built-in partition, supporting message copy and high error tolerance, which is very suitable for large-scale message processing applications.

kafka official website

Kafka is commonly used for two broad categories of applications:

  • Establish a real-time flow data pipeline to reliably acquire data between systems or applications.
  • Build real-time streaming applications to transform or respond to data flows.

kafka monitoring

After Kafka is built and put into use, it is necessary to use some management tools for the convenience of operation and maintenance. At present, Kafka monitoring solutions seem to be many, but there is not a "large and comprehensive" general solution, and each framework has its own advantages.

Common monitoring tools

  • Kafka Manager
  • Kafka Offset Monitor
  • Kafka Eagle
  • JmxTool
  • ... ...

Tool comparison

Installation environment: Centos 7.6

Tool name Characteristic Remarks
Kafka Manager Implement common JMX monitoring at the broker level; monitor the consumer consumption progress; and manage multiple clusters directly on the page. Compilation and installation are time-consuming; access control is not available; alarms cannot be configured; memory is consumed.
Kafka Eagle It can realize the common JMX monitoring at the broker level, monitor the consumer consumption progress, and manage multiple clusters directly on the page. The installation method is simple, binary package decompression is available; alarms can be configured (pin, wechat, email). A database (mysql or sqlite) is required.
Kafka Offset Monitor Didn't do any further research... Support for the new versions of consumer and security is poor; < br / > the project has not been maintained for nearly 2 years.
JmxTool Use with Influxdb and Grafana More complicated

###centos7 deployment Kafka Eagle

No scheme is the most appropriate, only which scheme is more suitable for their own use scenarios. Here I choose Kafka eagle, because Kafka manager eats too much memory...

Binary packages are recommended for installation Kafka Eagle installation document .

To configure

######################################
# multi zookeeper & kafka cluster list
######################################
# Configure multiple zookeeper & Kafka clusters
kafka.eagle.zk.cluster.alias=cluster-test,cluster-prod
cluster-test.zk.list=tvm11:2181,tvm12:2181,tvm13:2181
cluster-prod.zk.list=vm11:2181,vm12:2181,vm13:2181

######################################
# broker size online list
######################################
# Number of cluster broker s
cluster-test.kafka.eagle.broker.size=20
cluster-prod.kafka.eagle.broker.size=20

######################################
# zk client thread limit
######################################
# Number of ZK connections (get kafka related information from zk)
kafka.zk.limit.size=25

######################################
# kafka eagle web port, default 8048 web port
######################################
# kafka eagle web port, default 8048
kafka.eagle.webui.port=8048

######################################
# kafka offset storage
######################################
# kafka offset, if the offset is stored in zk, this configuration is not required
cluster-test.kafka.eagle.offset.storage=kafka
cluster-prod.kafka.eagle.offset.storage=kafka

######################################
# kafka metrics, 30 days by default
######################################
# Monitor the details of kafka. When kafka is started, it is required to open jmx port, that is, jmx port ='9999 '$kafka house / bin / kafka server start.sh $kafka house / config / server.properties
kafka.eagle.metrics.charts=true
# The monitoring data is reserved for 30 days by default and adjusted as required
kafka.eagle.metrics.retain=30


######################################
# kafka sql topic records max
######################################
# If offset is out of range occurs, enable this property -- Only suitable for kafka sql
kafka.eagle.sql.topic.records.max=5000
kafka.eagle.sql.fix.error=false

######################################
# delete kafka topic token
######################################
# Delete kafka topic token -- Set to delete the topic token, so that administrators can have the right to delete
kafka.eagle.topic.token=keadmin

######################################
# kafka sasl authenticate
######################################
# kafka sasl certification related
cluster-test.kafka.eagle.sasl.enable=false
cluster-test.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster-test.kafka.eagle.sasl.mechanism=SCRAM-SHA-256
cluster-test.kafka.eagle.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka" password="kafka-eagle";
cluster-test.kafka.eagle.sasl.client.id=
cluster-test.kafka.eagle.sasl.cgroup.enable=false
cluster-test.kafka.eagle.sasl.cgroup.topics=

cluster-prod.kafka.eagle.sasl.enable=false
cluster-prod.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster-prod.kafka.eagle.sasl.mechanism=PLAIN
cluster-prod.kafka.eagle.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka" password="kafka-eagle";
cluster-prod.kafka.eagle.sasl.client.id=
cluster-prod.kafka.eagle.sasl.cgroup.enable=false
cluster-prod.kafka.eagle.sasl.cgroup.topics=

# Storage engine for storing Metrics information
######################################
# kafka sqlite jdbc driver address
######################################
#kafka.eagle.driver=org.sqlite.JDBC
#kafka.eagle.url=jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db
#kafka.eagle.username=root
#kafka.eagle.password=www.kafka-eagle.org

######################################
# kafka mysql jdbc driver address
######################################
kafka.eagle.driver=com.mysql.jdbc.Driver
kafka.eagle.url=jdbc:mysql://ip:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
kafka.eagle.username=username
kafka.eagle.password=password

**Note: * * the premise of using jmx monitoring is that kafka itself enables to view jmx ﹣ port parameters.

web interface management

  • web interface address: http://ip:8048/ke

  • Default administrator account / password: admin/123456 (displayed at startup)

  • Dashboard:

  • Topic management / consumer management

  • Alarm management / user authority management

The overall use is relatively simple, no need to elaborate.

Recommended Learning

Please pay attention to the official account of the landlord.

Tags: Big Data kafka JDBC MySQL SQLite

Posted on Fri, 20 Mar 2020 04:04:07 -0400 by jeev