producer of kafka manometry

background Recently, we built a large data platform by ourselves. Due to time problems and tight schedule, we could not measure the platform.Now that the platform is built, it is planned to test the platform components one by one. Welcome to correct, don't spray! Pressure Target Tests the ability of the Kafka cluster to write and consume messag ...

Posted on Sat, 04 Apr 2020 11:37:12 -0400 by yogicasey

Kafka partition strategy and customization

Default partition policy The default partition strategy is: positive (bytearray generates 32-bit hash value)% numpartitions The result of this formula is that the number of positive integers between 0 - (numarticles-1) is approximately equal, that is to say, the default partition strategy of kafka is that no matter how many partitions we give, ...

Posted on Wed, 01 Apr 2020 05:17:14 -0400 by hightechredneck

About kafka monitoring tool

Summary Apache Kafka is a fast, scalable, high-throughput, fault-tolerant distributed "publish subscribe" message system, written in Scala and Java language, which can deliver messages from one endpoint to another. Compared with traditional message middleware (such as ActiveMQ, RabbitMQ), Kafka has the characteristics of high throug ...

Posted on Fri, 20 Mar 2020 04:04:07 -0400 by jeev

"partition.assignment.strategy" exception handling caused by spark streaming connection kafka

Server running environment: spark 2.4.4 + scall 2.11.12 + kafka 2.2.2 Because the business is relatively simple, kafka only has fixed topics, so it always uses the following script to perform real-time flow calculation spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.4.4 --py-files /data/service/xxx.zip /data/service/x ...

Posted on Tue, 17 Mar 2020 23:48:00 -0400 by echo64

Talk about skywalking's kafka-plugin

order This paper mainly studies kafka-plugin of skywalking skywalking-plugin.def skywalking-6.6.0/apm-sniffer/apm-sdk-plugin/kafka-plugin/src/main/resources/skywalking-plugin.def kafka-0.11.x/1.x/2.x=org.apache.skywalking.apm.plugin.kafka.define.CallbackInstrumentation kafka-0.11.x/1.x/2.x=org.apache.skywalking.apm.plugin.kafka.define.KafkaCons ...

Posted on Tue, 17 Mar 2020 14:05:13 -0400 by bapan

Kafka offset management

1. definition Each partition in Kafka consists of a series of orderly and immutable messages, which are continuously appended to the partition. Each message in the partition has a sequential sequence number, which is used by the partition to uniquely identify a message. Offset records the sequence ...

Posted on Thu, 12 Mar 2020 08:17:51 -0400 by codrgii

ELK+Kafka+Filebeat analyze Nginx logs

1, Basic structure host name IP application role nginx 10.1.1.3 Nginx nginx 10.1.1.3 Filebeat kafka1 10.1.1.4 zookeeper kafka1 10.1.1.4 Kafka1 kafka2 10.1.1.5 Kafka2 logstash 10.1.1.6 Logstash es1 10.1.1.7 ES1 data/master es2 10.1.1.8 ES2+Kibana data/master es3 10.1.1.9 ES3 d ...

Posted on Sun, 08 Mar 2020 05:19:25 -0400 by imagineek

Log Analysis series (external 3): platform security

This series of stories is a pure fiction, if there are similarities, it is a coincidence After realizing the unified log analysis platform, the first problem for small B is the security of the platform. If the platform used for security analysis is successfully attacked, then small B will be swept out o ...

Posted on Wed, 04 Mar 2020 07:11:17 -0500 by likethegoddess

The integration of Flink SQL client 1.10 and hive to read real-time data

Looking forward to processing streaming data in the form of pure SQL, flink 1.10 introduced Hive integration, which is available in production, and has stronger streaming SQL processing capacity. Let's try it this time~~   ​   [outline] 1. Environmental preparation 2. SQL Client and hive integrated configuration 3. Reading kafka data with ...

Posted on Wed, 04 Mar 2020 01:51:22 -0500 by xtheonex

[actual combat] installation, deployment and simple use of zookeeper+kafka cluster

Catalog Asynchronous update: Message Queuing Message queuing: kafka zookeeper cluster construction scala build kafka cluster construction Asynchronous update: Message Queuing The multi-level cache architecture is mainly used to solve the data with low timeliness, such as the basic information ...

Posted on Sun, 01 Mar 2020 04:45:13 -0500 by tariel