Detailed explanation of Kafka core concepts_ Step on the blog - CSDN blog

preface

This article inherits the core concepts of topic and kafka, including cluster fragmentation, persistent message storage mechanism and leader election mechanism; This article continues to analyze topic, message storage mechanism, and production produce rs and consumers   Detailed explanation of the principle of Consumer, and transaction mode

Message storage mechanism

message number

Consumer offset,   Serial number offset. Every consumer has a serial number and offset when consuming. There is no need to confirm the message, because there is a serial number to judge where the consumer has visited.

TimeIndex time index

• The message should have a timestamp
• Time indexing .timeindex data structure { Timestamp, sequence number }

This is a lot of places, including whether we are developing projects or not. You need to save the time, because data analysis or anything will appear many times. Data analysis and viewing need to be queried according to the time dimension. This is later. Continue time indexing.

When the message starting at a certain time is to be obtained, it is obtained from the time index according to the time stamp >= The sequence number of the first message at that time; Then pull data from the serial number.

Does this feel like an index in a database.

Broker About Timestamp Global default configuration parameters for:

  You can configure the creation time or log writing time. The default is the message creation time

Topic About Timestamp Configuration parameters:

 

How to find the offset of segmentation in consumers is through the following methods.

  The returned partition and the start number of the partition   Finally, you can get the services you can consume according to this

 

  Configuration parameters of Topic

Official document parameter configuration:   Apache Kafka

Parameter modification

> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic 
-- partitions 1 --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1

  You can also change or set the override value later by using the alter configs command. This example resets my topic

> bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics 
--entity- name my-topic --alter --add-config max.message.bytes=128000

bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity- name my-topic --describe

> bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics 
-- entity-name my-topic --alter --delete-config max.message.bytes

Delete Topic

[root@node4 latest]# bin/kafka-topics.sh --create --bootstrap-server
 localhost:9092 --replication-factor 1 --partitions 1 --topic test-1

# Send a message [root@node4 latest]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-1

Acknowledgement result of Callback asynchronous processing

Obtain the confirmation result asynchronously.

  There is no message data we need in RecordMetadata. Properties can be seen

You need to make a record correlation to know that the data is that message.

You want to know the exceptions, including the timeout exception, the server exception not found, and so on.

  Using in spring

Send messages directly using Kafkatemple, also through

/*** Set a {@link ProducerListener} which will be invoked when Kafka acknowledges 
* a send operation. By default a {@link LoggingProducerListener} is configured
 * which logs errors only. 
* @param producerListener the listener; may be {@code null}. 
*/ 
public void setProducerListener(@Nullable ProducerListener<K, V> producerListener) { 
    this.producerListener = producerListener;
 }

You can give us the corresponding records

Producer interceptor interceptor

The order of interception has no priority, only the order of addition

Mode of use

Specify the implementation class name through interceptor.classes:

//Configure ProducerInterceptor 
props.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, "com.study.xxx.MyProducerInterceptor1,com.study.xxx.MyProducerInterceptor2");

Idempotent mode

That is, how to send messages only once to prevent repeated messages

Transaction mode

Configure the isolation level, including read commit, etc.

Consumer

Consumers, in Kafka, consumers use the poll pull mode to obtain messages, and the data will always exist if it is not deleted.

public static void main(String[] args) {
		// For consumer parameters, see: http://kafka.apachecn.org/documentation.html#newconsumerconfigs
		Properties props = new Properties();
		props.put("bootstrap.servers", "localhost:9092");
		props.put(CommonClientConfigs.CLIENT_ID_CONFIG, "client-1");
		// Set consumption group
		props.put("group.id", "test");
		// Enable automatic consumption offset submission
		// If this value is set to true, the consumer will periodically save the offset value of the current consumption to zookeeper. When the consumer fails to restart, this value will be used as the value for new consumption.
		props.put("enable.auto.commit", "true");
		// The interval between automatic consumption and offset submission
		props.put("auto.commit.interval.ms", "1000");
		// Deserializer for key
		props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
		// Deserializer for message
		props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

		String topic = "test";

		try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);) {
			// Subscribe to topics
			consumer.subscribe(Arrays.asList(topic));

			while (true) {
				// Kafka is a pull mode. The time parameter of poll tells Kafka: if there is no data at present, how long to wait before responding
				ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1L));
				for (ConsumerRecord<String, String> record : records)
					System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());

				// Set where to start pulling messages (if necessary)
				// consumer.seek(new TopicPartition(topic, 0), 0);
			}
		}
	}

Consumer is non thread safe. In the case of multithreading, one thread has one consumer.

Consumer group

The concept of consumption group, why this concept, to achieve the concept of consumption load balance.

• Consumers are identified by a consumer group name. Each record published to topic is assigned to a consumer instance in the subscription consumer group. Consumer instances can be distributed in multiple processes or on multiple machines.
• If all consumer instances are in the same consumer group, the message record will be load balanced to each consumer instance
• If all consumer instances are in different consumer groups, each message record will be broadcast to all consumer processes

• Generally, each topic will have some consumption groups. One consumption group corresponds to a "logical subscriber". A consumer group consists of many consumer instances, which is easy to expand and fault-tolerant. This is the concept of publish and subscribe, except that subscribers are a group of consumers rather than a single process.
• The way to realize consumption in Kafka is to divide the fragments in the log into each consumer instance, so that each instance is the only consumer in the fragment at any time. Maintaining the consumption relationship in the consumption group is dynamically processed by Kafka protocol. If new instances join the group, they will take over some partition partitions from other members of the group; If an instance disappears, the owned partition will be distributed to the remaining instances.
• Kafka only guarantees that the records in the partition are orderly, but does not guarantee the order of different partitions in the subject. If you want all messages to be consumed in order, you can use a topic with only one partition, which means that there is only one consumer process for each consumer group.

group rebalance

Here, the group rebalancing and consumption group rebalancing delay parameters are set

Event triggering the counterbalance:

• Changes in the number of consumers
• Increase or decrease of slice

Will lead to the start of this balance, which is the reason for the balance.

How to specify the consumption group id in spring

Directly refers to the default id we set, that is, the consumer group id

  How does Kafka know that consumers are offline and need rebalance?

It also comes from heartbeat and session.  

Beatheart Session

Heartbeat and session    heartbeat.interval.ms ,session.timeout.ms  , max.poll.interval.ms

There are default values here. There's a heartbeat to monitor. The timeout time of the session will cause the update of the consumer's status

Consumption offset

// Enable automatic consumption offset submission
 // If this value is set to true, the consumer will periodically save the offset value of the current consumption to zookeeper. When the consumer fails to restart, this value will be used as the value for new consumption. props.put("enable.auto.commit", "true"); 
// The interval between automatic consumption and offset submission
 props.put("auto.commit.interval.ms", "1000");

• Repeated consumption
• Lost data (lost message)

This is the problem of automatic submission. Only judge and pull again.

Manual submission

Use code

//Set manual submission props.put("enable.auto.commit", "false");
 ...
 //Manually synchronize submit consumer offset to zookeeper 
  consumer.commitSync();

Here, you can catch exceptions when throwing exceptions.

Submission method of Consumer

Flexibly submit consumption offset value by slice  

 

• max.poll.records   The maximum number of records returned in a single call to poll().

Here you can see the parameter configuration on the official website.

This parameter is used to prevent repetition and always take the value of a partition.

Flow Control of consumption

• pause(Collection partitions) pauses the pull of a subset
• resume(Collection partitions) restores the pull of subsets

affair

Consumer's configuration parameters

Kafka Chinese document - ApacheCN

API in Spring

• @KafkaListener
• @TopicPartition
• @PartitionOffffset
• KafkaListenerErrorHandler