18_ View theme - modify theme - delete theme

1 view topics

The kafka-topics.bat script has five instruction types: create, list, describe, alter, and delete. The list and describe instructions can be used to easily view the topic information. We have touched on the usage of the describe instruction in the previous content, which will be described in more detail in this section.

You can view all the currently available topics through the list command, for example:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/kafka –list
__consumer_offsets
topic-create
topic-demo
topic-config

In the previous chapters, we use the describe command to view the information of a single topic. If we do not use -- topic to specify a topic, the details of all topics will be displayed Topic also supports specifying multiple topics, for example:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/kafka --describe --topic topic-create,topic-demo
Topic:topic-create	PartitionCount:4	ReplicationFactor:2	Configs:
	Topic: topic-create	Partition: 0	Leader: 2	Replicas: 2,0	Isr: 2,0
	Topic: topic-create	Partition: 1	Leader: 0	Replicas: 0,1	Isr: 0,1
	Topic: topic-create	Partition: 2	Leader: 2	Replicas: 1,2	Isr: 2,1
	Topic: topic-create	Partition: 3	Leader: 2	Replicas: 2,1	Isr: 2,1
Topic:topic-demo	PartitionCount:4	ReplicationFactor:3	Configs:
	Topic: topic-demo	Partition: 0	Leader: 2	Replicas: 2,1,0	Isr: 2,0,1
	Topic: topic-demo	Partition: 1	Leader: 2	Replicas: 0,2,1	Isr: 2,0,1
	Topic: topic-demo	Partition: 2	Leader: 2	Replicas: 1,0,2	Isr: 2,0,1
	Topic: topic-demo	Partition: 3	Leader: 2	Replicas: 2,0,1	Isr: 2,0,1

When using the describe command to view topic information, you can also specify three additional parameters: topics with overrides, under replicated partitions and unavailable partitions to add some additional functions.

Add the topics with overrides parameter to find out all topics that contain override configurations. It will only list topics that contain configurations different from those of the cluster. Note that when you use the topics with overrides parameter, only the first line of information that originally only uses the describe instruction is displayed. The reference example is as follows:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/kafka --describe --topics-with-overrides
Topic:__consumer_offsets	PartitionCount:50	ReplicationFactor:1	Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic:topic-config	PartitionCount:1	ReplicationFactor:1	Configs:cleanup.policy=compact,max.message.bytes=10000

Both the under replicated partitions and unavailable partitions parameters can find the problem partition. The under replicated partitions parameter allows you to find all partitions that contain retired replicas. The partition containing the failed replica may be in the process of synchronization, or there may be synchronization exceptions. At this time, the ISR set of the partition is smaller than the AR set. Focus on monitoring the partitions queried through this parameter, because it may mean that a broker in the cluster has failed or the synchronization efficiency has been reduced.

For example, referring to the environment of topic Create, we offline node2 nodes in the cluster, and then use this parameter to view the information of topic Create, as shown below:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/kafka --describe --topic topic-create --under-replicated-partitions
	Topic: topic-create	Partition: 1	Leader: 0	Replicas: 0,1	Isr: 0
	Topic: topic-create	Partition: 2	Leader: 2	Replicas: 1,2	Isr: 2
	Topic: topic-create	Partition: 3	Leader: 2	Replicas: 2,1	Isr: 2

When you restore node2 node and execute the same command, you can see that no information is displayed:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/kafka --describe --topic topic-create --under-replicated-partitions

Through the unavailable partitions parameter, you can view the partitions without leader copies in the topic. These partitions are offline and unavailable to external producers and consumers.

For example, referring to the topic create environment, we offline node2 and node3 nodes in the cluster, and then use this parameter to view the topic create information, as shown below:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --describe --topic topic-create --unavailable-partitions
	Topic: topic-create	Partition: 2	Leader: -1	Replicas: 1,2	Isr: 1
	Topic: topic-create	Partition: 3	Leader: -1	Replicas: 2,1	Isr: 1

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --describe --topic topic-create 
Topic:topic-create	PartitionCount:4	ReplicationFactor:2	Configs:
	Topic: topic-create	Partition: 0	Leader: 0	Replicas: 2,0	Isr: 0
	Topic: topic-create	Partition: 1	Leader: 0	Replicas: 0,1	Isr: 0
	Topic: topic-create	Partition: 2	Leader: -1	Replicas: 1,2	Isr: 1
	Topic: topic-create	Partition: 3	Leader: -1	Replicas: 2,1	Isr: 1

When we restore node2 and node3 and execute the same command, we can see that there is no information:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --describe --topic topic-create --unavailable-partitions

2. Modify the theme

After a topic is created, we are still allowed to make certain modifications to it, such as modifying the number of partitions, modifying the configuration, etc. this modification function is provided by the alter instruction in the kafka-topics.bat script.

Let's first look at how to increase the number of topic partitions. Take topic config as an example. The current number of partitions is 1 and modified to 3. The example is as follows:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --alter --topic topic-config --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/kafka --describe --topic topic-config
Topic:topic-config	PartitionCount:3	ReplicationFactor:1	Configs:
    Topic: topic-config	Partition: 0	Leader: 2	Replicas: 2	Isr: 2
    Topic: topic-config	Partition: 1	Leader: 0	Replicas: 0	Isr: 0
    Topic: topic-config	Partition: 2	Leader: 1	Replicas: 1	Isr: 1

Note the warning message above: when the message in the topic contains a key (that is, the key is not null), the behavior of calculating partitions according to the key will be affected. When the number of partitions in topic config is 1, messages will be sent to this partition regardless of the key value of the message; When the number of partitions increases to 3, the partition number will be calculated according to the key of the message. The message originally sent to partition 0 may now be sent to partition 1 or partition 2. This will also affect the order of established messages, so you must think twice when increasing the number of partitions. For topics based on key calculation, it is recommended to set the number of partitions at the beginning to avoid adjusting them later.

At present, Kafka only supports increasing the number of partitions and does not support reducing the number of partitions. For example, if we change the number of partitions in topic config to 1, an exception of InvalidPartitionException will be reported. An example is as follows:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --alter --topic topic-config --partitions 1
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Error while executing topic command : The number of partitions for a topic can only be increased. Topic topic-config currently has 3 partitions, 1 would not be an increase.
[2018-09-10 19:28:40,031] ERROR org.apache.kafka.common.errors.InvalidPartitionsException: The number of partitions for a topic can only be increased. Topic topic-config currently has 3 partitions, 1 would not be an increase.
 (kafka.admin.TopicCommand$)
Why not support partition reduction? according to Kafka With the existing code logic, this function can be realized, but it will also increase the complexity of the code sharply. Many factors need to be considered to realize this function, such as how to deal with the messages in the deleted partition? If the message disappears with the partition, the reliability of the message cannot be guaranteed; If it needs to be retained, it needs to consider how to retain it. Directly stored at the end of the existing partition, the timestamp of the message will not be incremented Spark,Flink Such components that require message timestamp (event time) will be affected; If existing partitions are inserted dispersedly, internal data replication will occupy a lot of resources when there is a large amount of messages, and how can the availability of this topic be guaranteed during replication? At the same time, sequential problems, transactional problems, and state machine switching between partitions and replicas have to be faced. In contrast, the benefit point of this function is very low. If you really need to implement this function, you can recreate a topic with a small number of partitions, and then copy the messages in the existing topic according to the established logic.

When creating a topic, there is an if not exists parameter to ignore some exceptions, and there are corresponding parameters here. If the topic to be modified does not exist, you can ignore exceptions through the if exists parameter. Next, modify the partition of a topic unknown that does not exist, and the error message "topic unknown does not exist" will be reported. The example is as follows:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --alter --topic topic-unknown --partitions 4
Error while executing topic command : Topic topic-unknown does not exist on ZK path localhost:2181/kafka
[2018-09-11 11:14:55,458] ERROR java.lang.IllegalArgumentException: Topic topic-unknown does not exist on ZK path localhost:2181/kafka
	at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:123)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:65)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
 (kafka.admin.TopicCommand$)

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --alter --topic topic-unknown --partitions 4 --if-exists

In addition to modifying the number of partitions, we can also use the alter instruction of kafka-topics.sh script to change the configuration of topics. When creating a theme, we can set the relevant parameters of the theme to be created through the config parameter, which can override the original default configuration. After creating the topic, we can also add or modify some configurations through the alter instruction and the config parameter to override their original values.

The following example demonstrates changing the max.message.bytes configuration value of topic config from 10000 to 20000. The example is as follows:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper 1ocalhost:2181/ kafka --describe --topic topic-config
Topic:topic-config	PartitionCount:1	ReplicationFactor:1	Configs:cleanup.policy=compact,max.message.bytes=10000
		Topic: topic-config	Partition: 0	Leader: 2	Replicas: 2	Isr: 2

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper 1ocalhost:2181/ kafka --alter --topic topic-config --config max.message.bytes=20000
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
         Going forward, please use kafka-configs.sh for this functionality
Updated config for topic "topic-config".

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper 1ocalhost:2181/ kafka --describe --topic topic-config
Topic:topic-config	PartitionCount:1	ReplicationFactor:1	Configs:max.message.bytes=20000,cleanup.policy=compact
		Topic: topic-config	Partition: 0	Leader: 2	Replicas: 2	Isr: 2

We override another configuration of topic config, segment.bytes (which seems to be equivalent to adding an action), for example:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --alter --topic topic-config --config segment.bytes=1048577
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
         Going forward, please use kafka-configs.sh for this functionality
Updated config for topic "topic-config".

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --describe --topic topic-config
Topic:topic-config	PartitionCount:1	ReplicationFactor:1	Configs:segment.bytes=1048577,cleanup.policy=compact,max.message.bytes=20000
	Topic: topic-config	Partition: 0	Leader: 2	Replicas: 2	Isr: 2

We can use the delete config parameter to delete the previously overwritten configuration and restore it to its original default value. The following example deletes all three modified configurations in topic config:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --alter --topic topic-config --delete-config segment.bytes
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
         Going forward, please use kafka-configs.sh for this functionality
Updated config for topic "topic-config".

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --alter --topic topic-config --delete-config max.message.bytes --delete-config cleanup.policy
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
         Going forward, please use kafka-configs.sh for this functionality
Updated config for topic "topic-config".

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --describe --topic topic-config
Topic:topic-config	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: topic-config	Partition: 0	Leader: 2	Replicas: 2	Isr: 2

It is noted that an alarm message will be prompted after the operation of changing (adding, deleting and changing) the configuration, indicating that the function of using the alter instruction of kafka-topics.sh script to change the theme configuration is obsolete (deprecated) and will be deleted in the future version. It is recommended to use kafka-configs.sh script to realize the relevant functions.

3 delete theme

If you decide not to use a topic anymore, the best way is to delete it, which can free up some resources, such as disks, file handles, etc. The delete instruction in the kafka-topics.sh script can be used to delete a topic, such as deleting a topic delete:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --delete --topic topic-delete
Topic topic-delete is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

You can see that there will be relevant prompt information after executing the delete command, which is related to the broker side configuration parameter delete.topic.enable. The delete.topic.enable parameter must be set to true to delete a topic. The default value of this parameter is true. If it is set to false, the operation of deleting a topic will be ignored. In the actual production environment, it is recommended to set the value of this parameter to true.

If the topic to be deleted is Kafka's internal topic, an error will be reported when deleting. As of Kafka 2.0.0, Kafka has two internal topics, namely__ consumer_offsets and__ transaction_state. The following example attempts to delete an internal theme__ consumer_offsets:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --delete --topic __consumer_offsets
Error while executing topic command : Topic __consumer_offsets is a kafka internal topic and is not allowed to be marked for deletion.
[2018-09-11 11:30:32,635] ERROR kafka.admin.AdminOperationException: Topic __consumer_offsets is a kafka internal topic and is not allowed to be marked for deletion.
...((omit several items)

An error will also be reported if you try to delete a topic that does not exist. For example, in the following example, try to delete a topic unknown that does not exist:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --delete --topic topic-unknown
Error while executing topic command : Topic topic-unknown does not exist on ZK path localhost:2181/kafka
[2018-09-11 23:43:22,186] ERROR java.lang.IllegalArgumentException: Topic topic-unknown does not exist on ZK path localhost:2181/kafka
...((omit several items)

Like the alter instruction, exceptions can also be ignored through the if exists parameter. Refer to the following:

[root@node1 kafka_2.11-2.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181/ kafka --delete --topic topic-unknown --if-exists
[root@node1 kafka_2.11-2.0.0]#

The behavior of deleting a topic using the kafka-topics.sh script is essentially just / admin / delete in ZooKeeper_ Create a node with the same name as the topic to be deleted under the topics path to mark the topic as the status to be deleted. Like creating a theme, Kafka's controller is also responsible for deleting the theme.

After understanding this principle, we can delete the theme directly through ZooKeeper's client. In the following example, the ZooKeeper client zkCli.sh is used to delete the topic delete:

[zk: localhost:2181/kafka (CONNECTED) 15] create /admin/delete_topics/topic-delete ""
Created /admin/delete_topics/topic-delete

We can also delete the theme manually. The metadata in the topic is stored in the / brokers/topics and / config/topics paths in ZooKeeper, and the message data in the topic is stored in the log.dir or log.dirs configured paths. We only need to manually delete the contents in these places. The following example demonstrates how to delete topic delete, which is divided into three steps. The order of the first step and the second step can be interchanged.

Step 1: delete the node / config / topics / topic delete in ZooKeeper.

[zk: localhost:2181/kafka (CONNECTED) 7] delete /config/topics/topic-delete

Step 2: delete the node / brokers / topics / topic delete and its child nodes in ZooKeeper.

[zk: localhost:2181/kafka (CONNECTED) 8] rmr /brokers/topics/topic-delete

Step 3: delete all files related to topic delete in the cluster.

#Execute the RM - RF / TMP / Kafka logs / topic delete * command in each broker node in the cluster to delete files related to topic delete
[root@node1 kafka_2.11-2.0.0]# rm -rf /tmp/kafka-logs/topic-delete*
[root@node2 kafka_2.11-2.0.0]# rm -rf /tmp/kafka-logs/topic-delete*
[root@node3 kafka_2.11-2.0.0]# rm -rf /tmp/kafka-logs/topic-delete*

Note that deleting a topic is an irreversible operation. Once deleted, all message data related to it will be deleted, so you should think twice before performing this operation.

This is the end of the introduction to the kafka-topics.sh script. For the convenience of readers, the following table lists all the parameters in the kafka-topics.sh script. Readers can also view help information by executing the kafka-topics.sh script without any parameters, or executing kafka-topics.sh – help.

Tags: Java kafka message queue

Posted on Tue, 23 Nov 2021 03:34:42 -0500 by watson516