RocketMQ theory learning and Practice

Message production

Routing table: in fact, it is a map, the key is the topic name, and the value is a QueueData list. Simply put, the key of the routing table is the name of the topic, and the value is the list of all brokernames involving the topic.

send message

Three ways to send messages

You need to specify a callback function

Four states of message sending

SEND_OK sent successfully

FLUSH_DISK_TIMEOUT disk brushing timeout occurs only when the disk brushing strategy is synchronous

FLUSH_SLAVE_TIMEOUT occurs only when the master-slave synchronization timeout occurs in the master slave mode

SLAVE_NOT_AVAILABLE appears only when there is no available slave master slave mode

Consumption news

Examples of message consumers

Message consumption mode

All consumer s consume all messages

Each consumer consumes only the messages they need (default mode)

Sequential consumption

Problem description

By default, the producer will send messages to different queue partition queues by RoundRobin polling; When consuming messages, messages will be pulled from multiple queues. In this case, the order of production and consumption cannot be guaranteed. If you only produce messages to one queue, and then consume messages from only one queue, the order can be guaranteed.

Ordered classification

When there is only one queue for sending and consuming, the order guaranteed is the order of the whole topic, which is called global order.

That is, a topic has only one queue.

If multiple queues participate, they can only ensure the order on the partition queue, which is called partition order.

When defining the Producer, we can specify the message queue selector, which is defined by our own implementation of the MessageQueueSelector interface.

When defining the selector selection algorithm, you generally need to use the selection key. This selection key can be message key or other data, but it must be unique.

The general selection algorithm is to modulo the selected key (or its hash value) and the number of queues contained in the topic, and the result is the QueueId of the selected queue. The general method is to obtain the selected key from the message and judge it. If the current Consumer needs to consume the message, it will consume it, and if not, it will not be processed.

The message that does not belong to the Consumer has been pulled away. Can the Consumer who should consume this message still get this message? Messages in the same Queue cannot be consumed by different consumers in the same Group. Therefore, consumers who consume messages with different selected key s of the same Queue must belong to different groups, and the consumption of consumers in different groups is isolated and does not affect each other.

producer:

How does rocketmq ensure an orderly summary

First, a single queue is ordered, but multiple partitions cannot guarantee that consumers can fetch messages in order. At this time, it is necessary to add a selector to the producer's production message (for example, the order number here) to take the mold and release the partition of the order number, so as to ensure that the relevant messages of the same order are delivered to the same partition, and the messages of the same partition are orderly. Therefore, the consumption of consumers is orderly at this time.

If the consumers are distributed in clusters, there will be multiple consumers with the same consumption in the same partition. At this time, the partition needs to be locked to ensure that only one consumer can enter the partition for consumption at the same time.

Delayed consumption

concept

When a message is written to the broker and consumed after a specified time, it is called a delayed message.

The delay message of rocketMQ is used to realize the function of timed tasks without using a timer. Typical application scenarios are the closing of overdue payment in e-commerce transactions, and the cancellation of ticket booking in 12306 when the order times out without payment.

In the e-commerce platform, an order creation will send a delay message to the background business system (consumer) after 30 minutes. If the order has been closed, it will not be processed; If it is not closed (payment is not completed), the goods are returned to inventory and the order is closed.

Delay level

The delay duration of a delayed message does not support any length of delay, but is classified by level. The delay level is defined in the MessageStoreConfig class of the rocketMQ server.

That is, the delay level is 1 for 1s, and the delay level is 3 for 10s

Delay message producer example

The difference from ordinary sending messages is that a delay level is added.

Transaction message

A scenario solution

In such A scenario, if steps 1, 2 and 3 are successful, that is, after the ICBC system has successfully deducted money, and 5 fails, user A has lost money and user B has not added money, will there be data inconsistency?

The transaction message is needed.

Basic concepts

RocketMQ provides a distributed transaction function similar to X/Open XA. The final consistency of distributed transactions can be achieved through transaction messages. XA is a distributed transaction solution and a distributed transaction processing mode.

For a message that cannot be delivered temporarily, the sender has successfully sent the message to the broker, but the broker has not received the final confirmation instruction. At this time, the message is marked as "temporarily undeliverable", that is, it cannot be seen by the consumer. Messages in this state are called "semi transaction messages".

Message query back, that is, re query the execution status of local transactions. In this example, re-enter the DB to check whether the withholding operation is successful.

Xa is a distributed transaction solution. The distributed transaction processing mode is based on XA protocol.

There are three important components in XA: TC, TM and RM

Transaction Coordinator, Transaction Coordinator. Maintain the status of global and branch transactions and drive global transaction commit or rollback.

broker acts as TC in RocketMQ

Transaction Manager, Transaction Manager. Define the scope of global transaction, start global transaction, commit or roll back global transaction. It is actually the initiator of the global transaction.

producer acts as TM in RocketMQ

Resource Manager, Resource Manager. Manage the resources of branch transactions and drive the registration and rollback of branch transactions.

Both broker and producer in RocketMQ are RM

1. TM sends an instruction to TC to start a global transaction
2. According to business requirements, RM will register branch transactions with TC one by one, and TC will send pre execution instructions to RM one by one
3. RM will perform local transaction pre execution after receiving the pre execution instruction
4. RM returns the pre execution result to TC. Of course, it may succeed or fail
5. After receiving the pre execution results of each RM, TC will summarize them to TM, and TM will issue instructions to TC according to the summary results. If the pre execution results of each RM are successful, send a global commit instruction. If one fails, send a global rollback
6. After receiving the instruction, TC sends a confirmation instruction to RM again

*Note:

• Transaction messages do not support latency
• Transaction messages should be idempotent, because transaction messages may be consumed more than once (because there is a case of committing after rollback)

Transaction listener:

Result: tagB failed because it rolled back, while tagC was successful. So the final messages that consumers can successfully consume are tagA and tagC

Batch message

Send limit

• Messages sent in batches must have the same topic
• Messages sent in batches must have the same disk brushing strategy
• Messages sent in bulk cannot be delayed messages or transaction messages

Code example

In order to prevent messages sent in batches from exceeding 4M (maximum capacity of single message), a message list splitter is defined.

consumer:

Message filtering

When subscribing to a message, consumers can specify not only the topic of the message to be subscribed to, but also more fine-grained filtering conditions than topic through message filtering. There are two main methods: sql filtering and tag filtering.

tag filtering

Specify the message tag to be subscribed through the subscribe() method of the consumer. If you want to subscribe to multiple messages, you can connect with |

sql filtering

Filter the user attributes embedded in the message through specific expressions. Complex message filtering can be realized through sql filtering. However, only push mode consumers can use it.

Message sending retry mechanism

The mechanism by which the producer resends messages that fail to be sent is called the message sending retry mechanism, also known as the message re delivery mechanism.

For message re delivery, you should pay attention to the following points:

• If the producer sends a message synchronously or asynchronously, the sending failure will be retried, but the oneway method will not be retried.
• Only ordinary messages have a send retry mechanism, and sequential messages do not
• The message re delivery mechanism can ensure that the message is sent successfully without loss as much as possible. However, it may cause message duplication, which is an unavoidable problem in rocketmq.
• Message duplication generally does not occur. When there is a large amount of messages and network jitter, message duplication will become a probability event
• The producer takes the initiative to resend, and the consumer's load change (rebalance will not lead to message duplication, but may lead to repeated consumption) will also lead to message duplication
• Message duplication is inevitable, but repeated consumption should be avoided as much as possible
• The solution to avoid repeated consumption of messages is to add a unique identifier (such as a message key) to the message so that consumers can judge the consumption of the message to avoid repeated consumption of messages
• There are three strategies for message sending retry: synchronous sending failure strategy, asynchronous sending failure strategy and message disk brushing failure strategy

If the message is sent again, try to select another broker for retransmission, that is, it will be sent to another queue of another topic. For example, the broker that failed to send a message last time is broker a, so rocketmq thinks that broker a is more likely to fail to send a message repeatedly, so it selects broker B to resend the message. That is, the broker has the function of failure isolation.

An idea to realize failure isolation function:

A concurrentHashmap is maintained in the Producer, where key is the timestamp of sending failure, value is the broker instance, and then a set is maintained, in which the broker instance without failure is stored. The selected target broker is selected from the set set, and then a scheduled task is defined to periodically clean out the brokers that have not been abnormal for a long time from the map set and move them to the set set.

Retry of sequential messages

Note: there is no failed retry policy for sending sequence messages, but there is a failed retry policy for consumption

Retry of unordered messages

For unordered messages (ordinary messages, delay messages, transaction messages), the retry of unordered messages only takes effect for the cluster consumption mode, and the broadcast mode does not provide retry.

Code address

https://gitee.com/gbss/when-iwasfree Rocketmq demo in

Start rocketmq

Start nameserver under bin directory

.\mqnamesrv.cmd

Start the broker in the bin directory

mqbroker -n localhost:9876 -c ../conf/broker.conf autoCreateTopicEnable=true

Start rocketmq console

Gitee (or github) Download rocketmq console

Refer to:

https://gitee.com/ralph81/rocketmq-console?_from=gitee_search

Just modify the address and port of the configuration file

For the convenience of execution, it can be printed into a jar package, and then only Java jar is needed

Unable to create topic error:

Solution:

https://my.oschina.net/u/3476125/blog/897429

In this way, the broker will not connect to the remote ip

success!:

Examples of multiple consumer consumption messages

The current situation is that each group (consumer producer) has only one instance, so production and consumption are both these (because topic will be selected). So when there are multiple consumers in a group, who will consume them? Or all consumption?

What happens when different groups of consumers grab the same message?

To verify this problem, you must create multiple instances in the same consumer group, so you need a consumer factory.

/** * Consumer manufacturing plant */ @Component public class ConsumerFactory { // Store all consumer instances, and key is the unique ID public static Map<String, DefaultMQPushConsumer> allDefaultMQPushConsumer; // Store all consumer instance names public static List<String> allDefaultMQPushConsumerNames; private static String nameSvrAddr = "localhost:9876"; public ConsumerFactory() { allDefaultMQPushConsumer = new HashMap<String, DefaultMQPushConsumer>(); allDefaultMQPushConsumerNames = new ArrayList<String>(); } /** * Get all consumer names * @return */ public static List<String> getAllDefaultMQPushConsumerNames(){ return allDefaultMQPushConsumerNames; } /** * Get existing consumer * @param consumerGroup * @param instanceName */ public static DefaultMQPushConsumer getSingleDefaultMQPushConsumer(String consumerGroup, String instanceName){ String key = consumerGroup + instanceName; if(!allDefaultMQPushConsumer.containsKey(key)){ System.out.println("Do not include this key"); return null; } return allDefaultMQPushConsumer.get(key); } /** * Create a consumer * @param consumerGroup Consumer group name * @param topic topic of consumption * @param tag Fill in "*" for all consumption tag s * @return */ public static String getSingleDefaultMQPushConsumer(String consumerGroup, String instanceName, String topic, String tag) throws MQClientException { DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(consumerGroup); consumer.setNamesrvAddr(nameSvrAddr); consumer.setInstanceName(instanceName); // Specify topic and tag consumer.subscribe(topic, tag); // consumerGroup + instanceName as key final String key = consumerGroup + instanceName; try { consumer.registerMessageListener(new MessageListenerConcurrently() { public ConsumeConcurrentlyStatus consumeMessage( List<MessageExt> msgs, ConsumeConcurrentlyContext context) { for (MessageExt msg : msgs) { System.out.println(key + " Received Message: " + new String(msg.getBody())); } return ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } }); consumer.start(); System.out.println("consumer init success"); allDefaultMQPushConsumerNames.add(key); allDefaultMQPushConsumer.put(key, consumer); } catch (MQClientException e) { System.out.println("establish consumer fail" + e.getErrorMessage()); } return "establish consumer success"; } /** * Modify nameSvrAddr */ public void changeNameSvrAddr(String newName) { nameSvrAddr = newName; } @PreDestroy private void destroyAllConsumer() { if (allDefaultMQPushConsumer != null && allDefaultMQPushConsumer.size() != 0) { for(String name : allDefaultMQPushConsumerNames){ DefaultMQPushConsumer defaultMQPushConsumer = allDefaultMQPushConsumer.get(name); defaultMQPushConsumer.shutdown(); } } System.out.println("All consumer All instances have been closed"); } }

Two scenarios:

Initialize multiple consumers with the same consumer group, topic and tag

consumer:

{ "consumerGroup":"defaultGroup", "instanceName":"consumer01", "topic":"topic_test", "tag":"*" } { "consumerGroup":"defaultGroup", "instanceName":"consumer02", "topic":"topic_test", "tag":"*" } { "consumerGroup":"defaultGroup", "instanceName":"consumer03", "topic":"topic_test", "tag":"*" }

At this time, send a message and get the following results:

It can be seen that only one consumer in the same consumer group can consume messages

Initialize multiple consumers with different consumer groups but the same topic and tag

{ "consumerGroup":"Group01", "instanceName":"consumer", "topic":"topic_B", "tag":"*" } { "consumerGroup":"Group02", "instanceName":"consumer", "topic":"topic_B", "tag":"*" } { "consumerGroup":"Group03", "instanceName":"consumer", "topic":"topic_B", "tag":"*" }

This time we consume topic_B

You can see that for the same message, each consumer group has one consumer.

conclusion

If you want a message to be consumed by multiple services, these services must be scattered in different consumer groups

The same message can only be consumed by one consumer in one consumer group at most

Add new comment

0 comments