1, Overview
The message consumption of RocketMQ includes two modes: push and pull. The pull mode is officially not recommended, so we mainly introduce the push mode.
Special note: the source code of this paper is based on RocketMQ4.8.
2, Push mode startup process
1. consumer code snippet
package com.example.demo.rocketmq;
import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import java.util.List;
/**
* @author Saint
*/
public class Consumer {
public static void main(String[] args) throws Exception {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("study-consumer");
consumer.setNamesrvAddr("127.0.0.1:9876");
// topic, filter * means no filter
consumer.subscribe("saint-study-topic", "*");
consumer.setConsumeTimeout(20L);
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
// Message propagation mode
consumer.setMessageModel(MessageModel.CLUSTERING);
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext consumeConcurrentlyContext) {
for (MessageExt msg : msgs) {
System.out.println(new String(msg.getBody()));
}
// ack mechanism
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.println("Consumer start. . . . . . ");
}
}
2. Determine the start process entry
In the consumer.start() line, we press F7 to enter the method and find that all its logic is done in the start() method in the DefaultMQPushConsumerImpl class. From here, we can determine that the entry is DefaultMQPushConsumerImpl#start().
There must be a lot of smart people here to ask. What does this traceDispatcher do? Black question marks.
From the comments, we can see that it is used to asynchronously transmit data. By default, it is null, that is, we can't use it normally, so we don't need to spend too much experience to see it.
3. Start process logic
Next, we continue to press F7 to enter the method. You can see that the consumer service is in the state of CREATE_JUST, and then we continue to dig into the interior of a start() method and pull out its pants.
Adhering to the habit of the majority of netizens, we first post the source code and corresponding comments for everyone to understand first.
In fact, when you look at the relatively new version of RocketMQ, you will find that annotations are like rare animals. It's really rare. It can be imagined that before the open source, most of us were Chinese annotations. The open source specified that the Chinese annotations could not be left, which foreigners could not understand. So eat it yourself. Fortunately, the design of RocketMQ is very close to the thinking of Chinese people, and there are not so many design patterns; Relatively easy to understand.
1) DefaultMQPushConsumerImpl#start() method
public synchronized void start() throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST:
log.info("the consumer [{}] start beginning. messageModel={}, isUnitMode={}", this.defaultMQPushConsumer.getConsumerGroup(),
this.defaultMQPushConsumer.getMessageModel(), this.defaultMQPushConsumer.isUnitMode());
// 0. Preset the consumer service status to "failed to start"
this.serviceState = ServiceState.START_FAILED;
// 1. Verify a bunch of configurations, such as the consumerGroup configuration rule, the message propagation method cannot be null (the default is cluster consumption -- CLUSTERING), and the number of concurrent consumption threads.
this.checkConfig();
// 2. copy subscription relationship, listening to the re delivery queue% RETRY%TOPIC.
this.copySubscription();
// 3. If the message propagation mode is cluster mode, modify the name of the consumer instance to PID
if (this.defaultMQPushConsumer.getMessageModel() == MessageModel.CLUSTERING) {
this.defaultMQPushConsumer.changeInstanceNameToPID();
}
// 4. Initialize the MQ client connection factory, where MQClientManager uses starving singleton mode
this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);
// 5. Message reload consumption
// Specify consumption group
this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
// Specify message propagation method
this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
// A queue allocation algorithm that specifies how message queues are allocated to each consumer client.
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
// Specify MQClient factory
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
// 6. Specifies the Pull model request wrapper
this.pullAPIWrapper = new PullAPIWrapper(
mQClientFactory,
this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
// Register message filtering hook
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);
// 7. Specify consumption progress (offset)
if (this.defaultMQPushConsumer.getOffsetStore() != null) {
this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
switch (this.defaultMQPushConsumer.getMessageModel()) {
// The broadcast mode offset is saved locally
case BROADCASTING:
this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
// The cluster mode offset is saved on the server
case CLUSTERING:
this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
default:
break;
}
this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
this.offsetStore.load();
// 8. Create consumer services
if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
// Sequential consumption
this.consumeOrderly = true;
this.consumeMessageService =
new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
// Parallel consumption
this.consumeOrderly = false;
this.consumeMessageService =
new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}
// Start consumer service - scheduled task
this.consumeMessageService.start();
// Register yourself with the broker (consumer)
boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
this.consumeMessageService.shutdown(defaultMQPushConsumer.getAwaitTerminationMillisWhenShutdown());
throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
mQClientFactory.start();
log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
// Change the status of the consumer to "running"
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The PushConsumer service state not OK, maybe started once, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
// Get the monitored topic routing information from the nameServer. If it is changed, modify it.
this.updateTopicSubscribeInfoWhenSubscriptionChanged();
// Check whether the consumer is registered with the broker
this.mQClientFactory.checkClientInBroker();
// Send heartbeat information to all broker s and upload the source file of FilterClass
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
// Wake up ReBalance service thread
this.mQClientFactory.rebalanceImmediately();
}
When we first started a consumer, the state of the consumer client was CREATE_JUST, in the Switch case logic, when serverState is create_ When just, the following logic is executed:
(1) Preset the consumer service status to start failed. This operation is believed that many bosses who have seen the source code of JUC will remember that JUC sits in Doug Lee's programming habit: preset the state first, and the subsequent logic is successfully submitted directly, or roll back.
(2) Then we really enter the startup process of RocketMQ. The first step is a very routine verification operation to verify a bunch of configurations, such as the consumerGroup configuration rules, the message propagation method can not be null (the default is cluster consumption - CLUSTERING), the number of concurrent consumption threads, etc. Interested brothers can follow in by themselves. You will find that this logic is too similar to the code we usually write.
(3) Step 2: copy the copy subscription relationship and listen to the re delivery queue% RETRY%TOPIC. This step is of little significance to the start-up process of our overall consumer, so don't enter, don't enter, and don't enter. Say something important three times.
(4) Step 3: judge whether the message propagation mode is cluster mode. If yes, modify the name of the consumer instance to PID.
(5) Step 4: initialize the MQ client connection factory - MQClientManagerFactory, and then initialize the MQClient. The MQClientManager here uses the starving singleton mode. MQClientInstance encapsulates the RocketMQ network processing API and is the network channel for message producers and consumers to deal with NameServer and Broker. In addition, the MQClientInstance instance obtained by different consumers and producers in the same JVM at startup is the same
(6) Step 5: specify the relevant configuration of message reload, such as consumption group, message propagation method, queue load policy, MQClient factory, etc.
(7) Step 6: specify to create a Pull model request wrapper, which is an API operation wrapper for pulling Broker messages.
(8) Step 7: specify the message consumption progress OffsetStore object and initialize the message consumption progress. In the cluster mode, the message consumption progress offset is saved in the broker, and in the broadcast mode, the message consumption progress offset is saved in the client consumer, that is, the local file. If it is in broadcast mode, the consumption progress file will be loaded from this disk.
From here, we can see that the naming rules of local files are: RocketMQ running directory / ID of MQClientInstance / groupname / offsets.json
(9) Step 8: create different consumemessageservice objects implemented by ConsumeMessageOrderlyService or ConsumeMessageConcurrentlyService for sequential consumption and start the consumption message service -- this is a scheduled task. Consumemessageservice is mainly responsible for message consumption and internally maintains a thread pool. The maximum and minimum number of core threads can be configured through parameters. Note that its blocking queue is unbounded.
(10) Step 9: the consumer registers itself with the broker. If the registration fails, roll back the state of the consumer service instance to CREATE_JUST, and cancel the scheduled task of the started consumption message. Otherwise, change the status of the consumer to RUNNING and start MQClientInstance.
Ho, what did you do to start MQClientInstance?
Lying trough, there are notes. Let's see what it means:
public void start() throws MQClientException {
synchronized (this) {
switch (this.serviceState) {
case CREATE_JUST:
this.serviceState = ServiceState.START_FAILED;
// If the nameserver address is empty, it will go to ` http:// + WS_DOMAIN_NAME + ":8080/rocketmq/" + WS_DOMAIN_SUBGROUP ` get,
// WS_DOMAIN_NAME is set by the configuration parameter rocketmq.namesrv.domain, WS_DOMAIN_SUBG is set by the configuration parameter rocketmq.namesrv.domain.subgroup
if (null == this.clientConfig.getNamesrvAddr()) {
this.mQClientAPIImpl.fetchNameServerAddr();
}
// Open the request and response channel, that is, the API for telematics service, producer and consumer clients to process message sending and consumption.
this.mQClientAPIImpl.start();
/**
* 1.Pull the latest nameServer information regularly for 2min
* 2.The default timing is 30 seconds to pull the latest broker and topic routing information (configurable)
* 3.The default timing is 30s to send heartbeat packets to the broker (configurable)
* 4.offset of default timing 5s persistent consumer (configurable)
* 5.Timed for 1 minute to dynamically adjust the number of threads in the thread pool
*/
this.startScheduledTask();
// Start message pull service
this.pullMessageService.start();
// Start load balancing service
this.rebalanceService.start();
// Start producer message push service
this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
log.info("the client factory [{}] start OK", this.clientId);
this.serviceState = ServiceState.RUNNING;
break;
case START_FAILED:
throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
default:
break;
}
}
}
- Start mqclientapimpl remote communication service, and the API for producer and consumer clients to process message sending and consumption.
- Start various scheduled tasks, such as regularly pulling the latest nameServer, broker and topic information, sending heartbeat to broker, persisting offset and adjusting the number of thread pools.
- Enable the message service pulled from the Broker for consumption by the consumer.
- Open the load balancing service for consumers and consumption queues about message consumption.
There will be a lot of talks in this place. Let's talk about broker related content later.
You may be confused that the state of this serviceState is not changed to START_FAILED? Don't throw exceptions here and exit directly!!!! Note that the serviceState here belongs to MQClientInstance itself, not the serviceState in DefaultMQPushConsumerImpl mentioned above.
The following will be executed regardless of the status of the consumer:
(11) Step 10: get the topic routing information from namesrv and modify it if it is changed.
(12) Step 11: verify the client with the broker and check whether the client is registered with the broker.
(13) Step 12: send heartbeat information to all broker s and upload the source file of FilterClass to FilterServer.
(14) Step 12: wake up the ReBalance service thread and load the queue immediately.
The details of steps (11) - (14) will be introduced in the next consumer subscribe process.
3, Start process in Pull mode
The start process of Pull mode is mainly reflected in the DefaultMQPullConsumerImpl class. Below, we post its start() method:
public synchronized void start() throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST:
this.serviceState = ServiceState.START_FAILED;
this.checkConfig();
this.copySubscription();
if (this.defaultMQPullConsumer.getMessageModel() == MessageModel.CLUSTERING) {
this.defaultMQPullConsumer.changeInstanceNameToPID();
}
this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQPullConsumer, this.rpcHook);
this.rebalanceImpl.setConsumerGroup(this.defaultMQPullConsumer.getConsumerGroup());
this.rebalanceImpl.setMessageModel(this.defaultMQPullConsumer.getMessageModel());
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPullConsumer.getAllocateMessageQueueStrategy());
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
this.pullAPIWrapper = new PullAPIWrapper(
mQClientFactory,
this.defaultMQPullConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);
if (this.defaultMQPullConsumer.getOffsetStore() != null) {
this.offsetStore = this.defaultMQPullConsumer.getOffsetStore();
} else {
switch (this.defaultMQPullConsumer.getMessageModel()) {
case BROADCASTING:
this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPullConsumer.getConsumerGroup());
break;
case CLUSTERING:
this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPullConsumer.getConsumerGroup());
break;
default:
break;
}
this.defaultMQPullConsumer.setOffsetStore(this.offsetStore);
}
this.offsetStore.load();
boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPullConsumer.getConsumerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
throw new MQClientException("The consumer group[" + this.defaultMQPullConsumer.getConsumerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
mQClientFactory.start();
log.info("the consumer [{}] start OK", this.defaultMQPullConsumer.getConsumerGroup());
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The PullConsumer service state not OK, maybe started once, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
}
From the perspective of code, it is almost the same as the start() method of defaultmqpushconsumerismpl class. The difference is that the start() method of defaultmqpushconsumerismpl class has the following fragments after the switch case logic:
// Get the monitored topic routing information from the nameServer. If it is changed, modify it. this.updateTopicSubscribeInfoWhenSubscriptionChanged(); // Check whether the consumer is registered with the broker this.mQClientFactory.checkClientInBroker(); // Send heartbeat information to all broker s and upload the source file of FilterClass this.mQClientFactory.sendHeartbeatToAllBrokerWithLock(); // Wake up ReBalance service thread this.mQClientFactory.rebalanceImmediately();
4, Summary
1. The role of several key categories
DefaultMQPushConsumerImpl is used by the client for message consumption. It creates consummessageservice, message consumption service, message progress saving object OffsetStore, and listener object MessageListener for message consumption.
MQClientInstance opens the request and response channels, that is, the remote communication service; Enable the message pull service PullMessageService and pull messages from the Broker; The load balancing service RebalanceService performs load balancing for consumer s and message queues.
In addition, DefaultMQPushConsumerImpl and MQClientInstance are deployed on the client; Like pulling messages from the Broker, the load balancing of the message queue is completed on the client.
2. consumer start process key points
Mainly check the configuration parameters;
Get MQClientInstance;
Set the consumption group, message propagation mode, load policy and other attributes for the reload service,
Create a pullAPIWrapper to pull messages by long polling;
Load offsetStore according to the message propagation mode;
Select the corresponding ConsumerMessageService consumption service according to whether it is sequential consumption and start it;
Start MQClientInstance, heartbeat broker, wake ReBalance service thread – load queue immediately.
Finally, set up a flag: supplement to the flow chart, which shall be supplemented before October 8, 2021; The supplement of the sequence diagram shall be made before October 9, 2021.
So far, the National Day owes four days of classes. Hi, PI Wan returns to his learning state.
