RocketMQ source code analysis - Consumer

catalog:

Understand the message Consumer:

Message consumption is carried out in group mode. A consumption group can contain multiple consumers. Each consumer can subscribe to multiple topics. There are two consumption modes between consumption groups: cluster (load balancing) mode and broadcast mode:

  • Cluster (load balancing) mode: only one message consumer is allowed to consume the same message under the topic
  • Broadcast mode: the same message under the theme will be consumed by all consumers in the cluster once

There are also two modes of message delivery between message consumers and servers, push mode and pull mode:

  • Push mode: after the message arrives at the server, the server pushes the message to the message consumer (the so-called push mode is actually implemented based on the pull mode, which will be explained in the following source code analysis)
  • Pull mode: when the message arrives at the server, the message consumer actively pulls the message from the server

The common message consumption mode is push mode, and the core class in push mode is DefaultMQPushConsumer:


It can be seen from the figure that two core parent class interfaces are implemented on the DefaultMQPushConsumer: MQPushConsumer and MQConsumer. To learn more about the DefaultMQPushConsumer class, let's take a look at the important methods of its parent class interface:

/**
* Gets the message queue that listens to the specified topic
*/
Set<MessageQueue> fetchSubscribeMessageQueues(final String topic) throws MQClientException;

/**
* Register concurrent event listeners
*/
void registerMessageListener(final MessageListenerConcurrently messageListener);

/**
* Register sequential event listener
*/
void registerMessageListener(final MessageListenerOrderly messageListener);

/**
* Subscribe to messages based on topics and filter messages using expressions
*/
void subscribe(final String topic, final String subExpression) throws MQClientException;

/**
* Subscribe to messages based on topics and specify a queue selector to filter messages
*/
void subscribe(final String topic, final MessageSelector selector) throws MQClientException;

/**
* Cancels the subscriber under the specified topic
*/
void unsubscribe(final String topic);

The above is the core method of the message consumer. Next, learn more about the important attributes of the DefaultMQPushConsumer implementation class:

protected final transient DefaultMQPushConsumerImpl defaultMQPushConsumerImpl; // Message push function implementation class

private String consumerGroup; // Consumer group

private MessageModel messageModel = MessageModel.CLUSTERING; // Consumption mode (cluster mode by default)

private ConsumeFromWhere consumeFromWhere = ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET; // Consumption start offset

private AllocateMessageQueueStrategy allocateMessageQueueStrategy; // Load policy of message queue in cluster mode

private Map<String /* topic */, String /* sub expression */> subscription = new HashMap<String, String>(); // Topic and message filter expression mapping table

private MessageListener messageListener; // Message listener

private OffsetStore offsetStore; // Message consumption progress memory

private int consumeThreadMin = 20; // Minimum number of threads for message consumers

private int consumeThreadMax = 20; // Maximum number of threads for message consumers

private int consumeConcurrentlyMaxSpan = 2000; // Maximum span of processing queue when concurrent messages are consumed

private long pullInterval = 0; // Pull message task interval

private int consumeMessageBatchMaxSize = 1; // Maximum number of messages consumed each time

private int pullBatchSize = 32; // Number of pulled messages

private boolean postSubscriptionWhenPull = false; // Whether to subscribe to messages every time you pull messages (default: false)

private boolean unitMode = false; // Whether to subscribe to messages in groups (false by default)

private int maxReconsumeTimes = -1; // Maximum number of retries for message consumption (the default is - 1, which means that you can retry up to 16 times)

private long consumeTimeout = 15; // Message consumption timeout

private long awaitTerminationMillisWhenShutdown = 0; // Waiting time before closing the consumer (considering that there may be messages being consumed, the default is no waiting)

Source code analysis:

We learned about the key methods and attributes of consumers in the previous section. This section starts to analyze the source code
First, find the consumer's startup portal:

public class Consumer {
    public static void main(String[] args) throws InterruptedException, MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_4"); // Create a consumer based on push mode and specify the group name
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET); // Set consumption start offset
        consumer.subscribe("TopicTest", "*"); // Subscribe to messages with tag matching expression conditions under the specified topic
        consumer.registerMessageListener(new MessageListenerConcurrently() { // Register concurrent listeners
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
                ConsumeConcurrentlyContext context) {
                System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start(); // Start consumer
        System.out.printf("Consumer Started.%n");
    }
}
  • Subscription message:
public void subscribe(String topic, String subExpression) throws MQClientException {
    this.defaultMQPushConsumerImpl.subscribe(withNamespace(topic), subExpression);
}
public void subscribe(String topic, String subExpression) throws MQClientException {
    try {
        SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(topic, subExpression); // Constructing subscription data objects based on topic and tag matching expressions
        this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData); // Save subscription data object
        if (this.mQClientFactory != null) {
            this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
        }
    } catch (Exception e) {
        throw new MQClientException("subscription exception", e);
    }
}
  • Register listener:
public void registerMessageListener(MessageListenerConcurrently messageListener) {
    this.messageListener = messageListener;
    this.defaultMQPushConsumerImpl.registerMessageListener(messageListener); // Register the listener in the consumer implementation base class
}
public void registerMessageListener(MessageListener messageListener) {
    this.messageListenerInner = messageListener;
}
  • Start consumer:

In order to understand the following source code more intuitively, first get familiar with the overall startup process of consumers, as shown in the figure below:


Based on the flow diagram above, learn about the specific source code implementation:

public void start() throws MQClientException {
    setConsumerGroup(NamespaceUtil.wrapNamespace(this.getNamespace(), this.consumerGroup)); // Set consumption group
    this.defaultMQPushConsumerImpl.start(); // Start consumer implementation base class
    if (null != traceDispatcher) {
        try {
            traceDispatcher.start(this.getNamesrvAddr(), this.getAccessChannel());
        } catch (MQClientException e) {
            log.warn("trace dispatcher start failed ", e);
        }
    }
}
public synchronized void start() throws MQClientException {
	// Execute the corresponding logic according to the service status
    switch (this.serviceState) {
        case CREATE_JUST: // Created but not started (initial state)
            log.info("the consumer [{}] start beginning. messageModel={}, isUnitMode={}", this.defaultMQPushConsumer.getConsumerGroup(), this.defaultMQPushConsumer.getMessageModel(), this.defaultMQPushConsumer.isUnitMode());
            
            this.serviceState = ServiceState.START_FAILED; // First identify the status as startup failure
            this.checkConfig(); // Check configuration
            this.copySubscription(); // Copy (build) subscription data
            if (this.defaultMQPushConsumer.getMessageModel() == MessageModel.CLUSTERING) { // If the consumption mode is cluster mode, the consumer client instance name is the process ID
                this.defaultMQPushConsumer.changeInstanceNameToPID();
            }
			
            this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook); // Building consumer client objects

			// Building rebalance objects
            this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
            this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
            this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
            this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);

            this.pullAPIWrapper = new PullAPIWrapper(
                mQClientFactory,
                this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
            this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);

            if (this.defaultMQPushConsumer.getOffsetStore() != null) {
                this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
            } else {
                switch (this.defaultMQPushConsumer.getMessageModel()) {
                    case BROADCASTING: // Broadcast mode to save the consumption progress locally
                        this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                        break;
                    case CLUSTERING: // In the cluster mode, the consumption progress is saved in the remote Broker
                        this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                        break;
                    default:
                        break;
                }
                this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
            }
            this.offsetStore.load();

            if (this.getMessageListenerInner() instanceof MessageListenerOrderly) { // The listener type is sequential listener
            	// Create sequential message consumption service
                this.consumeOrderly = true;
                this.consumeMessageService = new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
            } else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) { // The listener type is concurrent listener
            	// Create concurrent message consumption service
                this.consumeOrderly = false;
                this.consumeMessageService = new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
            }

            this.consumeMessageService.start(); // Start message consumption service

            boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this); // Register the consumer instance object for the consumer client
            if (!registerOK) { // Registration failed, error thrown
                this.serviceState = ServiceState.CREATE_JUST;
                this.consumeMessageService.shutdown(defaultMQPushConsumer.getAwaitTerminationMillisWhenShutdown());
                throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup() + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL), null);
            }

            mQClientFactory.start(); // Start consumer client
            
            log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
            
            this.serviceState = ServiceState.RUNNING; // After the client is started, set the service status to running
            break;
        // Other states, throw wrong
        case RUNNING:
        case START_FAILED:
        case SHUTDOWN_ALREADY:
            throw new MQClientException("The PushConsumer service state not OK, maybe started once, " + this.serviceState + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK), null);
        default:
            break;
    }

    this.updateTopicSubscribeInfoWhenSubscriptionChanged();
    this.mQClientFactory.checkClientInBroker();
    this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
    this.mQClientFactory.rebalanceImmediately();
}
  • Start message consumption service:

There are two types of message consumption services: sequential service and concurrent service. Because concurrent service types are more commonly used, the following source codes are analyzed based on the source code of concurrent services

public void start() {
    this.cleanExpireMsgExecutors.scheduleAtFixedRate(new Runnable() { // Start scheduled task
        @Override
        public void run() {
            cleanExpireMsg(); // Clean up expired messages
        }
    }, this.defaultMQPushConsumer.getConsumeTimeout(), this.defaultMQPushConsumer.getConsumeTimeout(), TimeUnit.MINUTES);
}
  • Registered consumer instance:
public boolean registerConsumer(final String group, final MQConsumerInner consumer) {
    if (null == group || null == consumer) {
        return false;
    }
    MQConsumerInner prev = this.consumerTable.putIfAbsent(group, consumer); // Save to the consumer instance mapping table in the form of < group name, consumer instance >
    if (prev != null) {
        log.warn("the consumer group[" + group + "] exist already.");
        return false;
    }
    return true;
}
  • Start consumer client:
public void start() throws MQClientException {
    synchronized (this) {
    	// Execute the corresponding logic according to the service status
        switch (this.serviceState) {
            case CREATE_JUST: // Created but not started (initial state)
                this.serviceState = ServiceState.START_FAILED; // First identify the service status as startup failure
                if (null == this.clientConfig.getNamesrvAddr()) { // Get the NameServer address. If the local cache is empty, get it remotely
                    this.mQClientAPIImpl.fetchNameServerAddr();
                }
                this.mQClientAPIImpl.start(); // Open the connection channel between client and server
                this.startScheduledTask(); // Start the data update plan task (NameServer address, routing information, sending heartbeat information to Broker, etc.)
                this.pullMessageService.start(); // Start pull message service
                this.rebalanceService.start(); // Start the create message pull request service
                this.defaultMQProducer.getDefaultMQProducerImpl().start(false); // Start producer push service
                log.info("the client factory [{}] start OK", this.clientId);
                this.serviceState = ServiceState.RUNNING; // The above startup operations are successful. Set the service status to running
                break;
            case START_FAILED: // Startup failed, error thrown
                throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
            default:
                break;
        }
    }
}

Through the above source code, you can see that many services have been started in the consumer client, but the key service should be the pull message service. At the beginning of this article, the message push mode is also implemented based on the message pull mode, and the key is here

  • Start message pull service:


The so-called PullMessageService is actually a thread class, so pay attention to its thread task run method:

public void run() {
     log.info(this.getServiceName() + " service started");

     while (!this.isStopped()) {
         try {
             PullRequest pullRequest = this.pullRequestQueue.take(); // Get pull request from pull request blocking queue
             this.pullMessage(pullRequest); // Pull message
         } catch (InterruptedException ignored) {
         } catch (Exception e) {
             log.error("Pull Message Service Run Method exception", e);
         }
     }

     log.info(this.getServiceName() + " service end");
 }
private void pullMessage(final PullRequest pullRequest) {
    final MQConsumerInner consumer = this.mQClientFactory.selectConsumer(pullRequest.getConsumerGroup()); // Get the consumer instance according to the group name
    if (consumer != null) {
        DefaultMQPushConsumerImpl impl = (DefaultMQPushConsumerImpl) consumer; // Strong conversion to push type consumer instance
        impl.pullMessage(pullRequest); // Pull a message by pulling a message request
    } else {
        log.warn("No matched consumer for the PullRequest {}, drop it", pullRequest);
    }
}
  • Message pull process:

public void pullMessage(final PullRequest pullRequest) {
    final ProcessQueue processQueue = pullRequest.getProcessQueue(); // Get information processing queue from pull message request
    if (processQueue.isDropped()) { // The processing queue has been discarded and returned directly
        log.info("the pull request[{}] is dropped.", pullRequest.toString());
        return;
    }

    pullRequest.getProcessQueue().setLastPullTimestamp(System.currentTimeMillis()); // Processing queue available, update timestamp

    try {
        this.makeSureStateOK(); // Check status
    } catch (MQClientException e) {
        log.warn("pullMessage exception, consumer state not ok", e);
        this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
        return;
    }

    if (this.isPause()) {
        log.warn("consumer was paused, execute pull request later. instanceName={}, group={}", this.defaultMQPushConsumer.getInstanceName(), this.defaultMQPushConsumer.getConsumerGroup());
        this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_SUSPEND);
        return;
    }

    long cachedMessageCount = processQueue.getMsgCount().get(); // Gets the maximum number of pending messages
    long cachedMessageSizeInMiB = processQueue.getMsgSize().get() / (1024 * 1024); // Gets the maximum pending message size

	// Message quantity flow control
    if (cachedMessageCount > this.defaultMQPushConsumer.getPullThresholdForQueue()) {
        this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
        if ((queueFlowControlTimes++ % 1000) == 0) {
            log.warn(
                "the cached message count exceeds the threshold {}, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, pullRequest={}, flowControlTimes={}",
                this.defaultMQPushConsumer.getPullThresholdForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, pullRequest, queueFlowControlTimes);
        }
        return;
    }

	// Message size flow control
    if (cachedMessageSizeInMiB > this.defaultMQPushConsumer.getPullThresholdSizeForQueue()) {
        this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
        if ((queueFlowControlTimes++ % 1000) == 0) {
            log.warn(
                "the cached message size exceeds the threshold {} MiB, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, pullRequest={}, flowControlTimes={}",
                this.defaultMQPushConsumer.getPullThresholdSizeForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, pullRequest, queueFlowControlTimes);
        }
        return;
    }

    if (!this.consumeOrderly) {
        if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
            this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
            if ((queueMaxSpanFlowControlTimes++ % 1000) == 0) {
                log.warn(
                    "the queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, pullRequest={}, flowControlTimes={}",
                    processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(),
                    pullRequest, queueMaxSpanFlowControlTimes);
            }
            return;
        }
    } else {
        if (processQueue.isLocked()) {
            if (!pullRequest.isPreviouslyLocked()) {
                long offset = -1L;
                try {
                    offset = this.rebalanceImpl.computePullFromWhereWithException(pullRequest.getMessageQueue());
                } catch (MQClientException e) {
                    this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
                    log.error("Failed to compute pull offset, pullResult: {}", pullRequest, e);
                    return;
                }
                boolean brokerBusy = offset < pullRequest.getNextOffset();
                log.info("the first time to pull message, so fix offset from broker. pullRequest: {} NewOffset: {} brokerBusy: {}",
                    pullRequest, offset, brokerBusy);
                if (brokerBusy) {
                    log.info("[NOTIFYME]the first time to pull message, but pull request offset larger than broker consume offset. pullRequest: {} NewOffset: {}",
                        pullRequest, offset);
                }

                pullRequest.setPreviouslyLocked(true);
                pullRequest.setNextOffset(offset);
            }
        } else {
            this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
            log.info("pull message later because not locked in broker, {}", pullRequest);
            return;
        }
    }

    final SubscriptionData subscriptionData = this.rebalanceImpl.getSubscriptionInner().get(pullRequest.getMessageQueue().getTopic()); // Get subscription information data
    if (null == subscriptionData) {
        this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
        log.warn("find the consumer's subscription failed, {}", pullRequest);
        return;
    }

    final long beginTimestamp = System.currentTimeMillis();

    PullCallback pullCallback = new PullCallback() { // Define the pull message callback function (it will not be explained here, but will be described in detail later in this article)
        @Override
        public void onSuccess(PullResult pullResult) {
            ...
        }
    };

    boolean commitOffsetEnable = false;
    long commitOffsetValue = 0L;
    if (MessageModel.CLUSTERING == this.defaultMQPushConsumer.getMessageModel()) {
        commitOffsetValue = this.offsetStore.readOffset(pullRequest.getMessageQueue(), ReadOffsetType.READ_FROM_MEMORY);
        if (commitOffsetValue > 0) {
            commitOffsetEnable = true;
        }
    }

    String subExpression = null;
    boolean classFilter = false;
    SubscriptionData sd = this.rebalanceImpl.getSubscriptionInner().get(pullRequest.getMessageQueue().getTopic());
    if (sd != null) {
        if (this.defaultMQPushConsumer.isPostSubscriptionWhenPull() && !sd.isClassFilterMode()) {
            subExpression = sd.getSubString();
        }

        classFilter = sd.isClassFilterMode();
    }

    int sysFlag = PullSysFlag.buildSysFlag(
        commitOffsetEnable, // commitOffset
        true, // suspend
        subExpression != null, // subscription
        classFilter // class filter
    );
    try {
        this.pullAPIWrapper.pullKernelImpl(
            pullRequest.getMessageQueue(),
            subExpression,
            subscriptionData.getExpressionType(),
            subscriptionData.getSubVersion(),
            pullRequest.getNextOffset(),
            this.defaultMQPushConsumer.getPullBatchSize(),
            sysFlag,
            commitOffsetValue,
            BROKER_SUSPEND_MAX_TIME_MILLIS,
            CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND,
            CommunicationMode.ASYNC, // asynchronous
            pullCallback
        ); // Interact with the server to pull and obtain messages
    } catch (Exception e) {
        log.error("pullKernelImpl exception", e);
        this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
    }
}
public PullResult pullKernelImpl(
    final MessageQueue mq,
    final String subExpression,
    final String expressionType,
    final long subVersion,
    final long offset,
    final int maxNums,
    final int sysFlag,
    final long commitOffset,
    final long brokerSuspendMaxTimeMillis,
    final long timeoutMillis,
    final CommunicationMode communicationMode,
    final PullCallback pullCallback
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
	// Get the Broker information of the corresponding consumer instance from the local cache. If the local cache is empty, update the remote NameServer routing information and then get it again
    FindBrokerResult findBrokerResult =
        this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
            this.recalculatePullFromWhichNode(mq), false);
    if (null == findBrokerResult) {
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
        findBrokerResult =
            this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
                this.recalculatePullFromWhichNode(mq), false);
    }

    if (findBrokerResult != null) {
        {
            // Check version
            if (!ExpressionType.isTagType(expressionType)
                && findBrokerResult.getBrokerVersion() < MQVersion.Version.V4_1_0_SNAPSHOT.ordinal()) {
                throw new MQClientException("The broker[" + mq.getBrokerName() + ", "
                    + findBrokerResult.getBrokerVersion() + "] does not upgrade to support for filter message by " + expressionType, null);
            }
        }
        int sysFlagInner = sysFlag;

        if (findBrokerResult.isSlave()) {
            sysFlagInner = PullSysFlag.clearCommitOffsetFlag(sysFlagInner);
        }

		// Encapsulate pull message request header
        PullMessageRequestHeader requestHeader = new PullMessageRequestHeader();
        requestHeader.setConsumerGroup(this.consumerGroup);
        requestHeader.setTopic(mq.getTopic());
        requestHeader.setQueueId(mq.getQueueId());
        requestHeader.setQueueOffset(offset);
        requestHeader.setMaxMsgNums(maxNums);
        requestHeader.setSysFlag(sysFlagInner);
        requestHeader.setCommitOffset(commitOffset);
        requestHeader.setSuspendTimeoutMillis(brokerSuspendMaxTimeMillis);
        requestHeader.setSubscription(subExpression);
        requestHeader.setSubVersion(subVersion);
        requestHeader.setExpressionType(expressionType);

        String brokerAddr = findBrokerResult.getBrokerAddr(); // Get Broker address
        if (PullSysFlag.hasClassFilterFlag(sysFlagInner)) {
            brokerAddr = computePullFromWhichFilterServer(mq.getTopic(), brokerAddr);
        }

        PullResult pullResult = this.mQClientFactory.getMQClientAPIImpl().pullMessage(
            brokerAddr,
            requestHeader,
            timeoutMillis,
            communicationMode,
            pullCallback); // Send asynchronous requests, pull messages, and return pull results

        return pullResult;
    }

    throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}

The interaction mode between the consumer client and the server is determined according to the passed in parameters. Here is asynchronous:

public PullResult pullMessage(
    final String addr,
    final PullMessageRequestHeader requestHeader,
    final long timeoutMillis,
    final CommunicationMode communicationMode,
    final PullCallback pullCallback
) throws RemotingException, MQBrokerException, InterruptedException {
    RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.PULL_MESSAGE, requestHeader);

    switch (communicationMode) {
        case ONEWAY:
            assert false;
            return null;
        case ASYNC: // Focus on asynchrony
            this.pullMessageAsync(addr, request, timeoutMillis, pullCallback);
            return null;
        case SYNC:
            return this.pullMessageSync(addr, request, timeoutMillis);
        default:
            assert false;
            break;
    }

    return null;
}
private void pullMessageAsync(
    final String addr,
    final RemotingCommand request,
    final long timeoutMillis,
    final PullCallback pullCallback
) throws RemotingException, InterruptedException {
	// Initiate an asynchronous request, get the message from the Broker, and process the response result in the callback function
    this.remotingClient.invokeAsync(addr, request, timeoutMillis, new InvokeCallback() {
        @Override
        public void operationComplete(ResponseFuture responseFuture) {
            RemotingCommand response = responseFuture.getResponseCommand();
            if (response != null) {
                try {
                    PullResult pullResult = MQClientAPIImpl.this.processPullResponse(response, addr); // Processing response results
                    assert pullResult != null;
                    pullCallback.onSuccess(pullResult); // The callback function successfully processed the result
                } catch (Exception e) {
                    pullCallback.onException(e); // The callback function handles the exception result
                }
            } else {
                if (!responseFuture.isSendRequestOK()) {
                    pullCallback.onException(new MQClientException("send request failed to " + addr + ". Request: " + request, responseFuture.getCause()));
                } else if (responseFuture.isTimeout()) {
                    pullCallback.onException(new MQClientException("wait response from " + addr + " timeout :" + responseFuture.getTimeoutMillis() + "ms" + ". Request: " + request,
                        responseFuture.getCause()));
                } else {
                    pullCallback.onException(new MQClientException("unknown reason. addr: " + addr + ", timeoutMillis: " + timeoutMillis + ". Request: " + request, responseFuture.getCause()));
                }
            }
        }
    });
}

At this point, I finally see that the request to pull the message has been sent. How does the receiving Broker handle the request? What is included in the returned response result?

  • Broker receives and processes requests:


Find the method entry for the Broker to process the pull message request:

private RemotingCommand processRequest(final Channel channel, RemotingCommand request, boolean brokerAllowSuspend) throws RemotingCommandException {
	...
	// Create message filter
    MessageFilter messageFilter;
    if (this.brokerController.getBrokerConfig().isFilterSupportRetry()) {
        messageFilter = new ExpressionForRetryMessageFilter(subscriptionData, consumerFilterData, this.brokerController.getConsumerFilterManager());
    } else {
        messageFilter = new ExpressionMessageFilter(subscriptionData, consumerFilterData, this.brokerController.getConsumerFilterManager());
    }

	// Obtain the message according to the group name, subject, queue id, queue offset, maximum number of obtained messages and message filter conditions of the request header
    final GetMessageResult getMessageResult = this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
            requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter);
    if (getMessageResult != null) {
    	// Encapsulate response results
        response.setRemark(getMessageResult.getStatus().name());
        responseHeader.setNextBeginOffset(getMessageResult.getNextBeginOffset());
        responseHeader.setMinOffset(getMessageResult.getMinOffset());
        responseHeader.setMaxOffset(getMessageResult.getMaxOffset());

        if (getMessageResult.isSuggestPullingFromSlave()) {
            responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly());
        } else {
            responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
        }

        switch (this.brokerController.getMessageStoreConfig().getBrokerRole()) {
            case ASYNC_MASTER:
            case SYNC_MASTER:
                break;
            case SLAVE:
                if (!this.brokerController.getBrokerConfig().isSlaveReadEnable()) {
                    response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
                    responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
                }
                break;
        }

        if (this.brokerController.getBrokerConfig().isSlaveReadEnable()) {
            // consume too slow ,redirect to another machine
            if (getMessageResult.isSuggestPullingFromSlave()) {
                responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly());
            }
            // consume ok
            else {
                responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getBrokerId());
            }
        } else {
            responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
        }

        switch (getMessageResult.getStatus()) {
            case FOUND: // Message obtained successfully
                response.setCode(ResponseCode.SUCCESS); 
                break;
            case MESSAGE_WAS_REMOVING: // The message exists in the next commitLog file. You need to retry pulling the message
                response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY); 
                break;
            case NO_MATCHED_LOGIC_QUEUE: // Queue not found
            case NO_MESSAGE_IN_QUEUE: // No packet messages in queue
                if (0 != requestHeader.getQueueOffset()) {
                    response.setCode(ResponseCode.PULL_OFFSET_MOVED);
                    // XXX: warn and notify me
                    log.info("the broker store no queue data, fix the request offset {} to {}, Topic: {} QueueId: {} Consumer Group: {}",
                        requestHeader.getQueueOffset(),
                        getMessageResult.getNextBeginOffset(),
                        requestHeader.getTopic(),
                        requestHeader.getQueueId(),
                        requestHeader.getConsumerGroup()
                    );
                } else {
                    response.setCode(ResponseCode.PULL_NOT_FOUND);
                }
                break;
            case NO_MATCHED_MESSAGE: // No message found. You need to retry pulling the message
                response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY); 
                break;
            case OFFSET_FOUND_NULL: // The message found according to the specified offset is null
                response.setCode(ResponseCode.PULL_NOT_FOUND);
                break;
            case OFFSET_OVERFLOW_BADLY: // Offset out of range
                response.setCode(ResponseCode.PULL_OFFSET_MOVED);
                // XXX: warn and notify me
                log.info("the request offset: {} over flow badly, broker max offset: {}, consumer: {}",
                    requestHeader.getQueueOffset(), getMessageResult.getMaxOffset(), channel.remoteAddress());
                break;
            case OFFSET_OVERFLOW_ONE: // Offset not found in queue
                response.setCode(ResponseCode.PULL_NOT_FOUND);
                break;
            case OFFSET_TOO_SMALL: // Offset not in queue
                response.setCode(ResponseCode.PULL_OFFSET_MOVED);
                log.info("the request offset too small. group={}, topic={}, requestOffset={}, brokerMinOffset={}, clientIp={}",
                    requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueOffset(),
                    getMessageResult.getMinOffset(), channel.remoteAddress());
                break;
            default:
                assert false;
                break;
        }
	}
	...
	// If the commitLog is marked as available and the Broker of the current operation is the master node, the consumption progress is updated
    boolean storeOffsetEnable = brokerAllowSuspend;
    storeOffsetEnable = storeOffsetEnable && hasCommitOffsetFlag;
    storeOffsetEnable = storeOffsetEnable && this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE;
    if (storeOffsetEnable) {
        this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(channel),
            requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset());
    }
    return response;
}

The above is the core logic for Broke to handle the message pull request and return the response
Next, let's go back to the consumer and see how to deal with the response result

  • Processing response results:
private PullResult processPullResponse(final RemotingCommand response, final String addr) throws MQBrokerException, RemotingCommandException {
	// The pull message status is determined according to the response result return code
    PullStatus pullStatus = PullStatus.NO_NEW_MSG;
    switch (response.getCode()) {
        case ResponseCode.SUCCESS: // Message obtained successfully
            pullStatus = PullStatus.FOUND;
            break;
        case ResponseCode.PULL_NOT_FOUND: // Message acquisition failed
            pullStatus = PullStatus.NO_NEW_MSG; 
            break;
        case ResponseCode.PULL_RETRY_IMMEDIATELY: // No matching message found, need to try again
            pullStatus = PullStatus.NO_MATCHED_MSG;
            break;
        case ResponseCode.PULL_OFFSET_MOVED: // The offset of the specified pull message is incorrect
            pullStatus = PullStatus.OFFSET_ILLEGAL;
            break;
        default:
            throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
    }

	// Encapsulate and return the pull result information
    PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) response.decodeCommandCustomHeader(PullMessageResponseHeader.class);
    return new PullResultExt(pullStatus, responseHeader.getNextBeginOffset(), responseHeader.getMinOffset(),
        responseHeader.getMaxOffset(), null, responseHeader.getSuggestWhichBrokerId(), response.getBody());
}
  • Callback function processing success / failure result:
PullCallback pullCallback = new PullCallback() {
    @Override
    public void onSuccess(PullResult pullResult) {
        if (pullResult != null) {
            pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult, subscriptionData);
            // Execute the corresponding logic according to the pulled message results (focus on the successful results)
            switch (pullResult.getPullStatus()) {
                case FOUND:
                    long prevRequestOffset = pullRequest.getNextOffset();
                    pullRequest.setNextOffset(pullResult.getNextBeginOffset());
                    long pullRT = System.currentTimeMillis() - beginTimestamp;
                    DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(), pullRequest.getMessageQueue().getTopic(), pullRT);

                    long firstMsgOffset = Long.MAX_VALUE;
                    if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
                        DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                    } else {
                        firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();
                        DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(), pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

                        boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList()); // Store the pulled message in processQueue
                        DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                            pullResult.getMsgFoundList(),
                            processQueue,
                            pullRequest.getMessageQueue(),
                            dispatchToConsume); // Submit the processQueue to the consumer message service

						// If pullInterval > 0, wait for pullInterval milliseconds before performing the next pull message operation; Otherwise, execute directly and pull the message
                        if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                            DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
                        } else {
                            DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        }
                    }

                    if (pullResult.getNextBeginOffset() < prevRequestOffset
                        || firstMsgOffset < prevRequestOffset) {
                        log.warn(
                            "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                            pullResult.getNextBeginOffset(),
                            firstMsgOffset,
                            prevRequestOffset);
                    }
                    break;
               	...
                default:
                    break;
            }
        }
    }

	/**
	* Handling exception results
	*/
    @Override
    public void onException(Throwable e) { 
        if (!pullRequest.getMessageQueue().getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
            log.warn("execute the pull request exception", e);
        }

        DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
    }
};
  • Submit processQueue to the consumer message service:
public void submitConsumeRequest(
    final List<MessageExt> msgs,
    final ProcessQueue processQueue,
    final MessageQueue messageQueue,
    final boolean dispatchToConsume) {
    final int consumeBatchSize = this.defaultMQPushConsumer.getConsumeMessageBatchMaxSize(); // Get the maximum size of a single consumption message
    if (msgs.size() <= consumeBatchSize) { // Message size < = the maximum size of a single consumption message. Direct consumption is allowed
        ConsumeRequest consumeRequest = new ConsumeRequest(msgs, processQueue, messageQueue); // According to the message content, processing queue and message queue, it is encapsulated into a consumption request object (actually a thread class, which is submitted to the thread pool for asynchronous processing and consumption of messages)
        try {
            this.consumeExecutor.submit(consumeRequest); // Submit the consumption request to the thread pool for processing
        } catch (RejectedExecutionException e) {
            this.submitConsumeRequestLater(consumeRequest);
        }
    } else { // Otherwise, the message needs to be cut and consumed in batches
        for (int total = 0; total < msgs.size(); ) {
            List<MessageExt> msgThis = new ArrayList<MessageExt>(consumeBatchSize);
            for (int i = 0; i < consumeBatchSize; i++, total++) {
                if (total < msgs.size()) {
                    msgThis.add(msgs.get(total));
                } else {
                    break;
                }
            }

            ConsumeRequest consumeRequest = new ConsumeRequest(msgThis, processQueue, messageQueue);
            
            try {
                this.consumeExecutor.submit(consumeRequest);
            } catch (RejectedExecutionException e) {
                for (; total < msgs.size(); total++) {
                    msgThis.add(msgs.get(total));
                }
                this.submitConsumeRequestLater(consumeRequest);
            }
        }
    }
}
  • Submit consumption thread:

After the consuming thread is submitted to the thread pool, the pool allocates the corresponding resources and time to execute the tasks in the thread. Therefore, go back to the consumer thread class ConsumeRequest itself and see what is the specific content of its thread task run method

public void run() {
    if (this.processQueue.isDropped()) {
        log.info("the message queue not be able to consume, because it's dropped. group={} {}", ConsumeMessageConcurrentlyService.this.consumerGroup, this.messageQueue);
        return;
    }

    MessageListenerConcurrently listener = ConsumeMessageConcurrentlyService.this.messageListener; // Get the listener that consumes concurrent services (the registration listener is explained at the beginning of this article)
    ConsumeConcurrentlyContext context = new ConsumeConcurrentlyContext(messageQueue);
    ConsumeConcurrentlyStatus status = null;
    defaultMQPushConsumerImpl.resetRetryAndNamespace(msgs, defaultMQPushConsumer.getConsumerGroup());

	// If any hook function is registered, the hook function is executed
    ConsumeMessageContext consumeMessageContext = null;
    if (ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.hasHook()) {
        consumeMessageContext = new ConsumeMessageContext();
        consumeMessageContext.setNamespace(defaultMQPushConsumer.getNamespace());
        consumeMessageContext.setConsumerGroup(defaultMQPushConsumer.getConsumerGroup());
        consumeMessageContext.setProps(new HashMap<String, String>());
        consumeMessageContext.setMq(messageQueue);
        consumeMessageContext.setMsgList(msgs);
        consumeMessageContext.setSuccess(false);
        ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.executeHookBefore(consumeMessageContext);
    }

    long beginTimestamp = System.currentTimeMillis();
    boolean hasException = false;
    ConsumeReturnType returnType = ConsumeReturnType.SUCCESS;
    try {
    	// Set the consumption start timestamp for the message
        if (msgs != null && !msgs.isEmpty()) {
            for (MessageExt msg : msgs) {
                MessageAccessor.setConsumeStartTimeStamp(msg, String.valueOf(System.currentTimeMillis()));
            }
        }
        status = listener.consumeMessage(Collections.unmodifiableList(msgs), context); // Call the consumeMessage method of the listener, pass in the message and message context, and execute the custom consumption logic (go back to the consumeMessage method of the concurrent listener registered at the beginning of this chapter)
    } catch (Throwable e) {
        log.warn("consumeMessage exception: {} Group: {} Msgs: {} MQ: {}", RemotingHelper.exceptionSimpleDesc(e), ConsumeMessageConcurrentlyService.this.consumerGroup, msgs, messageQueue);
        hasException = true;
    }
	...
}

At this point, the core source code analysis of the message consumer is over

Tags: Java RocketMQ

Posted on Sat, 25 Sep 2021 05:05:22 -0400 by stepn