1. Preface

In message oriented middleware, consumers generally need to return an ACK to the Broker for a message that has been successfully consumed. Its purpose is to let the Broker know that the message has been successfully consumed and there is no need to deliver it to other consumers for retry. In RocketMQ, the specific implementation of this process is "reporting the consumption location". RocketMQ has no way to return an ACK for a single message, and the Consumer can only report the message offset consumed by the MessageQueue.
​

When A Consumer instance is started, all instances in the same Group will perform A "rebalancing" operation to reassign messagequeues to themselves. The default allocation policy is "average allocation", which means that the same MessageQueue will only be consumed by one Consumer instance, that is, there will only be one Consumer reporting consumption point for the same MessageQueue, and there is no conflict problem. However, the Consumer instance is A multi-threaded concurrent consumption message. Please think about this question: * * thread A and thread B consume the same queue concurrently. The pulled messages are M1 and M2, and the corresponding offsets are 1 and 2. Thread B successfully consumed m2, but thread A has not consumed M1. At this time, how should the Consumer report the consumption location** If you directly report 2, M1 has not been consumed. Once consumption fails, M1 may be lost. How does RocketMQ do it?
​

2. OffsetStore

After reading the source code of RocketMQ, we will find that OffsetStore is the interface used by RocketMQ to manage Consumer consumption sites. Through the interface definition, we can know what functions it has.

public interface OffsetStore {
    /**
     * Load consumption progress
     * 1.Cluster: there is no need to load, and the progress is maintained by the Broker
     * 2.Broadcasts: loading from local files
     */
    void load() throws MQClientException;

    /**
     * Update queue consumption progress
     * @param mq
     * @param offset
     * @param increaseOnly Update to offset or increment on the original basis?
     */
    void updateOffset(final MessageQueue mq, final long offset, final boolean increaseOnly);

    /**
     * Get consumption progress
     * @param mq
     * @param type Get type: memory, disk file
     * @return
     */
    long readOffset(final MessageQueue mq, final ReadOffsetType type);

    /**
     * Persist consumption progress of all queues
     * 1.Cluster: report to Broker
     * 2.Broadcast: writing local files
     * When reporting the consumption progress, write to the memory first, and execute the scheduled task once every 5 seconds.
     * @param mqs
     */
    void persistAll(final Set<MessageQueue> mqs);

    /**
     * Persist consumption progress of a single queue
     * @param mq
     */
    void persist(final MessageQueue mq);

    /**
     * Delete consumption progress of queue
     */
    void removeOffset(MessageQueue mq);

    /**
     * Clone the queue consumption progress under a given Topic
     * @param topic
     * @return
     */
    Map<MessageQueue, Long> cloneOffsetTable(String topic);

    /**
     * Report the consumption progress to the Broker, only for cluster consumption
     * @param mq
     * @param offset
     * @param isOneway
     */
    void updateConsumeOffsetToBroker(MessageQueue mq, long offset, boolean isOneway) throws RemotingException,
        MQBrokerException, InterruptedException, MQClientException;
}

There are three methods of core concern, which will be analyzed in this paper:

method	explain
load	Load consumption sites from disk for broadcast mode
updateOffset	Update queue consumption sites
persistAll	Persistent consumption sites

There are two kinds of RocketMQ message consumption modes: cluster consumption and broadcast consumption. The corresponding implementation classes are remoteberokeroffsetstore and LocalFileOffsetStore respectively. It can also be seen from the name that the consumption progress in the cluster mode is managed by the Broker, and the consumption progress in the broadcast mode is managed by the client itself.

2.1 LocalFileOffsetStore

In broadcast mode, messages need to be consumed by all Consumer instances, so the consumption progress of each instance is different. Therefore, the consumption progress will be managed by the Consumer client. The consumption progress will be persisted to the disk file in the way of Json. The default path is home/.rocketmq_offsets/${ClientId}/${groupName}/offsets.json. When the Consumer starts, it will be loaded from the local disk file. The corresponding method is readLocalOffset().

private OffsetSerializeWrapper readLocalOffset() throws MQClientException {
    String content = null;
    try {
        // Read persistent file
        content = MixAll.file2String(this.storePath);
    } catch (IOException e) {
        log.warn("Load local offset store file exception", e);
    }
    if (null == content || content.length() == 0) {
        // During persistence, the old data will be backed up to the. bak file first
        // If the official file fails to load, it will try to restore through the backup file
        return this.readLocalOffsetBak();
    } else {
        OffsetSerializeWrapper offsetSerializeWrapper = null;
        try {
            offsetSerializeWrapper =
                OffsetSerializeWrapper.fromJson(content, OffsetSerializeWrapper.class);
        } catch (Exception e) {
            log.warn("readLocalOffset Exception, and try to correct", e);
            return this.readLocalOffsetBak();
        }

        return offsetSerializeWrapper;
    }
}

Read the Json file from disk, and then put the consumption progress of the queue into the Map container.

OffsetSerializeWrapper offsetSerializeWrapper = this.readLocalOffset();
if (offsetSerializeWrapper != null 
    && offsetSerializeWrapper.getOffsetTable() != null) {
    offsetTable.putAll(offsetSerializeWrapper.getOffsetTable());
}

When the Consumer successfully consumes the message, it will update the consumption location according to the Offset of the message. In broadcast mode, you only need to modify the value in the Map container.

@Override
public void updateOffset(MessageQueue mq, long offset, boolean increaseOnly) {
    if (mq != null) {
        AtomicLong offsetOld = this.offsetTable.get(mq);
        if (null == offsetOld) {
            // Queue that does not exist, write Offset
            offsetOld = this.offsetTable.putIfAbsent(mq, new AtomicLong(offset));
        }
        if (null != offsetOld) {
            if (increaseOnly) {
                // On the original basis
                MixAll.compareAndIncreaseOnly(offsetOld, offset);
            } else {
                offsetOld.set(offset);
            }
        }
    }
}

updateOffset only modifies the data in memory. The Consumer will start the scheduled task and persist it once every five seconds. In broadcast mode, persistence is to write the data in the Map container to the disk file in the way of Json, and the code will not be pasted.

2.2 RemoteBrokerOffsetStore

In the cluster mode, messages only need to be consumed by one of the Consumer instances. The consumption progress of the same Consumer group is consistent, so the consumption progress will be handed over to the Broker for management.
​

Since the consumption progress is managed by the Broker, the load() method of all remoteberokeroffsetstore implementation classes is empty and does nothing.
​

After the Consumer successfully consumes the message, it will also call the updateOffset method to update the queue consumption location. The code is the same as above. At this time, only the consumption location is written to the Map container in memory and will not be reported to the Broker immediately. Because the consumption of messages is a very frequent action, if every consumption is reported, it will put great pressure on consumers and brokers.
​

If the service hangs without calling the persistAll method after updateOffset, the Broker will think that the consumed messages have not been consumed, resulting in repeated message delivery. However, it doesn't matter. RocketMQ itself allows repeated message delivery, and its service quality is "at least once, multiple times allowed".
​

The scheduled task started by the Consumer will be persisted once every 5 seconds. For the cluster mode, the so-called "persistence" is to report the consumption point of the queue to the Broker. In the persistAll method, the Consumer will traverse the offsetTable and report the consumption sites of MessageQueue by one. The method of reporting sites is updateConsumeOffsetToBroker(). The reporting consumption site needs to interact with the Broker, which involves network requests. By default, it uses "one-way request". It doesn't care whether the reporting is successful or not. Even if it is unsuccessful, it doesn't care. It's a big deal that messages are consumed repeatedly.

/**
 * Report consumption sites to Broker
 * @param mq queue
 * @param offset Consumption site
 * @param isOneway One way request?
 */
@Override
public void updateConsumeOffsetToBroker(MessageQueue mq, long offset, boolean isOneway) throws RemotingException,
    MQBrokerException, InterruptedException, MQClientException {
    // Find Broker address
    FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
    if (null == findBrokerResult) {
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
        findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
    }

    if (findBrokerResult != null) {
        // Build request Header
        UpdateConsumerOffsetRequestHeader requestHeader = new UpdateConsumerOffsetRequestHeader();
        requestHeader.setTopic(mq.getTopic());
        requestHeader.setConsumerGroup(this.groupName);
        requestHeader.setQueueId(mq.getQueueId());
        requestHeader.setCommitOffset(offset);

        // Send request to Broker
        if (isOneway) {
            this.mQClientFactory.getMQClientAPIImpl().updateConsumerOffsetOneway(
                findBrokerResult.getBrokerAddr(), requestHeader, 1000 * 5);
        } else {
            this.mQClientFactory.getMQClientAPIImpl().updateConsumerOffset(
                findBrokerResult.getBrokerAddr(), requestHeader, 1000 * 5);
        }
    } else {
        throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
    }
}

3. Reporting process

After pulling the message, PullMessageService will submit a consumption request to ConsumeMessageService, wake up the message consumption thread to consume the message, and then process the consumption result according to the message consumption status. It's easy to deal with the success of consumption. Just update the consumption location directly. In the case of consumption failure, in the broadcast mode, the consumption failure message will be recorded in the log and then directly discarded; In the cluster mode, the message of consumption failure will be returned to the Broker for re delivery.

The method of processing consumption results is processConsumeResult. The ProcessQueue object caches the messages pulled by the Consumer. It is stored in TreeMap at the bottom. Key is the message Offset and Value is the message. TreeMap uses a red black tree structure, and its keys are ordered from small to large. The message of successful consumption will be deleted from the TreeMap. The consumption location reported by the Consumer each time will always be the smallest key of the TreeMap, that is, the smallest message Offset. This answers the question raised at the beginning. Although thread B consumes M2, M1 has not been consumed. The firstKey in TreeMap is still 1. At this time, consumption site 2 will not be reported. Only M1 is consumed, it will be reported together with 2.

/**
 * Processing consumption results
 * 1.Relevant data statistics
 * 2.Send the failure message back to the Broker according to the ackIndex
 * 3.Failed to send back. Submit asynchronous task and retry consumption after 5s
 * 4.Report consumption sites
 */
public void processConsumeResult(
    final ConsumeConcurrentlyStatus status,
    final ConsumeConcurrentlyContext context,
    final ConsumeRequest consumeRequest) {
    
    // The message before ackIndex represents success and the message after ackIndex represents failure
    int ackIndex = context.getAckIndex();
    if (consumeRequest.getMsgs().isEmpty())
        return;

    // statistical data
    switch (status) {
        case CONSUME_SUCCESS:
            if (ackIndex >= consumeRequest.getMsgs().size()) {
                ackIndex = consumeRequest.getMsgs().size() - 1;
            }
            int ok = ackIndex + 1;
            int failed = consumeRequest.getMsgs().size() - ok;
            this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), ok);
            this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), failed);
            break;
        case RECONSUME_LATER:
            // Consumption fails. The ackIndex is reset and the whole batch of messages need to be reprocessed
            ackIndex = -1;
            this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(),
                consumeRequest.getMsgs().size());
            break;
        default:
            break;
    }

    switch (this.defaultMQPushConsumer.getMessageModel()) {
        case BROADCASTING:// In broadcast mode, throw it away directly
            for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
                MessageExt msg = consumeRequest.getMsgs().get(i);
                log.warn("BROADCASTING, the message consume failed, drop it, {}", msg.toString());
            }
            break;
        case CLUSTERING:// In cluster mode, it is returned to the Broker
            // Message return failure list
            List<MessageExt> msgBackFailed = new ArrayList<MessageExt>(consumeRequest.getMsgs().size());
            // The message before ackIndex indicates that the consumption is successful, and the message after ackIndex fails needs to be sent back to the Broker
            for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
                MessageExt msg = consumeRequest.getMsgs().get(i);
                // Return to Broker
                boolean result = this.sendMessageBack(msg, context);
                if (!result) {
                    // Failed to send back. It is temporarily stored in msgBackFailed and will be consumed again later
                    msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
                    msgBackFailed.add(msg);
                }
            }

            if (!msgBackFailed.isEmpty()) {
                // The failed message is deleted from msgs, and the location of the failed message will not be reported
                consumeRequest.getMsgs().removeAll(msgBackFailed);
                // Submit a consumption request and retry after 5s
                this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
            }
            break;
        default:
            break;
    }

    long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
    if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
        // Report the consumption location, excluding the message of failed return Broker
        this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
    }
}

If the consumption fails and the message is returned to the Broker, it will not be deleted from the ProcessQueue, and the consumption location will not be reported.

4. Summary

RocketMQ manages the consumption progress through the OffsetStore interface. In the broadcast mode, it is managed by the Consumer client, and in the cluster mode, it is managed by the Broker. After the Consumer consumes the message, it will only update the consumption location in memory, and the timing task is responsible for persistence once every 5 seconds. The persistence of broadcast mode is to write data to disk files, and the persistence of cluster mode is to report consumption sites to brokers.
​

The bottom layer of the Consumer uses TreeMap to store the pulled messages. The Key is the message Offset and the Value is the message itself. Therefore, it is arranged according to the Offset of the message. When reporting the consumption point, the first Key, that is, the minimum Offset, is always reported. During concurrent consumption, it doesn't matter if messages with large Offset are consumed first, and Offset will not be reported directly. This brings another problem. There is the possibility of missing reports, which will lead to repeated consumption of messages. Therefore, for more sensitive data, the Consumer needs to be idempotent in consumption.