RocketMQ source code analysis 12 (sequential message flow)

RocketMQ version 4.6.0 records the process of viewing the source code

Sequential consumption is a little more complicated, mainly because sequential consumption has an impact on rebalancing and message pulling.

Counterbalance

Let's first look at the rebalancing. The main difference is that after calculating the consumption queue allocated to each consumer according to the load balancing policy, judge whether the queue subscribed by the consumer is in the update process queuetableinrebalance method. Before creating a pull task for the newly allocated consumption queue, you need to obtain the lock first.
RebalanceImpl

private boolean updateProcessQueueTableInRebalance(final String topic, final Set<MessageQueue> mqSet,
    final boolean isOrder) {
    boolean changed = false;

    // The message queue to which the current consumer has been assigned
    Iterator<Entry<MessageQueue, ProcessQueue>> it = this.processQueueTable.entrySet().iterator();
    while (it.hasNext()) {
        Entry<MessageQueue, ProcessQueue> next = it.next();
        MessageQueue mq = next.getKey();
        ProcessQueue pq = next.getValue();

        // Queues that do not belong to this topic are skipped
        if (mq.getTopic().equals(topic)) {
            // If the reassigned queue does not contain the old queue mq, it means that after rebalancing, the queue is assigned to other consumers,
            // Therefore, it is necessary to pause the current consumer's consumption of the queue, set the ProcessQueue to dropped=true, and remove it from the local cache
            if (!mqSet.contains(mq)) {
                pq.setDropped(true);
                if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                    // Remove the queue from the subscription cache
                    it.remove();
                    changed = true;
                    log.info("doRebalance, {}, remove unnecessary mq, {}", consumerGroup, mq);
                }
            } else if (pq.isPullExpired()) {
                switch (this.consumeType()) {
                    case CONSUME_ACTIVELY:
                        break;
                    case CONSUME_PASSIVELY:
                        pq.setDropped(true);
                        if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                            it.remove();
                            changed = true;
                            log.error("[BUG]doRebalance, {}, remove unnecessary mq, {}, because pull is pause, so try to fixed it",
                                consumerGroup, mq);
                        }
                        break;
                    default:
                        break;
                }
            }
        }
    }

    List<PullRequest> pullRequestList = new ArrayList<PullRequest>();
    for (MessageQueue mq : mqSet) {
        // If there is no new queue in the old cache subscription queue, it means that the consumer is assigned to the new queue after rebalancing
        // You need to create a PullRequest pull message task to pull messages
        if (!this.processQueueTable.containsKey(mq)) {
            // For sequential consumption, you need to lock the queue in the broker
            // If the lock is obtained, a pull task is created to pull the message,
            // If the lock is not obtained, skip and wait for the next rebalancing to obtain the lock again (the queue may not be consumed when the lock is not obtained)
            if (isOrder && !this.lock(mq)) {
                log.warn("doRebalance, {}, add a new mq failed, {}, because lock failed", consumerGroup, mq);
                continue;
            }

            // Delete the consumption progress of the message queue from memory
            this.removeDirtyOffset(mq);
            // Create a new ProcessQueue corresponding to the message queue
            ProcessQueue pq = new ProcessQueue();
            // Calculate where to pull from
            long nextOffset = this.computePullFromWhere(mq);
            if (nextOffset >= 0) {
                ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
                if (pre != null) {
                    log.info("doRebalance, {}, mq already exists, {}", consumerGroup, mq);
                } else {
                    log.info("doRebalance, {}, add a new mq, {}", consumerGroup, mq);
                    PullRequest pullRequest = new PullRequest();
                    pullRequest.setConsumerGroup(consumerGroup);
                    pullRequest.setNextOffset(nextOffset);
                    pullRequest.setMessageQueue(mq);
                    pullRequest.setProcessQueue(pq);
                    pullRequestList.add(pullRequest);
                    changed = true;
                }
            } else {
                log.warn("doRebalance, {}, add new mq failed, {}", consumerGroup, mq);
            }
        }
    }

    // To execute the pull task immediately is to put the pull request task into the pull request queue
    this.dispatchPullRequest(pullRequestList);

    return changed;
}

Remove queues that no longer belong to

1. If the reassigned queue does not contain the old queue mq, it means that after rebalancing, the queue is assigned to other consumers, so it is necessary to suspend the current consumer's consumption of the queue, set the ProcessQueue to dropped=true, remove it from the local cache, and release the lock in the broker at the same time.
RebalancePushImpl

@Override
public boolean removeUnnecessaryMessageQueue(MessageQueue mq, ProcessQueue pq) {
    // Persist the consumption progress of the MessageQueue
    this.defaultMQPushConsumerImpl.getOffsetStore().persist(mq);
    this.defaultMQPushConsumerImpl.getOffsetStore().removeOffset(mq);
    // If it is sequential consumption and cluster mode, you need to release the lock
    if (this.defaultMQPushConsumerImpl.isConsumeOrderly()
        && MessageModel.CLUSTERING.equals(this.defaultMQPushConsumerImpl.messageModel())) {
        try {
            // Avoid conflicts with message queue consumption. If the lock acquisition fails, it indicates that the message in the processQueue is executing custom consumption logic,
            // The removal of the message queue fails. Wait for the next reallocation of the consumption queue before removing it.
            // If you remove the message queue without obtaining the lock, another Consumer and the current Consumer may consume the message queue at the same time, resulting in messages that cannot be consumed sequentially
            if (pq.getLockConsume().tryLock(1000, TimeUnit.MILLISECONDS)) {
                try {
                    // Request the broker to release the lock and return true
                    return this.unlockDelay(mq, pq);
                } finally {
                    pq.getLockConsume().unlock();
                }
            } else {
                log.warn("[WRONG]mq is consuming, so can not unlock it, {}. maybe hanged for a while, {}",
                    mq,
                    pq.getTryUnlockTimes());

                pq.incTryUnlockTimes();
            }
        } catch (Exception e) {
            log.error("removeUnnecessaryMessageQueue Exception", e);
        }

        return false;
    }
    return true;
}

This is the first difference from concurrent consumption. Before removal, the lock occupied by the broker must be released (the release logic is described below). Before release, the consumption lock of processQueue must be obtained to avoid conflict with message queue consumption. If the acquisition of the consumption lock fails, it indicates that the message in the processQueue is executing the custom consumption logic (because the consumption lock must be obtained before consumption), the removal of the message queue fails. Wait for the next reassignment of the consumption queue. If you remove the message queue without obtaining the consumption lock, another Consumer and the current Consumer may consume the message queue at the same time, so that the messages of the message queue cannot be consumed sequentially.

Assign to new queue

2. If there is no new queue in the old cache subscription queue, it means that after rebalancing, the consumer is assigned to a new queue. You need to create a PullRequest pull message task to pull messages. This is also different from concurrent consumption. If it is sequential consumption, you need to lock the queue in the broker. If it is successfully locked, create a pull task to pull messages. Otherwise, skip and wait for the next rebalancing to obtain the lock again.

/**
 * Apply to the broker to lock the message queue
 */
public boolean lock(final MessageQueue mq) {
    // Get master broker address
    FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(), MixAll.MASTER_ID, true);
    if (findBrokerResult != null) {
        // Request body for lock queue
        LockBatchRequestBody requestBody = new LockBatchRequestBody();
        requestBody.setConsumerGroup(this.consumerGroup);
        requestBody.setClientId(this.mQClientFactory.getClientId());
        // Indicates the queue to lock
        requestBody.getMqSet().add(mq);

        try {
            // Send a lock queue request to the broker synchronously. The request code is LOCK_BATCH_MQ, return the successfully locked queue
            // In fact, this is to return whether the mq has successfully obtained the lock
            Set<MessageQueue> lockedMq =
                this.mQClientFactory.getMQClientAPIImpl().lockBatchMQ(findBrokerResult.getBrokerAddr(), requestBody, 1000);
            for (MessageQueue mmqq : lockedMq) {
                ProcessQueue processQueue = this.processQueueTable.get(mmqq);
                // If the queue that currently obtains the lock itself is the queue to which it has been allocated, mark the lock and update the lock time
                // The message queue is locked successfully. If there is no local message processing queue, the locking success will be set in the lockAll() method

                // The processQueue corresponding to the newly allocated message queue is empty. locked=true will not be set here, so the task cannot be pulled at this time,
                // When ConsumeMessageOrderlyService is started, it will execute the lockAll method every 20s to lock the allocated queue (locked=true and update the lock time),
                // Only then can the first pull task of the queue be performed
                if (processQueue != null) {
                    processQueue.setLocked(true);
                    processQueue.setLastLockTimestamp(System.currentTimeMillis());
                }
            }

            // Returns whether the lock of the queue was successfully obtained
            boolean lockOK = lockedMq.contains(mq);
            log.info("the message queue lock {}, {} {}",
                lockOK ? "OK" : "Failed",
                this.consumerGroup,
                mq);
            return lockOK;
        } catch (Exception e) {
            log.error("lockBatchMQ exception, " + mq, e);
        }
    }

    return false;
}

On the Broker side, the request to lock the queue is processed by the admin Broker processor

/**
 * Batch lock queue request
 */
private RemotingCommand lockBatchMQ(ChannelHandlerContext ctx,
    RemotingCommand request) throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(null);
    LockBatchRequestBody requestBody = LockBatchRequestBody.decode(request.getBody(), LockBatchRequestBody.class);

    // Lock the message queue through the rebalance lock manager and return the successfully locked consumption queue
    // Locking failure means that the message queue is locked by other consumers and has not expired
    Set<MessageQueue> lockOKMQSet = this.brokerController.getRebalanceLockManager().tryLockBatch(
        requestBody.getConsumerGroup(),
        requestBody.getMqSet(),
        requestBody.getClientId());

    LockBatchResponseBody responseBody = new LockBatchResponseBody();
    // Send back the queue response with successful locking
    responseBody.setLockOKMQSet(lockOKMQSet);

    response.setBody(responseBody.encode());
    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);
    return response;
}

Unlock the message queue through the rebalance lock manager rebalance lock manager

public class RebalanceLockManager {

    /**
     * The lock expiration time is 30s by default. If it is not configured, it is 60s by default. The Consumer needs to constantly refresh the expiration time of the lock. By default, it is configured to refresh every 20s
     */
    private final static long REBALANCE_LOCK_MAX_LIVE_TIME = Long.parseLong(System.getProperty(
        "rocketmq.broker.rebalance.lockMaxLiveTime", "60000"));
    private final Lock lock = new ReentrantLock();
    /**
     * Save the locking status of the consumption queue of each consumption group. The consumption group name is the key instead of topic, because each topic may be subscribed by multiple consumption groups,
     * Each consumption group does not affect each other. Each consumption group can lock the same consumption queue at the same time, so it is saved in the unit of consumption group
     */
    private final ConcurrentMap<String/* group */, ConcurrentHashMap<MessageQueue, LockEntry>> mqLockTable =
        new ConcurrentHashMap<String, ConcurrentHashMap<MessageQueue, LockEntry>>(1024);
    
    /**
     * Attempt to lock the message queue
     *
     * @param mqs Indicates the queue in which the consumer attempted to apply for a lock
     */
    public Set<MessageQueue> tryLockBatch(final String group, final Set<MessageQueue> mqs,
        final String clientId) {
        // The queues that have been locked and unlocked by the consumer corresponding to clientId
        Set<MessageQueue> lockedMqs = new HashSet<MessageQueue>(mqs.size());
        Set<MessageQueue> notLockedMqs = new HashSet<MessageQueue>(mqs.size());

        for (MessageQueue mq : mqs) {
            // Judge whether the consumption queue has been locked by the consumer corresponding to clientId
            if (this.isLocked(group, mq, clientId)) {
                lockedMqs.add(mq);
            } else {
                notLockedMqs.add(mq);
            }
        }

        if (!notLockedMqs.isEmpty()) {
            try {
                this.lock.lockInterruptibly();
                try {
                    // The message queue locking status of the consumer group indicates which consumer in the consumer group has locked the message queue
                    ConcurrentHashMap<MessageQueue, LockEntry> groupValue = this.mqLockTable.get(group);
                    if (null == groupValue) {
                        groupValue = new ConcurrentHashMap<>(32);
                        this.mqLockTable.put(group, groupValue);
                    }

                    // For queues that are not locked by clientId, start trying to lock them
                    for (MessageQueue mq : notLockedMqs) {
                        LockEntry lockEntry = groupValue.get(mq);
                        // If it is empty, the queue has not been locked and can be directly locked by me (clientId)
                        if (null == lockEntry) {
                            lockEntry = new LockEntry();
                            lockEntry.setClientId(clientId);
                            groupValue.put(mq, lockEntry);
                            log.info(
                                "tryLockBatch, message queue not locked, I got it. Group: {} NewClientId: {} {}",
                                group,
                                clientId,
                                mq);
                        }
                        // If I lock it, update the lock time and add it to the lock queue
                        if (lockEntry.isLocked(clientId)) {
                            lockEntry.setLastUpdateTimestamp(System.currentTimeMillis());
                            lockedMqs.add(mq);
                            continue;
                        }
                        // This means that the queue is locked by other consumers
                        String oldClientId = lockEntry.getClientId();
                        // If it expires, I'll lock it directly
                        if (lockEntry.isExpired()) {
                            lockEntry.setClientId(clientId);
                            lockEntry.setLastUpdateTimestamp(System.currentTimeMillis());
                            log.warn(
                                "tryLockBatch, message queue lock expired, I got it. Group: {} OldClientId: {} NewClientId: {} {}",
                                group,
                                oldClientId,
                                clientId,
                                mq);
                            lockedMqs.add(mq);
                            continue;
                        }

                        log.warn(
                            "tryLockBatch, message queue locked by other client. Group: {} OtherClientId: {} NewClientId: {} {}",
                            group,
                            oldClientId,
                            clientId,
                            mq);
                    }
                } finally {
                    this.lock.unlock();
                }
            } catch (InterruptedException e) {
                log.error("putMessage exception", e);
            }
        }

        // Returns the locked queue
        return lockedMqs;
    }
}

Step 1: first, divide the queue to apply for lock into two sets according to whether it has been locked by the consumer corresponding to clientId.
Judge whether it is locked by the clientId:

/**
 * Determines whether the specified message queue has been locked by the clientId consumer client
 */
private boolean isLocked(final String group, final MessageQueue mq, final String clientId) {
    // The message queue locking status of the consumer group indicates which consumer in the consumer group has locked the message queue
    ConcurrentHashMap<MessageQueue, LockEntry> groupValue = this.mqLockTable.get(group);
    if (groupValue != null) {
        LockEntry lockEntry = groupValue.get(mq);
        if (lockEntry != null) {
            // If the queue was previously locked by the consumer client and has not expired, update the lock time, which is equivalent to re counting the expiration time
            boolean locked = lockEntry.isLocked(clientId);
            if (locked) {
                lockEntry.setLastUpdateTimestamp(System.currentTimeMillis());
            }

            return locked;
        }
    }
    // The queue is not locked under the consumption group
    return false;
}

Step 2: obtain the locking status of the message queue under the consumption group.
Step 3: the iteration starts to try to lock the queue that is not locked by the consumer (clientId).
Step 4: finally, the queue successfully locked is returned.
By the way, here is the logic of releasing the lock:

/**
 * Release the lock of clientId on mqs these queues
 */
public void unlockBatch(final String group, final Set<MessageQueue> mqs, final String clientId) {
    try {
        this.lock.lockInterruptibly();
        try {
            ConcurrentHashMap<MessageQueue, LockEntry> groupValue = this.mqLockTable.get(group);
            if (null != groupValue) {
                for (MessageQueue mq : mqs) {
                    LockEntry lockEntry = groupValue.get(mq);
                    if (null != lockEntry) {
                        if (lockEntry.getClientId().equals(clientId)) {
                            groupValue.remove(mq);
                            log.info("unlockBatch, Group: {} {} {}",
                                group,
                                mq,
                                clientId);
                        } else {
                            log.warn("unlockBatch, but mq locked by other client: {}, Group: {} {} {}",
                                lockEntry.getClientId(),
                                group,
                                mq,
                                clientId);
                        }
                    } else {
                        log.warn("unlockBatch, but mq not locked, Group: {} {} {}",
                            group,
                            mq,
                            clientId);
                    }
                }
            } else {
                log.warn("unlockBatch, group not exist, Group: {} {}",
                    group,
                    clientId);
            }
        } finally {
            this.lock.unlock();
        }
    } catch (InterruptedException e) {
        log.error("putMessage exception", e);
    }
}

It is easier to release the lock. Cycle the queue to release the lock to determine whether it is locked by the consumer. If so, remove the lock record.

Then go back to the lock method above. After locking the queue from the broker, set the corresponding processing queue processQueue to the locked state, so that you can pull messages.

Set<MessageQueue> lockedMq =
    this.mQClientFactory.getMQClientAPIImpl().lockBatchMQ(findBrokerResult.getBrokerAddr(), requestBody, 1000);
for (MessageQueue mmqq : lockedMq) {
    ProcessQueue processQueue = this.processQueueTable.get(mmqq);
    // If the queue that currently obtains the lock itself is the queue to which it has been allocated, mark the lock and update the lock time
    // The message queue is locked successfully. If there is no local message processing queue, the locking success will be set in the lockAll() method

    // The processQueue corresponding to the newly allocated message queue is empty. locked=true will not be set here, so the task cannot be pulled at this time,
    // When ConsumeMessageOrderlyService is started, it will execute the lockAll method every 20s to lock the allocated queue (locked=true and update the lock time),
    // Only then can the first pull task of the queue be performed
    if (processQueue != null) {
        processQueue.setLocked(true);
        processQueue.setLastLockTimestamp(System.currentTimeMillis());
    }
}

However, if it is a newly allocated message queue, there is no processQueue, so the processQueue will not be locked here. Because the locked attribute of processQueue is false by default, the processQueue created for the newly allocated message queue is not locked and cannot pull messages. However, when starting the sequential consumption service, a background timing task will be started to set the locking status and update the locking time for the message queue assigned by the consumer.
ConsumeMessageOrderlyService

/**
 * Start sequential consumption service
 */
public void start() {
    if (MessageModel.CLUSTERING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel())) {
        // Lock the queue every 20s
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                ConsumeMessageOrderlyService.this.lockMQPeriodically();
            }
        }, 1000 * 1, ProcessQueue.REBALANCE_LOCK_INTERVAL, TimeUnit.MILLISECONDS);
    }
}

public synchronized void lockMQPeriodically() {
    if (!this.stopped) {
        this.defaultMQPushConsumerImpl.getRebalanceImpl().lockAll();
    }
}

Use the RebalanceImpl component to handle

/**
 * Lock all message queues assigned to
 */
public void lockAll() {
    // Build a map according to the processQueueTable. The key is the broker name and the value is the message queue assigned by the consumer on the broker
    HashMap<String, Set<MessageQueue>> brokerMqs = this.buildProcessQueueTableByBrokerName();

    Iterator<Entry<String, Set<MessageQueue>>> it = brokerMqs.entrySet().iterator();
    while (it.hasNext()) {
        Entry<String, Set<MessageQueue>> entry = it.next();
        final String brokerName = entry.getKey();
        final Set<MessageQueue> mqs = entry.getValue();

        if (mqs.isEmpty())
            continue;

        // Find the master broker corresponding to the broker name and send the request for batch locking message queue to the broker
        FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(brokerName, MixAll.MASTER_ID, true);
        if (findBrokerResult != null) {
            LockBatchRequestBody requestBody = new LockBatchRequestBody();
            requestBody.setConsumerGroup(this.consumerGroup);
            requestBody.setClientId(this.mQClientFactory.getClientId());
            requestBody.setMqSet(mqs);

            try {
                // Returns the message queue that was successfully locked
                Set<MessageQueue> lockOKMQSet =
                    this.mQClientFactory.getMQClientAPIImpl().lockBatchMQ(findBrokerResult.getBrokerAddr(), requestBody, 1000);

                // Set the locked corresponding processing queue locked to true and update the locking time
                for (MessageQueue mq : lockOKMQSet) {
                    ProcessQueue processQueue = this.processQueueTable.get(mq);
                    if (processQueue != null) {
                        if (!processQueue.isLocked()) {
                            log.info("the message queue locked OK, Group: {} {}", this.consumerGroup, mq);
                        }

                        processQueue.setLocked(true);
                        processQueue.setLastLockTimestamp(System.currentTimeMillis());
                    }
                }
                // If the lock is not successful, set the locked to false
                for (MessageQueue mq : mqs) {
                    if (!lockOKMQSet.contains(mq)) {
                        ProcessQueue processQueue = this.processQueueTable.get(mq);
                        if (processQueue != null) {
                            processQueue.setLocked(false);
                            log.warn("the message queue locked Failed, Group: {} {}", this.consumerGroup, mq);
                        }
                    }
                }
            } catch (Exception e) {
                log.error("lockBatchMQ exception, " + mqs, e);
            }
        }
    }
}

Pull message

When pulling messages, you will first judge whether the processQueue is locked. If it is not locked, it will be delayed for 3s, and then put the pullRequest back into the pull task. You can pull messages only when it is locked.
DefaultMQPushConsumerImpl

/**
 * Pull messages according to the pull task
 */
public void pullMessage(final PullRequest pullRequest) {
    final ProcessQueue processQueue = pullRequest.getProcessQueue();
    if (processQueue.isDropped()) {
        log.info("the pull request[{}] is dropped.", pullRequest.toString());
        return;
    }

    // Omit...

    // Concurrent consumption
    if (!this.consumeOrderly) {
        // Current limiting
        if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
            this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
            if ((queueMaxSpanFlowControlTimes++ % 1000) == 0) {
                log.warn(
                    "the queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, pullRequest={}, flowControlTimes={}",
                    processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(),
                    pullRequest, queueMaxSpanFlowControlTimes);
            }
            return;
        }
    } else {
        // Sequential consumption
        if (processQueue.isLocked()) {
            // If the consumption queue pulls messages for the first time, the pull offset is calculated first
            if (!pullRequest.isLockedFirst()) {
                final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue());
                boolean brokerBusy = offset < pullRequest.getNextOffset();
                log.info("the first time to pull message, so fix offset from broker. pullRequest: {} NewOffset: {} brokerBusy: {}",
                    pullRequest, offset, brokerBusy);
                if (brokerBusy) {
                    log.info("[NOTIFYME]the first time to pull message, but pull request offset larger than broker consume offset. pullRequest: {} NewOffset: {}",
                        pullRequest, offset);
                }

                pullRequest.setLockedFirst(true);
                pullRequest.setNextOffset(offset);
            }
        } else {
            // If the queue is not locked, put the pullRequest back into the pull task after a delay of 3s
            // Messages can be pulled only when they are locked
            this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
            log.info("pull message later because not locked in broker, {}", pullRequest);
            return;
        }
    }

    // To send a pull request...
}

Consumption news

After the message is pulled, it will be handed over to the consuming thread for consumption

/**
 * Sequential consumption thread
 */
class ConsumeRequest implements Runnable {
    private final ProcessQueue processQueue;
    private final MessageQueue messageQueue;

    public ConsumeRequest(ProcessQueue processQueue, MessageQueue messageQueue) {
        this.processQueue = processQueue;
        this.messageQueue = messageQueue;
    }

    public ProcessQueue getProcessQueue() {
        return processQueue;
    }

    public MessageQueue getMessageQueue() {
        return messageQueue;
    }

    @Override
    public void run() {
        if (this.processQueue.isDropped()) {
            log.warn("run, the message queue not be able to consume, because it's dropped. {}", this.messageQueue);
            return;
        }

        // Get message queue lock object
        final Object objLock = messageQueueLock.fetchLockObject(this.messageQueue);
        // Every time a message is pulled, a consumption request will be submitted, so multiple threads may be stuck here, and only one thread will execute down forever,
        // Since several processqueues are fixed, messages are fetched from the fixed processQueue no matter which thread executes down,
        // Ensure that the same message queue will be consumed by only one thread at the same time. Messages are taken in order and will not be out of order.
        // Different message queues can be executed in parallel
        synchronized (objLock) {
            // (broadcast mode) or (cluster mode & & broker message queue distributed lock is valid)
            if (MessageModel.BROADCASTING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel())
                || (this.processQueue.isLocked() && !this.processQueue.isLockExpired())) {
                final long beginTime = System.currentTimeMillis();
                for (boolean continueConsume = true; continueConsume; ) {
                    if (this.processQueue.isDropped()) {
                        log.warn("the message queue not be able to consume, because it's dropped. {}", this.messageQueue);
                        break;
                    }

                    // If the message queue has not been locked, send a request to the broker to lock the message queue, and then resubmit the consumption request
                    if (MessageModel.CLUSTERING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel())
                        && !this.processQueue.isLocked()) {
                        log.warn("the message queue not locked, so consume later, {}", this.messageQueue);
                        ConsumeMessageOrderlyService.this.tryLockLaterAndReconsume(this.messageQueue, this.processQueue, 10);
                        break;
                    }

                    // If the locked message queue expires, first send a request to the broker to lock the message queue (update the locking time), and then resubmit the consumption request
                    if (MessageModel.CLUSTERING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel())
                        && this.processQueue.isLockExpired()) {
                        log.warn("the message queue lock expired, so consume later, {}", this.messageQueue);
                        ConsumeMessageOrderlyService.this.tryLockLaterAndReconsume(this.messageQueue, this.processQueue, 10);
                        break;
                    }

                    // The thread has been executing for more than 60s. After resubmitting the consumption request, the thread ends and is handed over to other threads for execution
                    long interval = System.currentTimeMillis() - beginTime;
                    if (interval > MAX_TIME_CONSUME_CONTINUOUSLY) {
                        ConsumeMessageOrderlyService.this.submitConsumeRequestLater(processQueue, messageQueue, 10);
                        break;
                    }

                    // The default is 1
                    final int consumeBatchSize =
                        ConsumeMessageOrderlyService.this.defaultMQPushConsumer.getConsumeMessageBatchMaxSize();

                    // It is different from concurrent consumption to get messages. When a concurrent consumption request is created, it has taken out and set which messages to consume from the processQueue,
                    // Here, sequential consumption is to retrieve messages from processQueue in order in the request
                    // Take consumebacksize messages out of msgTreeMap in order and put them into consumingMsgOrderlyTreeMap
                    List<MessageExt> msgs = this.processQueue.takeMessags(consumeBatchSize);
                    defaultMQPushConsumerImpl.resetRetryAndNamespace(msgs, defaultMQPushConsumer.getConsumerGroup());
                    if (!msgs.isEmpty()) {
                        final ConsumeOrderlyContext context = new ConsumeOrderlyContext(this.messageQueue);

                        ConsumeOrderlyStatus status = null;

                        ConsumeMessageContext consumeMessageContext = null;
                        if (ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.hasHook()) {
                            consumeMessageContext = new ConsumeMessageContext();
                            consumeMessageContext
                                .setConsumerGroup(ConsumeMessageOrderlyService.this.defaultMQPushConsumer.getConsumerGroup());
                            consumeMessageContext.setNamespace(defaultMQPushConsumer.getNamespace());
                            consumeMessageContext.setMq(messageQueue);
                            consumeMessageContext.setMsgList(msgs);
                            consumeMessageContext.setSuccess(false);
                            // init the consume context type
                            consumeMessageContext.setProps(new HashMap<String, String>());
                            ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.executeHookBefore(consumeMessageContext);
                        }

                        long beginTimestamp = System.currentTimeMillis();
                        ConsumeReturnType returnType = ConsumeReturnType.SUCCESS;
                        boolean hasException = false;
                        try {
                            // Obtain the consumption lock before consumption
                            this.processQueue.getLockConsume().lock();
                            if (this.processQueue.isDropped()) {
                                log.warn("consumeMessage, the message queue not be able to consume, because it's dropped. {}",
                                    this.messageQueue);
                                break;
                            }

                            // Execute your own consumption logic
                            status = messageListener.consumeMessage(Collections.unmodifiableList(msgs), context);
                        } catch (Throwable e) {
                            log.warn("consumeMessage exception: {} Group: {} Msgs: {} MQ: {}",
                                RemotingHelper.exceptionSimpleDesc(e),
                                ConsumeMessageOrderlyService.this.consumerGroup,
                                msgs,
                                messageQueue);
                            hasException = true;
                        } finally {
                            // Release consumption lock
                            this.processQueue.getLockConsume().unlock();
                        }

                        // Omit state transition

                        // Processing consumption results
                        continueConsume = ConsumeMessageOrderlyService.this.processConsumeResult(msgs, status, context, this);
                    } else {
                        continueConsume = false;
                    }
                }
            } else {
                if (this.processQueue.isDropped()) {
                    log.warn("the message queue not be able to consume, because it's dropped. {}", this.messageQueue);
                    return;
                }

                // If the processQueue is not locked or the lock has expired, the broker will lock the current queue,
                // Whether the lock is successful or not, the consumption request will be resubmitted
                ConsumeMessageOrderlyService.this.tryLockLaterAndReconsume(this.messageQueue, this.processQueue, 100);
            }
        }
    }

}

Step 1: first obtain the message queue lock object to ensure that only one thread consumes the queue at the same time, so that it can be ordered.
Step 2: (broadcast mode) or (cluster mode & & broker message queue distributed lock is valid), otherwise it will send a request to the broker to lock the current queue, and then resubmit the consumption request.
Step 3: if it is a cluster mode, go back to the second step.
Step 4: if the thread has been executed for more than 60s, the thread ends after resubmitting the consumption request and is handed over to other threads for execution.
Step 5: take out messages from processQueue in sequence, which is also different from concurrent consumption. Concurrent consumption has taken out messages from processQueue when submitting consumption request, and can be consumed directly here.
Step 6: obtain the consumption lock in processQueue before consumption.
Step 7: call the consumption listener to consume.
Step 8: release the consumption lock.
Step 9: processing consumption results is also different from concurrent consumption

/**
 * Processing sequence consumption results
 */
public boolean processConsumeResult(
    final List<MessageExt> msgs,
    final ConsumeOrderlyStatus status,
    final ConsumeOrderlyContext context,
    final ConsumeRequest consumeRequest
) {
    boolean continueConsume = true;
    long commitOffset = -1L;
    if (context.isAutoCommit()) {
        switch (status) {
            case COMMIT:
            case ROLLBACK:
                log.warn("the message queue consume result is illegal, we think you want to ack these message {}",
                    consumeRequest.getMessageQueue());
            case SUCCESS:
                // Clear the consumingmsgordlytreemap in ProcessQueue and return the offset of the last message
                commitOffset = consumeRequest.getProcessQueue().commit();
                this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size());
                break;
            // Consumption fails. Put the message back into processQueue, suspend the consumption queue for a while, and continue consumption later
            case SUSPEND_CURRENT_QUEUE_A_MOMENT:
                this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size());
                // Check the number of retries. If the maximum number of retries has been reached, forward the message to the dead letter queue,
                // If the maximum number of times has not been reached, suspend the current queue for a while
                if (checkReconsumeTimes(msgs)) {
                    // Remove these messages from consumingMsgOrderlyTreeMap
                    // Put the messages back to the original location in msgTreeMap, so that the messages taken out are still these messages, so as to realize the retry message
                    consumeRequest.getProcessQueue().makeMessageToCosumeAgain(msgs);
                    // Resubmit the consumption request after 10ms
                    this.submitConsumeRequestLater(
                        consumeRequest.getProcessQueue(),
                        consumeRequest.getMessageQueue(),
                        context.getSuspendCurrentQueueTimeMillis());
                    // When a message to retry is encountered, the thread will not continue to execute
                    continueConsume = false;
                } else {
                    // The message is forwarded to the dead letter queue. At this time, when it is processed successfully, the message queue does not need to hang up and continues to consume the following messages
                    commitOffset = consumeRequest.getProcessQueue().commit();
                }
                break;
            default:
                break;
        }
    } else {
        switch (status) {
            case SUCCESS:
                this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size());
                break;
            case COMMIT:
                commitOffset = consumeRequest.getProcessQueue().commit();
                break;
            case ROLLBACK:
                consumeRequest.getProcessQueue().rollback();
                this.submitConsumeRequestLater(
                    consumeRequest.getProcessQueue(),
                    consumeRequest.getMessageQueue(),
                    context.getSuspendCurrentQueueTimeMillis());
                continueConsume = false;
                break;
            case SUSPEND_CURRENT_QUEUE_A_MOMENT:
                this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size());
                if (checkReconsumeTimes(msgs)) {
                    consumeRequest.getProcessQueue().makeMessageToCosumeAgain(msgs);
                    this.submitConsumeRequestLater(
                        consumeRequest.getProcessQueue(),
                        consumeRequest.getMessageQueue(),
                        context.getSuspendCurrentQueueTimeMillis());
                    continueConsume = false;
                }
                break;
            default:
                break;
        }
    }

    // The message processing queue is not dropped, and the effective consumption progress is submitted
    if (commitOffset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
        this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), commitOffset, false);
    }

    return continueConsume;
}

Here, in order to ensure the order, if the consumption fails, you need to put the message back to the original position in msgTreeMap in processQueue, suspend the consumption queue for a while, and continue the consumption later. In this way, the extracted messages are still these messages, so as to realize the retry message.

reference material

"Confucian ape technology nest - taking you to become a master of message middleware from 0"

Tags: RocketMQ

Posted on Tue, 28 Sep 2021 03:16:21 -0400 by stevehaysom