MQ repeated consumption means that multiple instances of the same application receive the same message, or the same instance receives the same message multiple times. If the consumer logic does not do idempotent processing, repeated consumption will be caused.
Message repetition is essentially a problem in MQ design least Once or exactly As for the problem of once, consumers certainly want to be exactly once, but MQ should ensure the reliability of message delivery. Messages that are not ack will be delivered repeatedly. Therefore, the consumer should ensure the idempotency of consumption. For example, after receiving the message, the consumer obtains the message ID from the message and writes it to Redis or database. When the message is received again, it will not be processed. In the scenario of repeated message delivery, in addition to retry, a large part comes from the load balancing stage. The messages pulled by the previous consumer instance listening to the Queue are not all ACK, and the new consumer instance listens to the Queue and pulls the messages again.
In order to solve the problem of repeated listening or missing listening in the load balancing stage, Weizhong bank adds a transition state in the change process of load balancing results. In the transition state, the Consumer will continue to retain the results of the previous load balancing until all the messages pulled by the original Consumer are ack.
The implementation of the transformation is to add a ConsumeQueueAccessLockManager class on the Broker side of RocketMQ to lock the Queue. When a new Consumer pulls a message, judge that if the Queue monitored by the Consumer has a message that has been delivered but has not received an ack and has not timed out, it is not allowed to obtain a lock. The Consumer is not allowed to obtain a lock and pull a message until all the messages delivered by the Queue have been acked or the consumption has timed out.
The logic of obtaining locks in consumqueueaccesslockmanager is as follows:
public synchronized boolean updateAccessControlTable(String group, String topic, String clientId, int queueId) {
if (group != null && topic != null && clientId != null) {
ConcurrentHashMap<String/*Topic*/, ConcurrentHashMap<Integer/*queueId*/, AccessLockEntry>> topicTable = accessLockTable.get(group);
if (topicTable == null) {
topicTable = new ConcurrentHashMap<>();
accessLockTable.put(group, topicTable);
LOG.info("group not exist, put group:{}", group);
}
ConcurrentHashMap<Integer/*queueId*/, AccessLockEntry> queueIdTable = topicTable.get(topic);
if (queueIdTable == null) {
queueIdTable = new ConcurrentHashMap<>();
topicTable.put(topic, queueIdTable);
LOG.info("topic not exist, put topic:{} into group {}", topic, group);
}
AccessLockEntry accessEntry = queueIdTable.get(queueId);
if (accessEntry == null) {
long deliverOffset = brokerController.getConsumeQueueManager().queryDeliverOffset(group, topic, queueId);
accessEntry = new AccessLockEntry(clientId, System.currentTimeMillis(), deliverOffset);
queueIdTable.put(queueId, accessEntry);
LOG.info("mq is not locked. I got it. group:{}, topic:{}, queueId:{}, newClient:{}",
group, topic, queueId, clientId);
return true;
}
//If the Queue is already occupied, the update time
if (clientId.equals(accessEntry.getClientId())) {
accessEntry.setLastAccessTimestamp(System.currentTimeMillis());
accessEntry.setLastDeliverOffset(brokerController.getConsumeQueueManager().queryDeliverOffset(group, topic, queueId));
return false;
}
//Only when the Queue is not occupied and is not a wakeup request can the lock be robbed
else {
long holdTimeThreshold = brokerController.getDeFiBusBrokerConfig().getLockQueueTimeout();
long realHoldTime = System.currentTimeMillis() - accessEntry.getLastAccessTimestamp();
boolean holdTimeout = (realHoldTime > holdTimeThreshold);
long deliverOffset = brokerController.getConsumeQueueManager().queryDeliverOffset(group, topic, queueId);
long lastDeliverOffset = accessEntry.getLastDeliverOffset();
if (deliverOffset == lastDeliverOffset) {
accessEntry.getDeliverOffsetNoChangeTimes().incrementAndGet();
} else {
accessEntry.setLastDeliverOffset(deliverOffset);
accessEntry.setDeliverOffsetNoChangeTimes(0);
}
long ackOffset = brokerController.getConsumeQueueManager().queryOffset(group, topic, queueId);
long diff = deliverOffset - ackOffset;
boolean offsetEqual = (diff == 0);
int deliverOffsetNoChangeTimes = accessEntry.getDeliverOffsetNoChangeTimes().get();
boolean deliverNoChange = (deliverOffsetNoChangeTimes >= brokerController.getDeFiBusBrokerConfig().getMaxDeliverOffsetNoChangeTimes());
if ((offsetEqual && deliverNoChange) || holdTimeout) {
LOG.info("tryLock mq, update access lock table. topic:{}, queueId:{}, newClient:{}, oldClient:{}, hold time threshold:{}, real hold time:{}, hold timeout:{}, offset equal:{}, diff:{}, deliverOffset no change:{}, deliverOffset:{}, ackOffset:{}",
topic,
queueId,
clientId,
accessEntry.getClientId(),
holdTimeThreshold,
realHoldTime,
holdTimeout,
offsetEqual,
diff,
deliverNoChange,
deliverOffset,
ackOffset);
accessEntry.setLastAccessTimestamp(System.currentTimeMillis());
accessEntry.setLastDeliverOffset(deliverOffset);
accessEntry.getDeliverOffsetNoChangeTimes().set(0);
accessEntry.setClientId(clientId);
return true;
}
LOG.info("tryLock mq, but mq locked by other client: {}, group: {}, topic: {}, queueId: {}, nowClient:{}, hold timeout:{}, offset equal:{}, deliverOffset no change times:{}", accessEntry.getClientId(),
group, topic, queueId, clientId, holdTimeout, offsetEqual, deliverOffsetNoChangeTimes);
return false;
}
}
return false;
}The logic of pulling messages in the DeFiPullMessageProcessor is as follows:
@Override
public RemotingCommand processRequest(final ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
final PullMessageRequestHeader requestHeader =
(PullMessageRequestHeader) request.decodeCommandCustomHeader(PullMessageRequestHeader.class);
ConsumerGroupInfo consumerGroupInfo = deFiBrokerController.getConsumerManager().getConsumerGroupInfo(requestHeader.getConsumerGroup());
if (deFiBrokerController.getDeFiBusBrokerConfig().getMqAccessControlEnable() == 1) {
//Access table control is performed only in cluster mode
if (consumerGroupInfo != null && consumerGroupInfo.getMessageModel() == MessageModel.CLUSTERING) {
ClientChannelInfo clientChannelInfo = consumerGroupInfo.getChannelInfoTable().get(ctx.channel());
if (clientChannelInfo != null) {
String group = consumerGroupInfo.getGroupName();
String topic = requestHeader.getTopic();
int queueId = requestHeader.getQueueId();
String clientId = clientChannelInfo.getClientId();
boolean acquired = deFiBrokerController.getMqAccessLockManager().updateAccessControlTable(group, topic, clientId, queueId);
boolean isAllowed = deFiBrokerController.getMqAccessLockManager().isAccessAllowed(group,topic,clientId,queueId);
//It is not a Queue assigned to itself. It returns null
if (!isAllowed) {
RemotingCommand response = RemotingCommand.createResponseCommand(PullMessageResponseHeader.class);
final PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) response.readCustomHeader();
LOG.info("pull message rejected. queue is locked by other client. group:{}, topic:{}, queueId:{}, queueOffset:{}, request clientId:{}",
requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getQueueOffset(), clientId);
responseHeader.setMinOffset(deFiBrokerController.getMessageStore().getMinOffsetInQueue(requestHeader.getTopic(), requestHeader.getQueueId()));
responseHeader.setMaxOffset(deFiBrokerController.getMessageStore().getMaxOffsetInQueue(requestHeader.getTopic(), requestHeader.getQueueId()));
responseHeader.setNextBeginOffset(requestHeader.getQueueOffset());
responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
response.setCode(ResponseCode.PULL_NOT_FOUND);
response.setRemark("mq is locked by other client.");
return response;
}
//After a Q is assigned, the offset is updated to the latest ackOffset to avoid message duplication
if (acquired) {
long nextBeginOffset = correctRequestOffset(group, topic, queueId, requestHeader.getQueueOffset());
if (nextBeginOffset != requestHeader.getQueueOffset().longValue()) {
RemotingCommand response = RemotingCommand.createResponseCommand(PullMessageResponseHeader.class);
final PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) response.readCustomHeader();
response.setOpaque(request.getOpaque());
responseHeader.setMinOffset(deFiBrokerController.getMessageStore().getMinOffsetInQueue(requestHeader.getTopic(), requestHeader.getQueueId()));
responseHeader.setMaxOffset(deFiBrokerController.getMessageStore().getMaxOffsetInQueue(requestHeader.getTopic(), requestHeader.getQueueId()));
responseHeader.setNextBeginOffset(nextBeginOffset);
responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
response.setCode(ResponseCode.PULL_NOT_FOUND);
response.setRemark("lock a queue success, update pull offset.");
LOG.info("update pull offset from [{}] to [{}] after client acquire a queue. clientId:{}, queueId:{}, topic:{}, group:{}",
requestHeader.getQueueOffset(), nextBeginOffset, clientId, requestHeader.getQueueId(),
requestHeader.getTopic(), requestHeader.getConsumerGroup());
return response;
}
else {
LOG.info("no need to update pull offset. clientId:{}, queueId:{}, topic:{}, group:{}, request offset: {}",
clientId, requestHeader.getQueueId(), requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getQueueOffset());
}
}
}
}
}
//...
return response;
}In the scenario of large message volume, the transformation on the Broker side can effectively reduce meaningless repeated delivery and is of great significance to saving network resources. Even if this transformation will affect the performance of the server side, the overall advantages far outweigh the disadvantages. This feature also has strong universality and is fully applicable to other projects. In other words, although great modifications have been made on the Broker side, repeated message delivery may still be caused in retry and other scenarios. The consumer side still needs to do a good job in idempotent processing of consumption.
-- The source code introduced in this paper comes from the Weizhong open source project DeFiBus
https://github.com/WeBankFinTech/DeFiBus/blob/develop/docs/cn/features/3-circuit-break-mechanism.md