Message confirmation mechanism of RabbitMq and its application in RabbitMq chat function

1. Causes of message loss

1. When the sender sends a message, the message is lost. It seems that the message has been sent
2.mq has received the message, but the consumer did not receive it when consuming


Ensure reliable arrival at p - > b, e - > Q and Q - > C; (picture from Shangri Valley)

2. Callback function and configuration on the production side

#Enable sender confirmation
spring.rabbitmq.publisher-confirm-type=correlated
#Enable the confirmation of the arrival queue of the sender message
spring.rabbitmq.publisher-returns=true
#As long as the queue arrives, the returnconfirm is called back asynchronously
spring.rabbitmq.template.mandatory=true

@Slf4j
@Configuration
public class MyRabbitConfig {
    @Autowired
    RabbitTemplate rabbitTemplate;
 
    @Bean
    public MessageConverter messageConverter(){
        return new Jackson2JsonMessageConverter();
    }
 
    /**
     *Customize RabbitTemplate
     * 1.The service calls back when it receives a message
     *    1)spring.rabbitmq.publisher-confirms=true
     *    2)Set the confirmation callback confirmCallback
     * 2.The message arrives in the queue correctly for callback
     *    1)spring.rabbitmq.publisher-returns=true
     *    2) spring.rabbitmq.template.mandatory=true
     *    Set the confirmation callback ReturnCallback
     * 3.Consumer side confirmation (ensure that each consumption is correctly consumed, and then the broker can delete this message)
     *    1)The default is automatic confirmation. As long as the message is received, the client will automatically confirm and the server will remove the message
     *       Question:
     *           We receive many messages and automatically reply to the server ack. Only one message is processed successfully. If it goes down, the message will be lost.
     *           In the consumer manual confirmation mode, as long as we do not explicitly tell MQ that the goods are signed in, there is no ACK
     *           The message is always unacknowledged, even if the Consumer goes down. The message will not be lost and will become ready again
     *   2)How to sign in:
     *      channel.basicAck(deliveryTag,false);Sign in acquisition
     *      channel.basicNack(deliveryTag,false,true);deny sb. a visa
     *
     */
 
    @PostConstruct   //After the configuration class object is created, execute this method
    public void initRabbitTemplate(){
        RabbitTemplate.ConfirmCallback confirmCallback = new RabbitTemplate.ConfirmCallback() {
            @Override
            public void confirm(CorrelationData correlationData, boolean ack, String cause) {
                if (ack){
                    System.out.println("Sent successfully");
                }else {
                    System.out.println("fail in send");
                }
            }
        };
        rabbitTemplate.setConfirmCallback(confirmCallback);
 
        rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {
            @Override
            public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
                System.out.println(message);
                System.out.println(replyCode);
                System.out.println(replyText);
            }
        });
    }
 
 
}

(the above code is from Shangsi Valley)

3. Consumer ack mode and chat related codes

Springboot rabbit provides three message confirmation modes:
AcknowledgeMode.NONE: unconfirmed mode (no matter whether the program is abnormal or not, as long as the listening method is executed, the message will be consumed. It is equivalent to automatic confirmation in rabbitmq, which is not recommended)
AcknowledgeMode.AUTO: automatic confirmation mode (by default, the consumer will automatically confirm if there is no exception, and will not confirm if there is any exception. Infinite retry will lead to dead loop of the program. Do not confuse it with automatic confirmation in rabbit)
AcknowledgeMode.MANUAL: manual confirmation mode (channel.basicAck needs to be called manually for confirmation, which can catch exceptions, control the number of retries, and even control the processing method of failure messages)

Manual confirmation mode is generally recommended
Examples are as follows:
Configuration of springboot

acknowledge-mode: none, auto, manual
none: automatic confirmation; auto: according to the situation; Manual: manual confirmation

spring:
  rabbitmq:
    host: localhost
    username: guest
    password: guest
    virtual-host: /
    listener:
      simple:
        acknowledge-mode: manual  // Start manual confirmation
        retry:
          enabled: true  // Turn on Retry
          max-attempts: 3  //max retries 
          initial-interval: 2000ms  //Retry interval
@RestController
@RequestMapping("/mq")
//@Api(tags = "mq related interface", description = "MqController | message module")
public class MqController {
    private static final Logger logger = LoggerFactory.getLogger(MqController.class);

    @Autowired
    private ProdServer prodServer;
    @Autowired
    WebSocketServer webSocketServer;

    @RabbitListener(queues = {FanoutRabbitConfig.DEFAULT_BOOK_QUEUE})
    public void listenerAutoAck(String text, Message message, Channel channel) {

        // If you manually ACK TODO, the message will be monitored and consumed, but the message still exists in the queue. If acknowledge mode is not configured, it will automatically ACK after consumption
        final long deliveryTag = message.getMessageProperties().getDeliveryTag();
        try {
            logger.info("[A consumer listens to a message] - [{}]", text);
            ChatDto.MqChat chatDto = JSONObject.parseObject(text, ChatDto.MqChat.class);
            Integer rs = WebSocketServer.sendInfo(chatDto.getMessage(), chatDto.getToUserId());
//            yanUserChatService.saveChat(chatDto.getOpenid(),chatDto,rs);
//            webSocketServer.sendMqMessage(text);

            // TODO notifies MQ that the message has been successfully consumed and can be ACK
            channel.basicAck(deliveryTag, false);
        } catch (IOException e) {
            try {
                // TODO processing failed, press MQ again
                channel.basicRecover();
            } catch (IOException e1) {
                e1.printStackTrace();
            }
        }
    }
}

 	@ApiOperation("Chat core method ")
    public void defaultMessage(@RequestBody ChatDto chatDto,
                               HttpServletRequest request)  throws IOException{

        String message = chatDto.getContent();
        String toUserId = chatDto.getEmail();
        String openid = Util.fromRequestToOpenid(request);
        Map<String, String> map = new HashMap<String, String>();
        map.put("message",message);
        map.put("toUserId",toUserId);
        map.put("openid",openid);
        rabbitTemplate.convertAndSend("fanoutExchange", "", JSONObject.toJSONString(map));
    }

websocket related code

 public static Integer sendInfo(String message,@PathParam("userId") String userId) throws IOException {
        log.info("Send message to:"+userId+"´╝îmessage:"+message);
        if(StringUtils.isNotBlank(userId)&&webSocketMap.containsKey(userId)){
            //It's better to encapsulate the message request in this place, otherwise the front end is difficult to handle
            JSONObject jsonObject =new JSONObject();
            //Add sender (prevent serial modification)
            jsonObject.put("type","receive");
            jsonObject.put("content",message);
            webSocketMap.get(userId).sendMessage(jsonObject.toJSONString());

            log.info("Send message to succeeded");
            return 1;
        }else{
            log.error("user"+userId+",Not online!");
            return 0;
        }
    }

The application of rabbitmq under the chat situation; Decouple the business logic after sending the message to improve the performance in the chat process.

4. How to ensure message reliability

1. Message loss

The message was sent out and did not reach the server due to network problems
Make a try catch method. Sending messages may cause network failure. After the failure, there must be a retry machine
The system can be recorded in the database, and the method of regular scanning and retransmission is adopted
Log whether each message status is received by the server
Do a good job of regular retransmission. If the message is not sent successfully, regularly scan the database for unsuccessful messages
The line resend message arrives at the Broker, and the Broker is successful only when it writes the message to disk (persistence). At this time, the Broker has not completed persistence and is down.
publisher must also add a confirmation callback mechanism to confirm successful messages and modify the database message status.
In the state of automatic ACK. Consumer receives the message but fails to get the message and goes down
The manual ACK must be enabled, and it can be removed only after the consumption is successful. In case of failure, or noAck and rejoin the team before processing

2. Duplicate message

The message consumption is successful, the transaction has been committed, and the machine goes down during ack. As a result, no ack is successful, and the Broker's message is changed from unack to ready again and sent to other consumers
Message consumption failed. Due to the retry mechanism, the message will be sent again automatically
After successful consumption, the server goes down during ack, the message changes from unack to ready, and the Broker sends it again
The consumer's business consumer interface should be designed to be idempotent. For example, inventory deduction has the status flag of the work order
Using the anti duplication table (redis/mysql), each message sent has a unique business ID, and it will not be processed after processing
Each message of rabbitMQ has a redelivered field, which can obtain whether it is redelivered rather than delivered for the first time

3. Message backlog

Consumer downtime backlog
Insufficient consumer spending power and backlog
The sender sends too much traffic
Online more consumers for normal consumption
Online special queue consumption service, take out messages in batches, record them in the database, and process them slowly offline
--------
Part IV original link: https://blog.csdn.net/weixin_40566934/article/details/119643740

5. Use of rabbitmq dead letter queue and delay queue

--------
Original link: https://blog.csdn.net/fu_huo_1993/article/details/88350188

Tags: Java RabbitMQ MQ

Posted on Sun, 12 Sep 2021 22:40:04 -0400 by ir4z0r