SpringBoot + RabbitMQ (ensure that the message is delivered successfully and consumed 100%)

Note: This article covers many aspects of RabbitMQ, such as:

1. Message sending confirmation mechanism

2. Consumption confirmation mechanism

3. Redelivery of messages

4. Consumption idempotence, etc

These are all developed around the overall flow chart above, so it is necessary to post them first, as shown in the figure

1. A brief introduction to the acquisition of 163 email authorization code

2. Write send mail tool class

3. Write RabbitMQ configuration file

4. Producer initiated call

5. Consumer email

6. The scheduled task pulls and resends the failed message regularly

7. Test and verification of various abnormal conditions

8. Extension: using dynamic agent to implement idempotent verification and message ack on the consumer side

1. Spring boot version 2.1.5.RELEASE. Some configuration properties in the old version may not be available and need to be configured in code form

2. RabbitMQ version 3.7.15

3. MailUtil: send mail tool class

4. RabbitConfig: rabbitmq related configuration

5. TestServiceImpl: producer, send message

6. MailConsumer: consumer, consume message, send mail

7. ResendMsg: scheduled task, resend failed message

Note: the above is the core code. None of the MsgLogService mapper xml is posted. For the complete code, please refer to my GitHub. Welcome to fork, https://github.com/wangzaiplus/springboot/tree/wxw

1. 163 access to email authorization code, as shown in the figure:

   

The authorization code is the configuration file spring.mail.password Required password

1. pom

<!--mq--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-amqp</artifactId> </dependency> <!--mail--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-mail</artifactId> </dependency>

1. rabbitmq, mailbox configuration

# rabbitmq spring.rabbitmq.host=localhost spring.rabbitmq.port=5672 spring.rabbitmq.username=guest spring.rabbitmq.password=guest #Turn on confirm callback P - > Exchange spring.rabbitmq.publisher-confirms=true #Turn on returnedMessage callback Exchange - > Queue spring.rabbitmq.publisher-returns=true #Set ack Queue - > C spring.rabbitmq.listener.simple.acknowledge-mode=manual spring.rabbitmq.listener.simple.prefetch=100 # mail spring.mail.host=smtp.163.com spring.mail.username=186****[email protected] spring.mail.password=*** spring.mail.from=186****[email protected] spring.mail.properties.mail.smtp.auth=true spring.mail.properties.mail.smtp.starttls.enable=true spring.mail.properties.mail.smtp.starttls.required=true

Note: password is the authorization code. username and from should be consistent

1. Table structure

CREATE TABLE `msg_log` ( `msg_id` varchar(255) NOT NULL DEFAULT '' COMMENT 'Message unique ID', `msg` text COMMENT 'Message body, json format', `exchange` varchar(255) NOT NULL DEFAULT '' COMMENT 'Switch', `routing_key` varchar(255) NOT NULL DEFAULT '' COMMENT 'Routing key', `status` int(11) NOT NULL DEFAULT '0' COMMENT 'state: 0 In delivery 1 Delivery successful 2 Delivery failed 3 Consumed', `try_count` int(11) NOT NULL DEFAULT '0' COMMENT 'retry count', `next_try_time` datetime DEFAULT NULL COMMENT 'Next retry time', `create_time` datetime DEFAULT NULL COMMENT 'Creation time', `update_time` datetime DEFAULT NULL COMMENT 'Update time', PRIMARY KEY (`msg_id`), UNIQUE KEY `unq_msg_id` (`msg_id`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Message delivery log';

Description: exchange routing_ The key field is used when the scheduled task resends a message

1. MailUtil

@Component @Slf4j public class MailUtil { @Value("$") private String from; @Autowired private JavaMailSender mailSender; /** * Send simple mail * * @param mail */ public boolean send(Mail mail) { String to = mail.getTo();// Target mailbox String title = mail.getTitle();// Message title String content = mail.getContent();// Message body SimpleMailMessage message = new SimpleMailMessage(); message.setFrom(from); message.setTo(to); message.setSubject(title); message.setText(content); try { mailSender.send(message); log.info("Mail sent successfully"); return true; } catch (MailException e) { log.error("Failed to send mail, to: {}, title: {}", to, title, e); return false; } } }

1. RabbitConfig

@Configuration @Slf4j public class RabbitConfig { @Autowired private CachingConnectionFactory connectionFactory; @Autowired private MsgLogService msgLogService; @Bean public RabbitTemplate rabbitTemplate() { RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory); rabbitTemplate.setMessageConverter(converter()); //Message successfully sent to Exchange rabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> { if (ack) { log.info("Message successfully sent to Exchange"); String msgId = correlationData.getId(); msgLogService.updateStatus(msgId, Constant.MsgLogStatus.DELIVER_SUCCESS); } else { log.info("Message sent to Exchange fail, {}, cause: {}", correlationData, cause); } }); //If setReturnCallback is triggered, mandatory=true must be set. Otherwise, if Exchange does not find the Queue, it will discard the message without triggering the callback rabbitTemplate.setMandatory(true); //Whether messages are routed from Exchange to Queue. Note: This is a failure callback. This method will only be called back if messages fail to be routed from Exchange to Queue rabbitTemplate.setReturnCallback((message, replyCode, replyText, exchange, routingKey) -> { log.info("Message from Exchange Route to Queue fail: exchange: {}, route: {}, replyCode: {}, replyText: {}, message: {}", exchange, routingKey, replyCode, replyText, message); }); return rabbitTemplate; } @Bean public Jackson2JsonMessageConverter converter() { return new Jackson2JsonMessageConverter(); } //Send mail public static final String MAIL_QUEUE_NAME = "mail.queue"; public static final String MAIL_EXCHANGE_NAME = "mail.exchange"; public static final String MAIL_ROUTING_KEY_NAME = "mail.routing.key"; @Bean public Queue mailQueue() { return new Queue(MAIL_QUEUE_NAME, true); } @Bean public DirectExchange mailExchange() { return new DirectExchange(MAIL_EXCHANGE_NAME, true, false); } @Bean public Binding mailBinding() { return BindingBuilder.bind(mailQueue()).to(mailExchange()).with(MAIL_ROUTING_KEY_NAME); } }

1. TestServiceImpl production message

@Service public class TestServiceImpl implements TestService { @Autowired private MsgLogMapper msgLogMapper; @Autowired private RabbitTemplate rabbitTemplate; @Override public ServerResponse send(Mail mail) { String msgId = RandomUtil.UUID32(); mail.setMsgId(msgId); MsgLog msgLog = new MsgLog(msgId, mail, RabbitConfig.MAIL_EXCHANGE_NAME, RabbitConfig.MAIL_ROUTING_KEY_NAME); msgLogMapper.insert(msgLog);//Message warehousing CorrelationData correlationData = new CorrelationData(msgId); rabbitTemplate.convertAndSend(RabbitConfig.MAIL_EXCHANGE_NAME, RabbitConfig.MAIL_ROUTING_KEY_NAME, MessageHelper.objToMsg(mail), correlationData);//Send message return ServerResponse.success(ResponseCode.MAIL_SEND_SUCCESS.getMsg()); } }

1. MailConsumer consumption message, sending mail

@Component @Slf4j public class MailConsumer { @Autowired private MsgLogService msgLogService; @Autowired private MailUtil mailUtil; @RabbitListener(queues = RabbitConfig.MAIL_QUEUE_NAME) public void consume(Message message, Channel channel) throws IOException { Mail mail = MessageHelper.msgToObj(message, Mail.class); log.info("Message received: {}", mail.toString()); String msgId = mail.getMsgId(); MsgLog msgLog = msgLogService.selectByMsgId(msgId); if (null == msgLog || msgLog.getStatus().equals(Constant.MsgLogStatus.CONSUMED_SUCCESS)) {//Consumption idempotence log.info("Repeated consumption, msgId: {}", msgId); return; } MessageProperties properties = message.getMessageProperties(); long tag = properties.getDeliveryTag(); boolean success = mailUtil.send(mail); if (success) { msgLogService.updateStatus(msgId, Constant.MsgLogStatus.CONSUMED_SUCCESS); channel.basicAck(tag, false);//Consumption confirmation } else { channel.basicNack(tag, false, true); } } }

Note: in fact, three things have been done: 1. Ensure consumption idempotence, 2. Send email, 3. Update message status, and manually ack

1. ResendMsg scheduled task resend failed message

@Component @Slf4j public class ResendMsg { @Autowired private MsgLogService msgLogService; @Autowired private RabbitTemplate rabbitTemplate; //Maximum delivery times private static final int MAX_TRY_COUNT = 3; /** * Pull the failed message every 30s and resend it */ @Scheduled(cron = "0/30 * * * * ?") public void resend() { log.info("Start scheduled tasks(Resend message)"); List<MsgLog> msgLogs = msgLogService.selectTimeoutMsg(); msgLogs.forEach(msgLog -> { String msgId = msgLog.getMsgId(); if (msgLog.getTryCount() >= MAX_TRY_COUNT) { msgLogService.updateStatus(msgId, Constant.MsgLogStatus.DELIVER_FAIL); log.info("Maximum retries exceeded, Message delivery failed, msgId: {}", msgId); } else { msgLogService.updateTryCount(msgId, msgLog.getNextTryTime());//Delivery times + 1 CorrelationData correlationData = new CorrelationData(msgId); rabbitTemplate.convertAndSend(msgLog.getExchange(), msgLog.getRoutingKey(), MessageHelper.objToMsg(msgLog.getMsg()), correlationData);//Redelivery log.info("The first " + (msgLog.getTryCount() + 1) + " Resend message (s)"); } }); log.info("End of scheduled task execution(Resend message)"); } }

Note: each message is bound to exchange routingKey. All messages can be re submitted to the same timing task

OK, so far, the code is ready. Now, test the normal process

1. Send request:

1. Background log:

   

2. Database message logging:

   

The status is 3, indicating that it has been consumed, and the number of message retries is 0, indicating that one delivery is successful

1. View mailbox

Step 1 lists a lot of knowledge points about RabbitMQ, which are very important and core. This article also involves the realization of these knowledge points. Next, we will verify them through exception tests (these verifications are carried out around the flowchart thrown at the beginning of this article, which is very important, so post it again)

1. The callback in case that the verification message fails to be sent to Exchange corresponds to the above figure p - > x

How to verify? You can randomly specify a nonexistent switch name and request interface to see whether a callback will be triggered

Failed to send, reason: reply code = 404, reply text = not_ FOUND - no Exchange ' mail.exchangeabcd 'in Vhost' / ', the callback can ensure that the message is correctly sent to Exchange, and the test is completed

1. The callback in case of failure to verify message routing from Exchange to Queue, corresponding to the figure above X - > Q

Similarly, modify the route key to nonexistent. If the route fails, a callback will be triggered

Sending failed because: route: mail.routing.keyabcd, replyCode: 312, replyText: NO_ROUTE

1. Verify that in the manual ack mode, the consumer must confirm (ack) manually, otherwise the message will be kept in the queue until it is consumed, corresponding to the figure Q - > C above

Consumer code channel.basicAck(tag, false); / / after the consumption confirmation is commented out, view the console and rabbitmq console

It can be seen that although the message is indeed consumed, the message is still saved by rabbitmq due to the manual confirmation mode and no manual confirmation at last. Therefore, the manual ack can ensure that the message must be consumed, but you must remember the basic ack

1. Verify the idempotence of the consumer

Next, remove the comments and restart the server. Since there is a message that has not been acked, it will listen to the message and consume it after restart. However, before consumption, it will judge whether the status of the message has not been consumed. It is found that status=3, that is, it has been consumed. Therefore, direct return ensures the idempotence of the consumer, even if the callback is not triggered due to the successful delivery of the network and other reasons, So it can be delivered many times without repeated consumption and business abnormality

1. Verify that the exception message on the consumer side will not be lost

Obviously, there may be exceptions in the consumer code. If we don't handle it, the business doesn't execute correctly, but the message is missing, which gives us the feeling that the message is lost. Because we have made exception capture in the consumer code, when the business is abnormal, it will trigger: channel.basicNack(tag, false, true);, which will tell rabbitmq that the message consumption failed and needs to be rejoined, It can be re delivered to other normal consumers for consumption, so as to ensure that messages are not lost

Test: the send method returns false directly

As you can see, because channel.basicNack(tag, false, true), unacked messages will be rejoined and consumed, which ensures that messages will not be lost

1. Verify message replay of scheduled task

In the actual application scenario, MQ may be down due to network reasons or the message is not persisted, which causes the callback method ConfirmCallback of delivery confirmation not to be executed, so that the message state of the database is always in the state of delivery. At this time, message re delivery is required, even if the message has been consumed

The timing task is only to ensure that the message is delivered 100% successfully, and the consumption idempotence of multiple delivery needs to be guaranteed by the consumer itself

We can comment out the code to update the message status after the callback and consumption are successful, start the scheduled task, and check whether it is re invested

It can be seen that the message will be re delivered three times and abandoned more than three times, and the message status will be set as delivery failure status. In this abnormal situation, it is necessary to manually intervene to find out the reason

I don't know if you find that in MailConsumer, the real business logic is just to send email mailUtil.send(mail), but we have to check the consumption idempotence before calling the send method. After sending, we need to update the message status to "consumed" and manually ack. In the actual project, there may be many producer consumer application scenarios, such as logging, rabbitmq is needed to send SMS and so on. If we write these repeated public codes every time, it is unnecessary and difficult to maintain. Therefore, we can extract the public codes and let the core business logic only care about its own implementation without other operations. In fact, it is AOP

To achieve this goal, there are many ways. You can use spring aop, interceptor, static agent or dynamic agent. Here, I use dynamic agent

The directory structure is as follows:

The core code is the implementation of the agent, so we will not paste all the codes here, but just provide a way of thinking. We should write the codes as succinctly and elegantly as possible

In fact, sending email is very simple, but in fact, there are many points that need to be noticed and improved. A seemingly small knowledge point can also lead to many problems, even involving all aspects, which need to be stepped on by myself. Of course, there are many points that need to be improved and optimized in my code. I hope that my little partner can give more opinions and suggestions

My code has been tested and verified by self-test. The pictures are drawn by myself or carefully cut by myself. I hope that my friends can learn something, like or pay attention to it when passing by. Thank you

5 June 2020, 00:58 | Views: 8292