1, springBoot integrates RabbitMQ
1. IDEA creates a SpringBoot project
2. Import related dependencies
<!--Import dependency-->
<dependencies>
<!--RabbitMQ rely on-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.47</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<!--swagger-->
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>2.9.2</version>
</dependency>
<!--RabbitMQ Test dependency-->
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
3. Configure RabbitMQ connection information
# rabbit connection information spring.rabbitmq.host=192.168.115.128 spring.rabbitmq.port=5672 spring.rabbitmq.username=admin spring.rabbitmq.password=123
2, Delay queue
1. Delay queue concept
The delay queue is orderly. The most important feature is reflected in its delay attribute. The elements in the delay queue want to be taken out and processed after or before the specified time. In short, the delay queue is a queue used to store the elements that need to be processed at the specified time.
2. Delay queue usage scenarios
1. If the order is not paid within ten minutes, it will be automatically cancelled
2. If the newly created store has not uploaded goods within ten days, it will automatically send a message reminder.
3. After successful registration, if the user does not log in within three days, a short message reminder will be sent.
4. The user initiates a refund, and if it is not handled within three days, notify the relevant operators.
5. After the scheduled meeting, all participants shall be notified to attend the meeting ten minutes before the scheduled time point
These scenarios have a feature that a task needs to be completed at a specified time point after or before an event occurs. For example, when an order generation event occurs, check the payment status of the order ten minutes later, and then close the unpaid order; It seems that using a scheduled task, polling the data all the time, checking once a second, taking out the data to be processed, and then processing is finished? If the amount of data is small, this can be done. For example, for the demand of "automatic settlement if the bill is not paid within one week", if the time is not strictly limited, but a week in a loose sense, running a regular task every night to check all unpaid bills is indeed a feasible scheme. However, for scenarios with large amount of data and strong timeliness, such as: "if the order is not paid within ten minutes, it will be closed ", there may be a lot of unpaid order data in the short term, even reaching the level of millions or even tens of millions during the event. It is obviously undesirable to still use polling for such a huge amount of data. It is likely that the inspection of all orders can not be completed in one second. At the same time, it will put great pressure on the database, fail to meet business requirements and have low performance.
3. TTL in RabbitMQ
3.1 what is TTL?
TTL is the attribute of a message or queue in RabbitMQ, indicating the maximum lifetime of a message or all messages in the queue, in milliseconds.
In other words, if a message has the TTL attribute set or enters the queue with the TTL attribute set, the message will become "dead letter" if it is not consumed within the time set by the TTL. If both the TTL of the queue and the TTL of the message are configured, the smaller value will be used. There are two ways to set the TTL.
3.2. Message setting TTL
It is flexible to set TTL for each message. The TTL of each message can be different, but if the TTL of the message is not, our message will never be outdated by default.
3.3. Queue setting TTL
When creating a queue, setting the "x-message-ttl" attribute of the queue has limitations, because when creating a queue, TTL is specified, and the TTL of the queue is determined and cannot be modified.
3, Delay queue TTL example (using dead letter)
1. Example introduction
Create two queues QA and QB. Set the TTL of the two queues to 10S and 40S respectively. Then create a switch X and a dead letter switch Y, both of which are direct. Create a dead letter queue QD. Their binding relationship is as follows:
2. Configuration file class (create switch, queue, binding relationship)
@Configuration
public class TtlQueueConfig {
// General switch
public static final String X_EXCHANGE = "X";
// General queue QA
public static final String QUEUE_A = "QA";
// Normal queue QB
public static final String QUEUE_B = "QB";
// Dead letter switch
public static final String Y_DEAD_LETTER_EXCHANGE = "Y";
// Dead letter queue QD
public static final String DEAD_LETTER_QUEUE = "QD";
// Routing key of dead letter queue
public static final String DEAD_LETTER_ROUTING_KEY = "YD";
// Routing key between queues A,B and switch
public static final String A_X_ROUTING_KEY = "XA";
public static final String B_X_ROUTING_KEY = "XB";
// 1. Declare that the general switch is a direct switch
@Bean("xExchange")
public DirectExchange xExchange() {
return new DirectExchange(X_EXCHANGE);
}
// 2. Declare that the dead letter switch is a direct switch
@Bean("yExchange")
public DirectExchange yExchange() {
return new DirectExchange(Y_DEAD_LETTER_EXCHANGE);
}
/**
* 3,Declare normal delay queue QA
* Use the queue builder to complete the queue creation (the queue builder is directly constructed according to our requirements), and directly create a persistent queue ()
* We can construct other parameters, but we can not use the default parameters
*/
@Bean("queueA")
public Queue queueQA() {
return QueueBuilder
.durable(QUEUE_A) // Construct persistent
.ttl(10000) // Set the message expiration time. If the message is not consumed after this time, it will be put into the dead letter queue
.deadLetterExchange(Y_DEAD_LETTER_EXCHANGE) // Switch for setting dead letter queue
.deadLetterRoutingKey(DEAD_LETTER_ROUTING_KEY) // Set the routing key of dead letter queue
.build(); // structure
}
/**
* 4,Declare that the creation process of common delay queue QB is the same as that of queue A
*/
@Bean("queueB")
public Queue queueQB() {
return QueueBuilder
.durable(QUEUE_B) // Construct persistent
.ttl(40000) // Set the message expiration time. If the message is not consumed after this time, it will be put into the dead letter queue
.deadLetterExchange(Y_DEAD_LETTER_EXCHANGE) // Switch for setting dead letter queue
.deadLetterRoutingKey(DEAD_LETTER_ROUTING_KEY) // Set the routing key of dead letter queue
.build(); // structure
}
/**
* 5,Declare dead letter queue
*/
@Bean("queueD")
public Queue queueQD() {
return QueueBuilder.durable(DEAD_LETTER_QUEUE).build();
}
/**
* 6,Declare queue A bound X switch
*
* @Qualifier According to the id injection of the bean specified by us, it is generally used with our @ autowrite
*/
@Bean
public Binding queueABindingX(@Qualifier("queueA") Queue queueA,
@Qualifier("xExchange") DirectExchange xExchange) {
return BindingBuilder.bind(queueA).to(xExchange).with(A_X_ROUTING_KEY);
}
/**
* 7,Declare queue B binding X switch
*
* @Qualifier According to the id injection of the bean specified by us, it is generally used with our @ autowrite
*/
@Bean
public Binding queueBBindingX(@Qualifier("queueB") Queue queueB,
@Qualifier("xExchange") DirectExchange xExchange) {
return BindingBuilder.bind(queueB).to(xExchange).with(B_X_ROUTING_KEY);
}
/**
* 8,Declare dead letter queue binding Y switch
*
* @Qualifier According to the id injection of the bean specified by us, it is generally used with our @ autowrite
*/
@Bean
public Binding queueDBindingY(@Qualifier("queueD") Queue queueB,
@Qualifier("yExchange") DirectExchange yExchange) {
return BindingBuilder.bind(queueB).to(yExchange).with(DEAD_LETTER_ROUTING_KEY);
}
}
3. Message producer code
@RestController
@Slf4j
@RequestMapping("/ttl")
public class SendMsgController {
@Autowired
private RabbitTemplate rabbitTemplate;
/**
* Test send message
*
* @param message
*/
@GetMapping("/sendMsg/{message}")
public void sendMsg(@PathVariable String message) {
log.info("Current time:{},Send a message to two TTL queue:{}", new Date(), message);
// Send to switch X and bind the queue with routing key XA
rabbitTemplate.convertAndSend("X", "XA", "Message from ttl For 10 S Queue of: " + message);
// Send to switch X and bind the queue with routing key XA
rabbitTemplate.convertAndSend("X", "XB", "Message from ttl For 40 S Queue of: " + message);
}
4. Message consumer code
@Slf4j
@Component
public class DeadLetterQueueConsumer {
public static final String DEAD_LETTER_QUEUE = "QD";
@RabbitListener(queues = DEAD_LETTER_QUEUE)
public void receiveD(Message message) throws Exception {
String msg = new String(message.getBody(), "UTF-8");
log.info("Current time:{},Received dead letter queue information{}", new Date().toString(), msg);
}
}
4, The above example TTL optimization
4.1. Above problems
After 10S, the first message becomes a dead letter message and is consumed by the consumer. After 40S, the second message becomes a dead letter message and is consumed. Such a delay queue is completed. However, if it is used in this way, it is necessary to add a queue for every new time demand. There are only 10S and 40S time options , if it takes one hour to process, you need to add a queue with a TTL of one hour. If you book a conference room and notify in advance, don't you need to add countless queues to meet the demand?
4.2 solutions
We add a new queue QC. Instead of setting the queue TTL, we set the message TTL when sending messages to this queue, so as to achieve queue reuse. For the same queue, only the TTL of each message is different.
/**
* Declare a generic delay queue
*/
@Bean("queueC")
public Queue queueQC() {
return QueueBuilder.durable(QUEUE_C) // Sustainable queue
.deadLetterExchange(Y_DEAD_LETTER_EXCHANGE) // Switch for setting dead letter queue
.deadLetterRoutingKey(DEAD_LETTER_ROUTING_KEY) // Set the routing key of dead letter queue
.build();
}
/**
* Bind queue C and our exchange X
*/
@Bean
public Binding queueCBindingX(@Qualifier("queueC") Queue queueC,
@Qualifier("xExchange") DirectExchange xExchange) {
return BindingBuilder.bind(queueC).to(xExchange).with(C_X_ROUTING_KEY);
}
Producer Code:
When sending a message, specify the TTL of our message and use the TTL setting of MessagePostProcessor message.
/**
* Test send message
* The message itself sets the specified ttl
*
* @param message
*/
@GetMapping("/sendMsg/{message}/{ttl}")
public void sendMsg(@PathVariable String message, @PathVariable String ttl) {
log.info("Current time:{},Send a message to TTL queue:{}", new Date(), message);
// The functional interface MessagePostProcessor needs a message object. After setting the message TTL, it returns message
rabbitTemplate.convertAndSend("X", "XC", message, (msg) -> {
// Set the length of time to send the message
msg.getMessageProperties().setExpiration(ttl);
// Return our message object
return msg;
});
}
4.3. Remaining problems
We send two messages. The first one is delayed by 20 seconds
http://localhost:8080/ttl/sendExpirationMsg/ Hello 1 / 20000
The second message is delayed for 2 seconds
http://localhost:8080/ttl/sendExpirationMsg/ Hello 1 / 2000
However, we found that message 2 should enter the dead letter queue earlier than our message 1, but it still enters our dead letter queue after the message is delayed for 20 seconds (the queue pays attention to first come first out). However, it does not meet our needs. We need external force to change it and complete it with the help of RabbitMQ plug-in.
It seems that there is no problem, but it was mentioned at the beginning that if the TTL is set on the message attribute, the message may not "die" on time ", because RabbitMQ only checks whether the first message has expired. If it has expired, it will be sent to the dead letter queue. If the delay time of the first message is long and the delay time of the second message is short, the second message will not be executed first.
5, The Rabbitmq plug-in implements delay queues
The problem mentioned above is indeed a problem. If you can't implement TTL on message granularity and make it die in time at the set TTL time, you can't design a general delay queue. How to solve it? Next, let's solve this problem.
1. RabbitMQ installs the delay queue plug-in
1. Download on the official website https://www.rabbitmq.com/community-plugins.html , Download
rabbitmq_delayed_message_exchange plug-in, then unzip and place it in the plug-in directory of RabbitMQ.
2. Enter the plugin directory under the RabbitMQ installation directory, execute the following command to make the plug-in effective, and then restart RabbitMQ
# 1 /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.8/plugins # 2 rabbitmq-plugins enable rabbitmq_delayed_message_exchange # 3 restart the RabbitMQ service systemctl restart rabbitmq-server
2. Check whether the plug-in is installed successfully
If the plug-in is installed successfully, when we select the switch, there will be one more type, which is the delay queue switch
3. Working principle of plug-in
Originally, we judged whether the message expired after the message entered the queue. Now we judge whether the message expired in the switch. The expired message is directly sent to the specified queue (there is no need for dead letter queue, which becomes very simple)
4. Configure queue instance, switch, binding relationship
Unlike the previous ones, we need to customize the switch type and specify the delay queue switch.
@Configuration
public class DelayedQueueConfig {
// Delay queue
public static final String DELAYED_QUEUE_NAME = "delayed.queue";
// Delay switch
public static final String DELAYED_EXCHANGE_NAME = "delayed.exchange";
// Delay switch and queue key
public static final String DELAYED_ROUTING_KEY = "delayed.routingKey";
/**
* Create a delay queue
*/
@Bean
public Queue delayedQueue() {
return QueueBuilder.durable(DELAYED_QUEUE_NAME).build();
}
/**
* Custom switch what we customize here is a delay switch
*/
@Bean
public CustomExchange delayedExchange() {
Map<String, Object> args = new HashMap<>();
//Customize the type of switch and use the delay switch provided by the delay plug-in
args.put("x-delayed-type", "direct");
// Switch name, switch type, persistence, not automatically deleted, parameter map
return new CustomExchange(DELAYED_EXCHANGE_NAME, "x-delayed-message", true, false, args);
}
/**
* Bind our switch and our queue
*
* @param queue
* @param delayedExchange
* @return
*/
@Bean
public Binding bindingDelayedQueue(@Qualifier("delayedQueue") Queue queue,
@Qualifier("delayedExchange") CustomExchange delayedExchange) {
return BindingBuilder.bind(queue).to(delayedExchange).with(DELAYED_ROUTING_KEY).noargs();
}
}
5. Message producer code
Unlike before, we set the message sending delay time, not the TTL of the message, and complete the operation by delaying our message sending.
/**
* Send delay message
*
* @param message
* @param delayTime
*/
@GetMapping("/sendDelayMsg/{message}/{delayTime}")
public void sendMsg(@PathVariable String message, @PathVariable Integer delayTime) {
log.info("Current time:{},Send a delay message{}ms Information to TTL queue:{}", new Date(), delayTime, message);
rabbitTemplate.convertAndSend(DelayedQueueConfig.DELAYED_EXCHANGE_NAME, DelayedQueueConfig.DELAYED_ROUTING_KEY, message, (msg) -> {
// Set the delay time when sending messages. Unit: ms
msg.getMessageProperties().setDelay(delayTime);
return msg;
});
}
6. Consumer code
/**
* Listen to our delay queue
*
* @param message
* @throws Exception
*/
@RabbitListener(queues = DelayedQueueConfig.DELAYED_QUEUE_NAME)
public void receiveD(Message message) throws Exception {
String msg = new String(message.getBody(), "UTF-8");
log.info("Current time:{},Received dead letter queue information{}", new Date().toString(), msg);
}
7. Effect demonstration
The effect is full of our needs
6, Summary
Delay queue is very useful in situations where delay processing is required. Using RabbitMQ to implement delay queue can make good use of RabbitMQ's features, such as reliable message sending, reliable message delivery and dead letter queue to ensure that messages are consumed at least once and messages that have not been processed correctly will not be discarded. In addition, RabbitMQ cluster features can be used to solve this problem To solve the single point of failure problem, the delay queue will not be unavailable or messages will not be lost because a single node hangs.
Of course, there are many other options for delay queue, such as using Java's DelayQueue, Redis's zset, Quartz or kafka's time wheel. These methods have their own characteristics, depending on the applicable scenarios
It is recommended to use RabbitMQ as the delay queue and use the plug-in to complete it, which is more in line with the requirements.