1. Delay queue - concept understanding
The delay queue is orderly. The most important feature is reflected in its delay attribute. The elements in the delay queue want to be taken out and processed after or before the specified time. In short, the delay queue is a queue used to store the elements that need to be processed at the specified time.
Usage scenario of delay queue:
- If the order is not paid within ten minutes, it will be automatically cancelled.
- If the newly created store has not uploaded products within ten days, it will automatically send a message reminder.
- After successful registration, if the user does not log in within three days, a short message reminder will be sent.
- The user initiates a refund, and if it is not handled within three days, notify the relevant operation personnel.
- After the scheduled meeting, all participants shall be notified to attend the meeting ten minutes before the scheduled time point.
These scenarios have a feature that a task needs to be completed at a specified time point after or before an event occurs. For example, when an order generation event occurs, check the payment status of the order ten minutes later, and then close the unpaid order; It seems that using a scheduled task, polling the data all the time, checking once a second, taking out the data to be processed, and then processing is finished? If the amount of data is small, this can be done. For example, for the demand of "automatic settlement if the bill is not paid within one week", if the time is not strictly limited, but a week in a loose sense, running a regular task every night to check all unpaid bills is indeed a feasible scheme. However, for scenarios with large amount of data and strong timeliness, such as: "if the order is not paid within ten minutes, it will be closed ", there may be a lot of unpaid order data in the short term, even reaching the level of millions or even tens of millions during the event. It is obviously undesirable to still use polling for such a huge amount of data. It is likely that the inspection of all orders can not be completed in one second. At the same time, it will put great pressure on the database, fail to meet business requirements and have low performance.
Therefore, this needs to be implemented in combination with delay queue and TTL.
What is TTL? TTL is the attribute of a message or queue in RabbitMQ, indicating the maximum lifetime of a message or all messages in the queue, in milliseconds. In other words, if a message has the TTL attribute set or enters the queue with the TTL attribute set, it will become a message if it is not consumed within the time set by TTL "Dead letter". If both the TTL of the queue and the TTL of the message are configured
TTL, then the smaller value will be used. There are two ways to set TTL.
- To set TTL properties on a message:
msg.getMessageProperties().setExpiration(ttlTime);
- To set TTL properties on a queue:
map.put("x-message-ttl",40000);If the TTL attribute of the queue is set, once the message expires, it will be discarded by the queue (if the dead letter queue is configured, it will be discarded into the dead letter queue) In the second way, even if the message expires, it will not be discarded immediately, because whether the message expires is determined before it is delivered to the consumer. If there is a serious message backlog in the current queue, the expired message may survive for a long time. In addition, it should be noted that if TTL is not set, it means that the message will never expire, If TTL is set to 0, it means that the message will be discarded unless it can be delivered directly to the consumer at this time.
In the last article, we introduced dead letter queue and TTL. So far, the two elements of using RabbitMQ to implement delay queue have been gathered. Next, we just need to integrate them and add a little seasoning to make the delay queue fresh. Think about it, delay queue is just how long you want messages to be processed, and TTL can just eliminate them On the other hand, the messages that become dead letters will be delivered to the dead letter queue. In this way, consumers only need to consume the messages in the dead letter queue all the time, because the messages inside are messages that they want to be processed immediately.
2. Delay queue - Application Example
Here we complete the code implementation of delay queue by integrating SpringBoot. The first is the pom file
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.28</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
</dependencies>Then let's write down the core configuration file first, mainly the related information of rabbitmq connection.
spring.rabbitmq.host=192.168.40.130 spring.rabbitmq.port=5672 spring.rabbitmq.username=root spring.rabbitmq.password=root
Next, we need to complete the definition of switch and queue and the binding between them. Because SpringBoot is integrated, this part can be written in a configuration class.
In this case, the delay queue is realized by setting TTL for the queue. The TTL of one queue is 10s and that of the other queue is 40s.
package com.szh.rabbitmq.config;
import org.springframework.amqp.core.*;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.HashMap;
import java.util.Map;
/**
*
*/
@Configuration
public class TtlQueueConfig {
//Common switch name
public static final String X_EXCHANGE = "X";
//Common queue name
public static final String QUEUE_A = "QA";
//Common queue name
public static final String QUEUE_B = "QB";
//Dead letter switch name
public static final String Y_DEAD_LETTER_EXCHANGE = "Y";
//Dead letter queue name
public static final String DEAD_LETTER_QUEUE = "QD";
//Declare normal Exchange
@Bean("xExchange")
public DirectExchange xExchange() {
return new DirectExchange(X_EXCHANGE);
}
//Declare dead letter Exchange
@Bean("yExchange")
public DirectExchange yExchange() {
return new DirectExchange(Y_DEAD_LETTER_EXCHANGE);
}
//Declare normal Queue
@Bean("queueA")
public Queue queueA() {
Map<String,Object> map = new HashMap<>(3);
//Set up dead letter switch
map.put("x-dead-letter-exchange",Y_DEAD_LETTER_EXCHANGE);
//Set dead letter RoutingKey
map.put("x-dead-letter-routing-key","YD");
//Set TTL
map.put("x-message-ttl",10000);
return QueueBuilder.durable(QUEUE_A).withArguments(map).build();
}
//Declare normal Queue
@Bean("queueB")
public Queue queueB() {
Map<String,Object> map = new HashMap<>(3);
//Set up dead letter switch
map.put("x-dead-letter-exchange",Y_DEAD_LETTER_EXCHANGE);
//Set dead letter RoutingKey
map.put("x-dead-letter-routing-key","YD");
//Set TTL
map.put("x-message-ttl",40000);
return QueueBuilder.durable(QUEUE_B).withArguments(map).build();
}
//Declare dead letter queue
@Bean("queueD")
public Queue queueD() {
return QueueBuilder.durable(DEAD_LETTER_QUEUE).build();
}
//Bind queue to switch
@Bean
public Binding queueABindingX(@Qualifier("queueA") Queue queueA,
@Qualifier("xExchange") DirectExchange xExchange) {
return BindingBuilder.bind(queueA).to(xExchange).with("XA");
}
@Bean
public Binding queueBBindingX(@Qualifier("queueB") Queue queueB,
@Qualifier("xExchange") DirectExchange xExchange) {
return BindingBuilder.bind(queueB).to(xExchange).with("XB");
}
@Bean
public Binding queueDBindingY(@Qualifier("queueD") Queue queueD,
@Qualifier("yExchange") DirectExchange yExchange) {
return BindingBuilder.bind(queueD).to(yExchange).with("YD");
}
}
With queues, switches and routingkeys between them, producers and consumers are also required.
The controller sends the request, which is naturally the producer, and the Consumer receives the message.
package com.szh.rabbitmq.controller;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Date;
/**
*
*/
@Slf4j
@RestController
@RequestMapping(value = "/ttl")
public class SendController {
@Autowired
private RabbitTemplate rabbitTemplate;
@GetMapping(value = "/sendMsg/{message}")
public void sendMsg(@PathVariable String message) {
log.info("Current time:{}，Send one to two TTL Queue:{}",new Date().toString(),message);
rabbitTemplate.convertAndSend("X","XA","Message from TTL For 10 s Queue for:" + message);
rabbitTemplate.convertAndSend("X","XB","Message from TTL For 40 s Queue for:" + message);
}
}
package com.szh.rabbitmq.listener;
import com.rabbitmq.client.Channel;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
import java.util.Date;
/**
*
*/
@Slf4j
@Component
public class DeadLetterQueueConsumer {
//receive messages
@RabbitListener(queues = "QD")
public void receiveD(Message message, Channel channel) throws Exception {
String msg = new String(message.getBody());
log.info("Current time:{}，Messages received from dead letter queue:{}",new Date().toString(),msg);
}
}
Finally, let's start the test.
As can be seen from the results, we first sent a message to two queues in MQ through the producer. This message was not consumed immediately. The first message became a dead letter message after 10S, and then consumed by the consumer. The second message became a dead letter message after 40S, and then consumed. In this way, the creation of a delay queue is completed.
However, if it is used in this way, a queue will be added every time a new time demand is added. There are only two time options: 10S and 40S. If it needs to be processed after one hour, it needs to add a queue with TTL of one hour. If it is a scenario of booking a meeting room and notifying in advance, it needs to add countless queues to meet the demand Demand?
Therefore, we need to talk about the optimization of delay queue (setting TTL attribute for messages).
3. Delay queue optimization - Application Example
The pom file and core configuration file are the same as the above case, so they are not given here. Next, refer to the architecture diagram (mainly switches X and Y, common queue QC and dead letter queue QD) for optimization.
First, define the queue, switch, and Binding between them in the configuration class.
package com.szh.rabbitmq.config;
import org.springframework.amqp.core.*;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.HashMap;
import java.util.Map;
/**
*
*/
@Configuration
public class TtlQueueConfig {
//Common switch name
public static final String X_EXCHANGE = "X";
//Common queue name
public static final String QUEUE_A = "QA";
//Common queue name
public static final String QUEUE_B = "QB";
//Dead letter switch name
public static final String Y_DEAD_LETTER_EXCHANGE = "Y";
//Dead letter queue name
public static final String DEAD_LETTER_QUEUE = "QD";
//Declare normal Exchange
@Bean("xExchange")
public DirectExchange xExchange() {
return new DirectExchange(X_EXCHANGE);
}
//Declare dead letter Exchange
@Bean("yExchange")
public DirectExchange yExchange() {
return new DirectExchange(Y_DEAD_LETTER_EXCHANGE);
}
//=================================Delay queue optimization=================================
//Common queue name
public static final String QUEUE_C = "QC";
//Declare normal Queue
@Bean("queueC")
public Queue queueC() {
Map<String,Object> map = new HashMap<>(3);
//Set up dead letter switch
map.put("x-dead-letter-exchange",Y_DEAD_LETTER_EXCHANGE);
//Set dead letter RoutingKey
map.put("x-dead-letter-routing-key","YD");
return QueueBuilder.durable(QUEUE_C).withArguments(map).build();
}
//Bind queue to switch
@Bean
public Binding queueCBindingX(@Qualifier("queueC") Queue queueC,
@Qualifier("xExchange") DirectExchange xExchange) {
return BindingBuilder.bind(queueC).to(xExchange).with("XC");
}
}
Here are the codes for producers and consumers.
package com.szh.rabbitmq.controller;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Date;
/**
*
*/
@Slf4j
@RestController
@RequestMapping(value = "/ttl")
public class SendController {
@Autowired
private RabbitTemplate rabbitTemplate;
@GetMapping(value = "/sendExpirationMsg/{message}/{ttlTime}")
public void sendMsg(@PathVariable String message,
@PathVariable String ttlTime) {
log.info("Current time:{}，Send a message with a duration of{}ms of TTL Message to queue QC: {}",new Date().toString(),ttlTime,message);
rabbitTemplate.convertAndSend("X","XC",message,msg -> {
msg.getMessageProperties().setExpiration(ttlTime);
return msg;
});
}
}
package com.szh.rabbitmq.listener;
import com.rabbitmq.client.Channel;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
import java.util.Date;
/**
*
*/
@Slf4j
@Component
public class DeadLetterQueueConsumer {
//receive messages
@RabbitListener(queues = "QD")
public void receiveD(Message message, Channel channel) throws Exception {
String msg = new String(message.getBody());
log.info("Current time:{}，Messages received from dead letter queue:{}",new Date().toString(),msg);
}
}
Finally, we start the test. From the results, we can see that we sent two messages to MQ. The TTL of one message is 20s and the TTL of the other message is 5s. Logically, it must be 5s. This message was first taken from the dead letter queue by the consumer, but the result seems not to give face. Both messages waited until 20s to become dead letter messages and then consumed by the consumer.
It seems that there is no problem, but it was mentioned at the beginning that if the TTL is set on the message attribute, the message may not "die" on time ", because RabbitMQ only checks whether the first message has expired. If it has expired, it will be sent to the dead letter queue. If the delay time of the first message is long and the delay time of the second message is short, the second message will not be executed first.
To solve the above problem, we need to use a plug-in rabbitmq_delayed_message_exchange-3.8.0.ez in RabbitMQ. I won't demonstrate this.