Catalogue of series articles
Message queue MQ (I) -- introduction, installation and use of management page of RabbitMQ
Message queue MQ (II) -- Spring Boot integration RabbitMQ
Message queue MQ (III) -- RabbitMQ advanced features and RabbitMQ cluster construction
preface
The last article made a simple application of RabbitMQ. Have we considered what to do if the producer sends it and the consumer doesn't receive it? There are many consumers, but I don't want to create message queues indefinitely. What should I do? In this article, we will talk about the possible problems of MQ in application and how to deal with them. In this way, the next time we "cook", we not only know how to do it, but also know how to remedy it when it is too salty.
Tip: the following is the main content of this article. The following cases can be used for reference
1, RabbitMQ advanced features
TCP/IP has three handshakes and four waves in order to know the accessibility of messages and the equality of messages on both sides. How to solve it in MQ?
Ready demo
1. Write MessageController to send messages to message queue:
/**
* Send message to message queue
*/
@RestController
public class MessageController {
//Inject RabbitMQ template object
@Autowired
private RabbitTemplate rabbitTemplate;
//Call the RabbitMQ template send message method to send the message content to the message queue
@RequestMapping("/direct/sendMsg")
public String sendMsg(String exchange,String routingkey,String
msg){
/**
* Send message to switch
* Parameter 1: exchange: switch
* Parameter 2: routingkey: routing key
* Parameter 3: msg: send message content
*/
rabbitTemplate.convertAndSend(exchange,routingkey,msg);
return "Delivered~";
}
}
2. yml configuration of producers and consumers:
# Message Queuing service address spring.rabbitmq.host=112.165.150.18 # Message queuing port spring.rabbitmq.port=5672 # Message queuing account spring.rabbitmq.username=guest # Message queuing account password spring.rabbitmq.password=guest # Set the virtual host. If not, the default is the root path (/) spring.rabbitmq.virtual-host=/guest
3. Write the Consumer service to receive messages from the message queue:
/**
* Message queue receive message listener
*/
@Component
@RabbitListener(queues = "order.A")
public class ConsumerQueueListener {
//After receiving the message, the listener calls this method to execute the business logic
@RabbitHandler
public void queueListenerHandle(String msg){
System.out.println("Order message{}，Content:"+msg);
}
}
4. For later tests:
http://localhost:8080/direct/sendMsg?exchange=order_ Exchange & routingkey = order. A & MSG = buy Apple phone
1. Producer confirmation
Question 1: the producer can send 100% messages to the message queue!
Two unexpected situations:
First, the consumer fails to send a message to MQ, and the message is lost;
Second, the switch fails to route to the queue, and the routing key is written incorrectly;
In order to ensure that my message is sent successfully, RabbitMQ introduces two methods to control the delivery reliability of messages: ① confirm confirmation mode; ② Return return mode
- product sends a message to the switch (Exchange). Whether it is successful or not, it will execute a method to confirm the callback, confirmCallback.
- If the delivery of exchange to the message queue fails, a return method returnCallback will be executed.
These two callback s are used to control the reliable delivery of messages
1.1 confirm mode
Enable the producer publishing message confirmation mode in the configuration file:
# Turn on the producer confirmation mode: (confirm), deliver to the switch, and call back whether it fails or succeeds
spring:
rabbitmq:
publisher-confirms = true
Write producer confirmation callback method:
//Send a message callback confirmation class, implement the callback interface ConfirmCallback, and override the confirm() method
@Component
public class MessageConfirmCallback implements
RabbitTemplate.ConfirmCallback {
/**
* After delivery to the switch, the method will be called back no matter whether the delivery is successful or failed
* @param correlationData Delivery of relevant data
* @param ack Post to switch
* @param cause Reason for delivery failure
*/
@Override
public void confirm(CorrelationData correlationData, boolean ack,
String cause) {
if (ack){
System.out.println("The message entered the switch successfully{}");
} else {
System.out.println("Message failed to enter switch{} ， Failure reason:" + cause);
}
}
}
In the RabbitTemplate, set the message publishing confirmation callback method
@Component
public class MessageConfirmCallback implements
RabbitTemplate.ConfirmCallback{
@Autowired
private RabbitTemplate rabbitTemplate;
/**
* After creating the RabbitTemplate object, execute the current method and set the callback confirmation method for the template object
* Set message confirmation callback method
* Set message fallback callback method
*/
@PostConstruct
public void initRabbitTemplate(){
//Set message confirmation callback method
rabbitTemplate.setConfirmCallback(this::confirm);
}
/**
* After delivery to the switch, the method will be called back no matter whether the delivery is successful or failed
* @param correlationData Delivery of relevant data
* @param ack Post to switch
* @param cause Reason for delivery failure
*/
@Override
public void confirm(CorrelationData correlationData, boolean ack,
String cause) {
if (ack){
System.out.println("The message entered the switch successfully{}");
} else {
System.out.println("Message failed to enter switch{} ， Failure reason:" + cause);
}
}
}
Request test:
- Test successful
http://localhost:8080/direct/sendMsg?exchange=order_ Exchange & routingkey = order. A & MSG = buy Apple phone
- Test failed
http://localhost:8080/direct/sendMsg?exchange=order_ XXXXXXXX & routingkey = order. A & MSG = buy Apple phone
1.2 return return mode
Characteristics of message fallback mode: when a message enters the switch and is routed to the queue, a callback method is executed if an exception occurs
- In the configuration file, enable the producer publishing message fallback mode
# Turn on the producer fallback mode: (returns), the switch routes messages to the queue, and calls back in case of exceptions spring.rabbitmq.publisher-returns=true
- In the MessageConfirmCallback class, implement the interface RabbitTemplate.ReturnCallback
@Component
public class RabbitConfirm implements RabbitTemplate.ConfirmCallback
,RabbitTemplate.ReturnCallback {
//Override the returnedMessage() method in the RabbitTemplate.ReturnCallback interface
/**
* When a message is delivered to the switch, the switch routes to the message queue. If an exception occurs, execute the returnedMessaged method
* @param message Post message content
* @param replyCode Return error status code
* @param replyText Return error content
* @param exchange Switch name
* @param routingKey Routing key
*/
@Override
public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
System.out.println("Error routing switch to message queue:>>>>>>>");
System.out.println("Switch:"+exchange);
System.out.println("Routing key:"+routingKey);
System.out.println("Error status code:"+replyCode);
System.out.println("Error reason:"+replyText);
System.out.println("Send message content:"+message.toString());
System.out.println("<<<<<<<<");
}
}
- In the RabbitTemplate, set the message publishing fallback callback method
@PostConstruct
public void initRabbitTemplate(){
//Set message confirmation callback method
rabbitTemplate.setConfirmCallback(this::confirm);
//Set message fallback callback method
rabbitTemplate.setReturnCallback(this::returnedMessage);
}
- Request test failed. Execute returnedMessage method:
http://localhost:8080/direct/sendMsg?exchange=order_ Exchange & routingkey = xxxxx & MSG = buy Apple phone
1.3 summary
Confirmation mode:
- Set publisher confirms = "true" to enable confirmation mode.
- Implement the RabbitTemplate.ConfirmCallback interface and override the confirm method
- Features: no matter whether the message is successfully delivered to the switch, the confirm method is called back. Only when the sending fails, the business code needs to be written for processing.
Return mode
- Set publisher returns = "true" to enable the return mode.
- Implement the RabbitTemplate.ReturnCallback interface and override the returnedMessage method
- Features: after a message enters the switch, the returnedMessage method is called back only when the route from exchange to queue fails;
2. Consumer confirmation
Question 2: consumers can receive 100% requests, and there can be no errors in the process of business execution!
ack refers to an Acknowledge, which has the meaning of confirmation. It is a confirmation mechanism for the consumer to receive a message;
1. Automatic confirmation: acknowledge = "none".
Once a message is received by the Consumer, it will automatically acknowledge receipt and remove the corresponding message from the message cache of RabbitMQ. However, in the actual business processing, it is likely that the message will be lost if an exception occurs in the business processing after the message is received.
2. Manual confirmation: acknowledge = "manual".
Manual confirmation method requires channel.basicAck() to be invoked manually after successful business processing. If there is an exception, the channel.basicNack() method is invoked to enable it to automatically send messages again.
3. Confirm according to the abnormal situation: acknowledge = "auto", (this method is troublesome and will not be explained).
2.1 consumer confirmation demo (manual confirmation)
Objective: customize the message listener for consumers to listen to the content of received messages and sign in manually; When the business system throws an exception, it refuses to sign in and returns to the queue
Configure yml
# Message Queuing service address spring.rabbitmq.host=112.165.150.18 # Message queuing port spring.rabbitmq.port=5672 # Message queuing account spring.rabbitmq.username=guest # Message queuing account password spring.rabbitmq.password=guest # Set the virtual host. If not, the default is the root path (/) spring.rabbitmq.virtual-host=/guest # Configure to enable manual sign in # Opening of simple mode and manual sign in spring.rabbitmq.listener.simple.acknowledge-mode=manual # Routing mode enable manual sign in spring.rabbitmq.listener.direct.acknowledge-mode=manual # Do you support retry spring.rabbitmq.listener.direct.retry.enabled=true
In the consumer project, create a custom listener class CustomAckConsumerListener
/**
* Customize the listener. After listening to the message, execute the onMessage method immediately
*/
@Component
@RabbitListener(queues = "order.A")
public class CustomAckConsumerListener {
/**
* Method executed after listening to the message
* @param message Message content
* @param channel Message channel
*/
@RabbitHandler
public void queueListenerHandle(String msg,Message message, Channel channel) throws Exception {
//Get message content
System.out.println("After receiving the message, execute the specific business logic{} Message content:"+msg);
//Get delivery label
MessageProperties messageProperties =
message.getMessageProperties();
long deliveryTag = messageProperties.getDeliveryTag();
try {
if (msg.contains("Apple")){
throw new RuntimeException("Apple phones are not allowed!!!");
}
/**
* Manually sign in messages
* Parameter 1: message delivery label
* Parameter 2: batch sign in: true sign in all at once, false, sign in only the current message
*/
channel.basicAck(deliveryTag,false);
System.out.println("Manual sign in completed:{}");
} catch (Exception ex){
/**
* Manually reject sign in
* Parameter 1: delivery label of current message
* Parameter 2: batch sign in: true sign in all at once, false, sign in only the current message
* Parameter 3: whether to return to the queue. true means to return to the queue, and false means not to return
*/
channel.basicNack(deliveryTag,false,true);
System.out.println("Reject sign in and return to the queue:{}"+ex);
}
}
}
Test successful
Test sending message, manual sign in, request address http://localhost:8080/direct/sendMsg?exchange=order_ Exchange & routingkey = order. A & MSG = buy Apple phone
Test failed
Reject the sign in message, the message returns to the queue, the request address contains apple, and an exception is thrown: http://localhost:8080/direct/sendMsg?exchange=order_ Exchange & routingkey = order. A & MSG = buy Apple phone
2.2 summary
- If you want to sign in messages manually, you need to customize the implementation of the message receiving listener and implement the ChannelAwareMessageListener interface
- Set the AcknowledgeMode mode
1.none: automatic
2.auto: abnormal mode
3.manual: Manual - Call the channel.basicAck method to sign the message
- Call the channel.basicNAck method to reject the message
3. Consumer end current limiting
Scenario 1: if relevant business functions need to be maintained in system A, the service of system A may need to be stopped. At this time, the message producer will always send messages to be processed in MQ. At this time, the consumer's service has been closed, resulting in A large number of messages accumulating in MQ. After system A is successfully started, consumers will pull A large number of messages accumulated in MQ to their own services at one time, resulting in the service processing A large number of businesses in A short time, which may lead to the collapse of system services. Therefore, it is very necessary to limit the current at the consumer end.
Scenario 2: when a large number of users register, high concurrency requests come, and the mail interface only supports a small amount of concurrency. At this time, it is also very necessary to limit the flow at the consumer end;
Consumer end current limiting configuration: set the listener container attribute container.setPrefetchCount(1); It means that the consumer will pull one message from mq to consume each time, and will not continue to pull the next message until the consumption is manually confirmed.
3.1 consumer end current limiting demo
Demonstrate the current limiting requirements of the consumer based on the project confirmed by the consumer, and the confirmation mode of the consumer must be manual confirmation
**Configure 1 message to be pulled each time**
# Message Queuing service address spring.rabbitmq.host=112.165.150.18 # Message queuing port spring.rabbitmq.port=5672 # Message queuing account spring.rabbitmq.username=guest # Message queuing account password spring.rabbitmq.password=guest # Set the virtual host. If not, the default is the root path (/) spring.rabbitmq.virtual-host=/guest # Configure to enable manual sign in # Opening of simple mode and manual sign in spring.rabbitmq.listener.simple.acknowledge-mode=manual # Routing mode enable manual sign in spring.rabbitmq.listener.direct.acknowledge-mode=manual # Do you support retry spring.rabbitmq.listener.direct.retry.enabled=true # Set the flow limit at the consumer end, and the number of messages pulled each time. The default is 250 spring.rabbitmq.listener.direct.prefetch=1
In the listener receiving message method, sleep for 3 seconds, otherwise the pull is too fast and the effect cannot be seen
/**
* Customize the listener. After listening to the message, execute the onMessage method immediately
*/
@Component
@RabbitListener(queues = "order.A")
public class CustomAckConsumerListener {
/**
* Method executed after listening to the message
* @param message Message content
* @param channel Message channel
*/
@RabbitHandler
public void queueListenerHandle(String msg,Message message, Channel channel) throws Exception {
//Get message content
System.out.println("After receiving the message, execute the specific business logic{} Message content:"+msg);
//Get delivery label
MessageProperties messageProperties = message.getMessageProperties();
long deliveryTag = messageProperties.getDeliveryTag();
try {
//Sleep for 3 seconds
Thread.sleep(3000);
if (msg.contains("Apple")){
throw new RuntimeException("Apple phones are not allowed!!!");
}
/**
* Manually sign in messages
* Parameter 1: message delivery label
* Parameter 2: batch sign in: true sign in all at once, false, sign in only the current message
*/
channel.basicAck(deliveryTag,false);
System.out.println("Manual sign in completed:{}");
} catch (Exception ex){
/**
* Manually reject sign in
* Parameter 1: delivery label of current message
* Parameter 2: batch sign in: true sign in all at once, false, sign in only the current message
* Parameter 3: whether to return to the queue. true means to return to the queue, and false means not to return
*/
channel.basicNack(deliveryTag,false,true);
System.out.println("Reject sign in and return to the queue:{}"+ex);
}
}
}
Restart consumer engineering and check whether the configuration of pulling 1 message each time in the control panel channel is effective
Send request multiple times to test
test http://localhost:8080/direct/sendMsg?exchange=order_ Exchange & routingkey = order. A & MSG = buy Apple phone
3.2 summary
- Set 1 spring.rabbitmq.listener.simple.prefetch=2 for each pull message
- Note that if you want to limit the flow at the consumer end, the message must be confirmed manually, and the acknowledge mode is MANUAL
4. TTL (message lifetime)
The full name of TTL is Time To Live. When the message reaches the survival time and has not been consumed, it will be cleared automatically
Except. RabbitMQ can set the expiration time for messages or for the entire Queue.
It makes no sense to set an expiration time for a single message
4.1 demo demo
-
In the RabbitMQ management console, add the message queue order.B and set the message expiration time to 5 seconds
-
In the RabbitMQ administrative console, bind the message queue order.B to the switch order_ On exchange
-
The test sends a message to the message queue order.B, which has no consumers receiving messages
-
Wait for 5 seconds and the message will disappear automatically
-
Set message expiration time
-
4.2 summary
- Set the usage parameter of queue expiration time: x-message-ttl, unit: ms, which will uniformly expire the messages of the whole queue.
- Because the queue is first in first out, it doesn't make sense to set the expiration time of a single message
For example, set the expiration time of message A to 10 seconds and the expiration time of message B to 5 seconds. However, if message A is sent to the queue first, message B will not be consumed or removed until message A is consumed or removed.
5. Dead letter queue
What should I do if the message is lost or failed to be sent? Let the message disappear?
Dead letter queue: when the message becomes Dead message, it can be re sent to another switch, which is Dead Letter Exchange (Dead Letter Exchange abbreviation: DLX).
There are three situations when a message becomes a dead letter:
- The queue message length reaches the limit;
- The consumer rejects the message (basicNack) and does not put the message back to the source queue, request = false;
- The message expiration setting exists in the source queue, and the message arrival timeout is not consumed;
Set dead letter queue binding dead letter switch:
Set parameters for the queue: x-dead-letter-exchange and x-dead-letter-routing-key
5.1 dead letter queue demo
In the RabbitMQ administrative console, create a dead letter queue deadQueue
In the RabbitMQ administrative console, create the dead letter switch deadExchange
The dead letter queue is bound to the dead letter switch, and the routing key is order.dead
Message queue order.B binding dead letter switch
Send a message to the message queue order.B [the message expiration time in the message queue order.B is 5 seconds]
Wait for 5 seconds, and the message in the message queue order.B enters the dead letter queue
5.2 summary
- Dead letter switch and dead letter queue are no different from ordinary ones
- When a message becomes a dead letter, if the queue is bound to a dead letter switch, the message will be rerouted to the dead letter queue by the dead letter switch
- There are three situations when a message becomes a dead letter:
The queue message length reaches the limit;
The consumer rejects the consumption message and does not return to the queue;
The message expiration setting exists in the original queue, and the message arrival timeout is not consumed;
6. Delay queue
What is a delay queue? That is, the message will not be consumed immediately after entering the queue, but only after reaching the specified time.
Delay queue application scenario
- If the order is not paid within 30 minutes after placing the order, the order shall be cancelled and the ticket shall be rolled back; 2. Send a text message to say hello to the new user after 7 days of successful registration.
Implementation methods: 1. Timer; 2. Delay queue
Note: the delay queue function is not provided in RabbitMQ. However, TTL + dead letter queue combination can be used to achieve the effect of delay queue.
2, RabbitMQ application problem - Message idempotency processing
Idempotency means that the results of one request or multiple requests initiated by the same operation are consistent. Just like when we buy a lollipop, we buy a 50 cents for the first time and 50 cents for the second time. We won't have different prices because of the number of times we buy it. That is, the impact of any multiple execution on the resource itself is the same as that of one execution.
In MQ, it refers to consuming multiple identical messages to get the same result as consuming the message once.
The idempotent operation principle of using optimistic locking mechanism to ensure messages:
Always assume the best situation. Every time you get the data, you think others will not modify it, so it will not be locked. However, when updating, you will judge whether others have updated the data during this period. You can use the version number mechanism and CAS algorithm. Optimistic locking is suitable for multi read applications, which can improve throughput.
First execution version: version=1
-- Query before update: id=1,money=4000,version=1 update account set money=money-500,version=version+1 where id=1 and version=1;
Second execution: version=2
-- Second execution: version=2，same SQL The statement cannot take effect update account set money=money-500,version=version+1 where id=1 and version=1;
The nth execution: version=2. Only the first execution is effective! Because version is 1
update account set money=money-500,version=version+1 where id=1 and version=1;
3, RabbitMQ cluster construction
Generally speaking, if it is only to learn RabbitMQ or verify the correctness of business engineering, it is OK to use its single instance deployment in the local environment or test environment. However, considering the reliability, concurrency, throughput and message stacking capacity of MQ middleware, RabbitMQ clustering scheme is generally considered in the production environment.
1. Cluster scheme principle
RabbitMQ, a message queuing middleware product, is written based on Erlang. Erlang language is naturally distributed (realized by synchronizing magic cookie s of each node of Erlang cluster). Therefore, RabbitMQ naturally supports clustering. Clustering is a way to ensure reliability. At the same time, it can increase message throughput through horizontal expansion. Here, only Erlang needs to be guaranteed_ If the parameters of the cookie are consistent, the cluster can communicate.
2. Cluster construction
Main reference official documents: https://www.rabbitmq.com/clustering.html
# docker run command interpretation docker run --link It can be used to link two containers, making the source container (the linked container) and the receiving container (actively unlink) Can communicate with each other. # -p map a port # -v mount data volume # --name sets an alias for the current container # -di start daemon container # -it launch interactive container # Command to execute after entering the container / bin/bash # -e set default parameters # --hostname sets the host name of the virtual machine in the current container # --link format: name:hostname # Name is the name of the source container; Hostname is the hostname of the source container.
2.1 start three RabbitMQ
# 1.1 start RabbitMQ1 docker run -d --hostname rabbitmq1 --name=m1 -p 15673:15672 -p 5673:5672 -e RABBITMQ_ERLANG_COOKIE='rabbitmqcookie' rabbitmq:management # -e injection parameters, RABBITMQ_ERLANG_COOKIE: erlang_cookie parameter, the nodes in the cluster must be consistent # 1.2 start RabbitMQ2 docker run -d --hostname rabbitmq2 --name=m2 -p 15674:15672 -p 5674:5672 -- link m1:rabbitmq1 -e RABBITMQ_ERLANG_COOKIE='rabbitmqcookie' rabbitmq:management # 1.3 start RabbitMQ3 docker run -d --hostname rabbitmq3 --name m3 -p 15675:15672 -p 5675:5672 -- link m2:rabbitmq2 --link m1:rabbitmq1 -e RABBITMQ_ERLANG_COOKIE='rabbitmqcookie' rabbitmq:management
2.2. Enter rabbitmq container m1 and reset rabbitmq service
- Stop rabbitmq service
- Reset rabbitmq service
- Start rabbitmq service
#Enter myrabbiratmq1 container docker exec -it m1 bash #Stop rabbit app rabbitmqctl stop_app #Reset rabbitmq rabbitmqctl reset #Start rabbit app rabbitmqctl start_app
2.3. Enter RabbitMQ container m2 and join the cluster: connect node 1rabbitmq service
- Stop rabbitmq service
- Reset rabbitmq service
- Join cluster: connect node 1rabbitmq services
- Start rabbitmq service
#3. Enter myrabbitmq2 container docker exec -it m2 bash #Stop rabbit app rabbitmqctl stop_app #Reset rabbitmq rabbitmqctl reset #Join cluster rabbitmqctl join_cluster --ram rabbit@rabbitmq1 ## --ram setting memory node #Start rabbit app rabbitmqctl start_app
2.4. Enter the RabbitMQ container m3 and join the cluster: connect the node 1rabbitmq service
- Stop rabbitmq service
- Reset rabbitmq service
- Join cluster: connect node 1rabbitmq services
- Start rabbitmq service
#4. Enter myrabbitmq3 container docker exec -it m3 bash #Stop rabbit app rabbitmqctl stop_app #Reset rabbitmq rabbitmqctl reset #Join the cluster hard disk node rabbitmqctl join_cluster rabbit@rabbitmq1 #Start rabbit app rabbitmqctl start_app
2.5. View cluster status after startup
#View cluster status rabbitmqctl cluster_status
3. Cluster problems
The RabbitMQ cluster mode configured above does not guarantee the high availability of the queue. Although the switch binds the contents of the queue and can be copied to any node in the cluster, the contents of the queue will not be copied. The downtime of the queue node directly causes that the queue cannot be applied and can only wait for the node to restart. Therefore, if the queue node is down or fails, it can be applied normally, To copy the contents of the queue to each node in the cluster, you need to create an image queue.
The image queue can synchronize the queue and message. When the primary queue dies, one of the secondary queues will become the primary queue to take over the work.
The image queue is based on the normal cluster mode, so you still have to configure the normal cluster before setting the image queue. After setting the image queue, it will be divided into one master node and multiple slave nodes. If the master node goes down, one of the slave nodes will be selected as the master node, and the original master node will become a slave node after it rises.
#The set image queue command can be executed on any node
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
Then stop the master node and test whether it can send and receive messages normally
