Get to know MQ
What is MQ
MQ(message queue), in its literal sense, is essentially a queue. FIFO is first in, first out, but the contents stored in the queue are messages. It is also a cross process communication mechanism for upstream and downstream message delivery. In the Internet architecture, MQ is a very common upstream and downstream "logical decoupling + physical decoupling" message communication service. After using MQ, the upstream of message sending only needs to rely on MQ without relying on other services.
Why MQ
1. Flow peak elimination
For example, if the order system can process 10000 orders at most, this processing capacity is more than enough to deal with orders in normal periods
In the normal period, we can return the results one second after placing an order. However, in the peak period, if 20000 orders are placed, the operating system can't handle them and can only limit them
Users are not allowed to place orders after the number of orders exceeds 10000. Using message queue as buffer, we can cancel this limit and divide the orders placed in one second
Scattered for a period of time. At this time, some users may not receive the successful operation of placing an order until more than ten seconds after placing an order, but it is more difficult than those who cannot place an order
The test is better.
2. Application decoupling
Taking e-commerce applications as an example, there are order system, inventory system, logistics system and payment system. After the user creates the order, if the coupling
Call the inventory system, logistics system and payment system. If any subsystem fails, the order operation will be abnormal. When transformed into based
After using message queuing, the problems of inter system calls will be reduced a lot. For example, the logistics system needs a few minutes to repair due to failure. stay
In these few minutes, the memory to be processed by the logistics system is cached in the message queue, and the user's order can be completed normally. When logistics
After the system is restored, you can continue to process the order information. Medium order users can't feel the failure of the logistics system, so as to improve the availability of the system.
3. Asynchronous processing
Some inter service calls are asynchronous. For example, a calls B, and B takes a long time to execute, but a needs to know when B can complete execution. In the past, there were generally two ways. A calls B's query api for a period of time. Or A provides a callback api. After B is executed, api is called to notify the A service. Neither of these two methods is very elegant. Using the message bus can easily solve this problem. After a calls the B service, a only needs to listen to the message processed by B. when B processing is completed, it will send a message to MQ, which will forward the message to the a service. In this way, service a does not need to call B's query api or provide callback api. Similarly, service B does not need to do these operations. Service a can also get the message of successful asynchronous processing in time.
Classification of MQ
1.ActiveMQ
Advantages: single machine throughput of 10000 class, timeliness of ms class, high availability, high availability based on master-slave architecture, low message reliability and low probability of data loss
Disadvantages: the official community now maintains ActiveMQ 5.x less and less, and uses less in high-throughput scenarios.
2.Kafka
Kafka, a message middleware for big data, is the killer of big data. When it comes to message transmission in the field of big data, Kafka cannot be bypassed,
It is famous for its million level TPS throughput, and quickly becomes a favorite in the field of big data. It plays an important role in the process of data acquisition, transmission and storage
Plays an important role. It has been adopted by LinkedIn, Uber, Twitter, Netflix and other large companies.
Advantages: excellent performance, single machine write TPS is about one million pieces / s, and the biggest advantage is high throughput. Timeliness ms level availability non
Chang Gao, kafka is distributed. One data has multiple copies, and a few machines go down. It will not lose data and lead to unavailability. Consumers will adopt
The message is obtained by Pull, and the message is orderly. Through control, it can ensure that all messages are consumed and only consumed once; Excellent third party
Kafka Web management interface Kafka manager; It is mature in the log field and is used by many companies and open source projects; Functional support:
The function is relatively simple. It mainly supports simple MQ functions. Real time calculation and log collection in the field of big data are used on a large scale
Disadvantages: the Kafka single machine has more than 64 queues / partitions, and the load will soar obviously. The more queues, the higher the load, and the transmission will be cancelled
The message response time becomes longer, and the short polling method is used. The real-time performance depends on the polling interval. Consumption failure does not support retry; Support message sequence,
However, when an agent goes down, it will cause message disorder and slow community update;
3.RocketMQ
RocketMQ is an open-source product from Alibaba and implemented in Java language. It refers to Kafka and makes its own design
Some improvements. It is widely used by Alibaba in order, transaction, recharge, stream computing, message push, log stream processing, binglog distribution, etc
View.
Advantages: single machine throughput is 100000, availability is very high, distributed architecture, message loss can be achieved, MQ function is relatively perfect, or points
It is distributed and has good scalability. It supports 1 billion level message accumulation and will not cause performance degradation due to accumulation. The source code is java and we can read it ourselves
Read the source code and customize your company's MQ
Disadvantages: there are not many supported client languages. At present, java and c + +, of which c + + is immature; Community activity is average, not in MQ
To implement JMS and other interfaces in the core, some systems need to modify a lot of code to migrate
4.RabbitMQ
Released in 2007, it is a reusable enterprise message system based on AMQP (Advanced message queuing protocol), which is the most popular at present
One of the mainstream message oriented middleware.
Advantages: due to the high concurrency of erlang language, the performance is good; With a throughput of 10000, MQ functions are relatively complete, robust, stable and easy to use
Use, cross platform and support multiple languages, such as Python, Ruby,. NET, Java, JMS, C, PHP, ActionScript, XMPP and STOMP
Etc., supporting AJAX with complete documents; The management interface provided by open source is very good, easy to use, and the community is highly active; The update frequency is quite high
Disadvantages: the commercial version needs to be charged, and the learning cost is high
MQ selection
1.Kafka
The main feature of Kafka is that it processes message consumption based on Pull mode and pursues high throughput. Its initial purpose is to collect logs
And transmission, which is suitable for the data collection business of Internet services that generate a large amount of data. Large companies suggest that it can be selected. If there is log collection function,
Must be the first choice kafka.
2.RocketMQ
Born in the field of financial Internet and demanding high reliability, especially the order deduction and business cutting in e-commerce
Peak, when a large number of transactions pour in, the back end may not be able to handle it in time. RoketMQ may be more reliable in terms of stability. These services
The scenario has been tested many times in Alibaba double 11. If your business has the above concurrent scenarios, it is recommended to choose RocketMQ.
3.RabbitMQ
Combined with the concurrency advantages of erlang language, it has good performance, timeliness, microsecond level, high community activity, and the management interface is very easy to use
Convenient. If your data volume is not so large, small and medium-sized companies give priority to RabbitMQ with complete functions
RabbitMQ
RabbitMQ concept
RabbitMQ is a message oriented middleware: it accepts and forwards messages. You can use it as a courier site when you want to send a package
When wrapping, you put your package in the express station, and the courier will eventually send your express to the recipient. According to this logic, RabbitMQ is
A courier station, a courier to help you deliver express. The main difference between RabbitMQ and express station is that it does not process express mail, but receives it,
Store and forward message data
Four core concepts
producer
The program that generates the data sending message is the producer
Switch
Switch is a very important part of RabbitMQ. On the one hand, it receives messages from producers, on the other hand, it sends messages
Push to queue. The switch must know exactly how to handle the messages it receives, whether to push them to a specific queue or push them
Sending to multiple queues or discarding messages depends on the switch type
queue
Queue is a data structure used internally by RabbitMQ. Although messages flow through RabbitMQ and applications, they can only be stored
Stored in the queue. The queue is only constrained by the memory and disk limitations of the host. It is essentially a large message buffer. Many producers can
To send messages to a queue, many consumers can try to receive data from a queue. This is how we use queues
consumer
Consumption and reception have similar meanings. Most of the time, consumers are a program waiting to receive messages. Please note that producers, consumers
Most of the time, the and message oriented middleware are not on the same machine. The same application can be both a producer and a consumer.
RabbitMQ core
Six core modes: simple mode, work queue mode, publish subscribe mode, routing mode, topic mode and publish confirmation mode
Introduction to each noun
- Broker: an application that receives and distributes messages. RabbitMQ Server is the Message Broker
- Virtual host: designed for multi tenancy and security factors, the basic components of AMQP are divided into a virtual group, which is similar to the concept of namespace in the network. When multiple different users use the services provided by the same RabbitMQ server, they can be divided into multiple vhosts. Each user creates an exchange / queue in its own vhost
- Connection: TCP connection channel between publisher / consumer and broker: if a connection is established every time RabbitMQ is accessed, the overhead of establishing TCP connection will be huge and the efficiency will be low when the message volume is large. A channel is a logical connection established within a connection. If the application supports multithreading, each thread usually creates a separate channel for communication. The AMQP method contains a channel id to help the client and message broker identify the channel, so the channels are completely separated. As a lightweight connection, channel greatly reduces the overhead of establishing TCP connection by the operating system
- Exchange: message arrives at the first stop of the broker, matches the routing key in the query table according to the distribution rules, and distributes the message to the queue. Common types are: direct (point-to-point), topic (publish subscribe) and fan out (multicast)
- Queue: the message is finally sent here to wait for the consumer to pick it up
- Binding: the virtual connection between exchange and queue. The binding can contain routing key s. The binding information is saved in the query table in exchange for the distribution basis of message s
RabbitMQ installation
1. Official website address
https://www.rabbitmq.com/download.html
2. File upload
Upload to / usr/local/RabbitMQ /
3. Installation documents (installed in the following order)
rpm -ivh erlang-21.3-1.el7.x86_64.rpm yum install socat -y rpm -ivh rabbitmq-server-3.8.8-1.el7.noarch.rpm
4. Common commands (executed in the following order)
Add boot start RabbitMQ service
chkconfig rabbitmq-server on
Start service
/sbin/service rabbitmq-server start
View service status
/sbin/service rabbitmq-server status
Stop service (select execute)
/sbin/service rabbitmq-server stop
Start the web management plug-in. Remember to stop the RbMQ service before starting it
rabbitmq-plugins enable rabbitmq_management
After the plug-in installation is completed, start RbMQ again
/sbin/service rabbitmq-server start
Restart the rabbitmq service, and then enter 192.168.163.128(Linux ip address): 15672 on the windows client. You need to open the port number and enter smoothly!
# Turn on the firewall firewall-cmd --permanent --add-port=15672/tcp # Restart effective firewall-cmd --reload
Use the default account password (guest) to access the address http://192.168.112.128:15672/ There is a permission problem
4. Add a new user
Create account, user name and password
rabbitmqctl add_user admin 123
Set user role, super administrator
rabbitmqctl set_user_tags admin administrator
Set user permissions
set_permissions [-p <vhostpath>] <user> <conf> <write> <read> rabbitmqctl set_permissions -p "/" admin ".*" ".*" ".*"
User user_admin has the configuration, write and read permissions of all resources in the virtual host / vhost1
View current users and roles
rabbitmqctl list_users
5. Log in again with the admin user
6. Reset command
The command to close the application is
rabbitmqctl stop_app
The command to clear is
rabbitmqctl reset
The restart command is
rabbitmqctl start_app
Simple queue
We will write two programs in Java. The producer who sends a single message and the consumer who receives and prints the message. We will cover some details in the Java API.
In the figure below, "P" is our producer and "C" is our consumer. The middle box is a queue - RabbitMQ represents the message buffer reserved by the consumer
POM file dependency
<!--appoint jdk Compiled version-->
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<!--rabbitmq Dependent client-->
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>5.8.0</version>
</dependency>
<!--A dependency on the operation file stream-->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
</dependencies>
Producer code
//Producer, send message
public class Producer
{
//Queue name
public static final String QUEUE_NAME= "hello";
//Send a message
public static void main(String[] args) throws IOException, TimeoutException {
//Create a connection factory
ConnectionFactory factory=new ConnectionFactory();
//The factory connects to the queue of RabbitMQ through the IP of RqbbitMQ
factory.setHost("192.168.112.128");
//Set user name
factory.setUsername("admin");
//Set password
factory.setPassword("123");
//Create connection
Connection connection = factory.newConnection();
//Acquire channel
Channel channel = connection.createChannel();
//Generate a queue
//Parameter 1: queue name
//Parameter 2: whether the messages in the queue are persistent. By default, the messages are stored in memory
//Parameter 3: whether the queue is only used by one message maker, that is, whether to share messages. true can be used by multiple consumers, and false can only be used by one consumer
//Parameter 4: automatically delete. After the last consumer disconnects, whether the queue is automatically deleted. true: automatically delete false: not automatically delete
//Parameter none: other parameters (delayed message...)
channel.queueDeclare(QUEUE_NAME,false,false,false,null);
//Send a message
String message="hello world";
//Send a message
//Parameter 1: to which switch
//Parameter 2: what is the key value of the route? This is the name of the queue
//Parameter 3: other parameter information
//Parameter 4: message body of sending message
channel.basicPublish("",QUEUE_NAME,null,message.getBytes(StandardCharsets.UTF_8));
System.out.println("Message sent");
}
}
Note: if connection error is reported, it is considered that the port number is not open. For the connection service, the requested port number is 5672, while for the visualizer service, the requested port number is 15672. Therefore, you need to open two ports 5672 and 15672. The test connection is successful!
Consumer code
//Consumer, receive message
public class Consumer
{
//The name of the queue
public static final String QUEUE_NAME="hello";
//receive messages
public static void main(String[] args) throws IOException, TimeoutException {
//Create a connection factory and return the newly created connection
Connection connection = getConnection();
//Create channel
Channel channel = connection.createChannel();
//Consumer News
//Parameter 1: which queue to consume
//Parameter 2: whether to respond automatically after successful consumption. true means automatic response, false means manual response
//Parameter 3: callback during message delivery
//Parameter 4: callback of consumer canceling consumption
channel.basicConsume(QUEUE_NAME,true,
//deliverCallback
(consumerTag,message)->{
//The message itself has a message header, message attributes and message body. Here we only need to get the message body
System.out.println(new String(message.getBody()));
},
//cancelCallback
consumerTag->{
System.out.println("Message interrupted");
});
}
private static Connection getConnection() throws IOException, TimeoutException {
//Create connection factory
ConnectionFactory factory=new ConnectionFactory();
factory.setHost("192.168.112.128");
factory.setPassword("123");
factory.setUsername("admin");
return factory.newConnection();
}
}
Work queue
The main idea of work queue (also known as task queue) is to avoid executing resource intensive tasks immediately and having to wait for them to complete. Instead, we arranged for the task to be carried out later. We encapsulate the task as a message and send it to the queue. The worker process running in the background will pop up the task and finally execute the job. When there are multiple worker threads, these worker threads will handle these tasks together.
Rotation training distribution message
In this case, we will start two worker threads, a message sending thread. Let's see how their two worker threads work.
Extraction tool class
/*
* This class is a tool class for creating channels for connection factories
* */
public enum RabbitMqUtils
{
//ctrl+shift+u becomes uppercase
INSTANCE;
public static RabbitMqUtils getInstance()
{
return INSTANCE;
}
// Get a connected channel
public Channel getChannel() throws IOException, TimeoutException {
// Create a connection factory
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("192.168.112.128");
factory.setUsername("admin");
factory.setPassword("123");
Connection connection = factory.newConnection();
com.rabbitmq.client.Channel channel = connection.createChannel();
return channel;
}
}
Start two worker threads
public class Worker01 {
// Queue name
public static final String QUEUE_NAME = "hello";
// Accept message
public static void main(String[] args) throws IOException, TimeoutException {
Channel channel = RabbitMqUtils.getInstance().getChannel();
// Accept message parameters
DeliverCallback deliverCallback = (consumerTag, message) -> {
System.out.println("Received message:"+new String(message.getBody()));
};
// Cancel consumption parameters
CancelCallback cancelCallback = consumerTag -> {
System.out.println(consumerTag+"Consumer cancels the callback logic of consumption excuse");
};
System.out.println("C2 Waiting to receive message");
// Message acceptance
channel.basicConsume(QUEUE_NAME,true,deliverCallback,cancelCallback);
}
}
Start 2 worker threads at the same time:
Producer code
public class Task01 {
// Queue name
public static final String QUEUE_NAME = "hello";
// Send a large number of messages
public static void main(String[] args) throws Exception
{
try (Channel channel = RabbitMqUtils.getInstance().getChannel())
{
// Declaration of queue
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
// Enter a message from the console
Scanner scanner = new Scanner(System.in);
while (scanner.hasNext())
{
String message = scanner.next();
channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
System.out.println("Send message complete:" + message);
}
}
}
}
Message response
concept
- It may take some time for a consumer to complete a task. What happens if one of the consumers processes a long task and only completes part of it, and suddenly it hangs up. Once RabbitMQ delivers a message to the consumer, it immediately marks the message for deletion. In this case, a consumer suddenly hangs up, and we will lose the message being processed. And subsequent messages sent to the consumer because it cannot be received.
- In order to ensure that messages are not lost during sending, RabbitMQ introduces a message response mechanism. Message response is: after receiving and processing the message, the consumer tells RabbitMQ that it has been processed, and RabbitMQ can delete the message.
Automatic response
The message is considered to have been successfully transmitted immediately after it is sent. This mode requires a trade-off between high throughput and data transmission security, because in this mode, if the connection or channel is closed on the consumer's side before the message is received, the message will be lost. On the other hand, of course, this mode can deliver overloaded messages on the consumer's side, There is no limit on the number of messages delivered. Of course, this may cause consumers to receive too many messages that are too late to process, resulting in the backlog of these messages, eventually running out of memory, and finally these consumer threads are killed by the operating system, Therefore, this model is only applicable when consumers can process these messages efficiently and at a certain rate.
Manual answer
Method of message response
- Channel.basicack (for positive confirmation)
RabbitMQ knows the message and processes it successfully. It can be discarded
- Channel.basicnack (for negative confirmation)
- Channel.basicreject (used for negative confirmation) is one less parameter than Channel.basicNack. It does not process the message. It is rejected directly and can be discarded.
Multiple explanation
The advantage of manual response is that it can respond in batches and reduce network congestion
true and false of multiple have different meanings:
- true indicates that unresponsive messages on the channel are answered in batch. For example, there are messages 5, 6, 7 and 8 transmitting tags on the channel. If the current tag is 8, these unresponsive messages of 5-8 will be confirmed to receive the message response
- false compared with the above, only the messages with tag=8 will be answered, and the three messages 5, 6 and 7 will not be confirmed to receive the message response
Note: a channel can carry more than one message at the same time
Message auto rejoin
If the consumer loses the connection for some reason (its channel has been closed, the connection has been closed or the TCP connection has been lost), resulting in the message not sending ACK confirmation, RabbitMQ will understand that the message has not been fully processed and will queue it again. If other consumers can handle it at this time, it will soon redistribute it to another consumer. In this way, even if a consumer dies occasionally, it can be ensured that no message will be lost.
Message manual answer
1 = = > producer:
//Messages are not lost when answering manually
//If an exception occurs at the consumer, the message is put back into the queue for re consumption
public class Producer
{
// Queue name
public static final String TASK_QUEUE_NAME = "ack_queue";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
// Declaration queue
channel.queueDeclare(TASK_QUEUE_NAME,false,false,false,null);
Scanner scanner = new Scanner(System.in);
while (scanner.hasNext()){
String message = scanner.next();
//Use default switch
channel.basicPublish("",TASK_QUEUE_NAME,null,message.getBytes(StandardCharsets.UTF_8));
System.out.println("Producer sends message:"+message);
}
}
}
2 = = = > consumers 1 and 2
public class ConsumerOne
{
// Queue name
public static final String TASK_QUEUE_NAME = "ack_queue";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
System.out.println("C1 Short waiting time for acceptance message processing");
DeliverCallback deliverCallback = (consumerTag, message) -> {
// Sleep for a second
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("The message received is:"+new String(message.getBody()));
//Conduct manual response
/*
* Parameter 1: tag of message
* Parameter 2: batch response; false: no batch response; true: batch response
* */
channel.basicAck(message.getEnvelope().getDeliveryTag(),false);
};
// Manual response is adopted
boolean autoAck = false;
channel.basicConsume(TASK_QUEUE_NAME,autoAck,deliverCallback,(consumerTag) -> {
System.out.println(consumerTag+"Consumer cancels consumer interface callback logic");
});
}
}
public class ConsumerTwo
{
// Queue name
public static final String TASK_QUEUE_NAME = "ack_queue";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
System.out.println("C2 Long waiting time for receiving message processing");
DeliverCallback deliverCallback = (consumerTag, message) -> {
// Sleep for a second
try {
TimeUnit.SECONDS.sleep(30);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("The message received is:"+new String(message.getBody()));
//Conduct manual response
/*
* Parameter 1: tag of message
* Parameter 2: batch response; false: no batch response; true: batch response
* */
channel.basicAck(message.getEnvelope().getDeliveryTag(),false);
};
// Manual response is adopted
boolean autoAck = false;
channel.basicConsume(TASK_QUEUE_NAME,autoAck,deliverCallback,(consumerTag) -> {
System.out.println(consumerTag+"Consumer cancels consumer interface callback logic");
});
}
}
3. Test
Consumer 1 receives the message after 1 second
Consumer 2 receives the message after 30 seconds
Because the processing time of ConsumerTwo is too long, we hang it up. At this time, the information will be returned to the queue and distributed to Worker03 for processing.
Stop consumer 2 while processing the message
The message is put back on the queue and consumed by consumer 1
Persistence
concept
We have just seen how to handle the situation that tasks are not lost, but how to ensure that the messages sent by the message producer are not lost after the RabbitMQ service is stopped. By default, when RabbitMQ exits or crashes for some reason, it ignores queues and messages unless told not to do so. Two things need to be done to ensure that messages are not lost: we need to mark both queues and messages as persistent.
Queue persistence
- The queues we created before are non persistent. rabbitmq if restarted, the queue will be deleted. If the queue is to be persistent, you need to set the durable parameter to persistent when declaring the queue
// Declaration queue // Persistence requires that the Queue be persistent boolean durable = true; channel.queueDeclare(TASK_QUEUE_NAME,durable,false,false,null);
- Note that if the previously declared queue is not persistent, you need to delete the original queue or re create a persistent queue, otherwise an error will occur
- After rerunning the code, the queue persistence succeeds
Message persistence
- To make the message persistent, you need to modify the code in the message producer, messageproperties, persistent_ TEXT_ Plan adds this attribute.
- Marking messages as persistent does not completely guarantee that messages will not be lost. Although it tells RabbitMQ to save the message to disk, there is still an interval point where the message is still cached when it is just ready to be stored on disk. There is no real write to disk at this time. The persistence guarantee is not strong, but it is more than enough for our simple task queue. If you need a stronger persistence strategy, refer to the release confirmation section below.
//Set the message sent by the producer as a persistent message (it is required to be saved to disk)
channel.basicPublish("",TASK_QUEUE_NAME, MessageProperties.PERSISTENT_TEXT_PLAIN
,message.getBytes(StandardCharsets.UTF_8));
System.out.println("Producer sends message:"+message);
Unfair distribution
- At the beginning, we learned that RabbitMQ uses rotation training to distribute messages, but in a certain scenario, this strategy is not very good. For example, there are two consumers processing tasks. One consumer 1 processes tasks very fast, while the other consumer 2 processes tasks very slowly, At this time, we still use the rotation training distribution method. It will come to the conclusion that the consumer with fast processing speed is idle for a large part of the time, while the consumer with slow processing is working all the time. This distribution method is actually not very good in this case, but RabbitMQ does not know this situation, and it still distributes fairly.
- To avoid this, we can set the parameter channel.basicQos(1)
// Set unfair distribution int prefetchCount = 1; channel.basicQos(prefetchCount);
Implementation principle of unfair distribution
This means that if I haven't finished processing this task or I haven't answered you, don't assign it to me. I can only process one task at present, and then rabbitmq will assign the task to the idle consumer who is not so busy. Of course, if all consumers haven't finished their tasks, the queue is still adding new tasks, The queue may be full. At this time, you can only add a new worker or change the strategy of other storage tasks.
The first time the message is distributed to which consumer, it depends on which consumer starts first, and then it will be recorded in the consumer list of rbMQ
channel.basicQos(n)； RabbitMQ will send N messages to consumers for the first distribution. After the consumer processes a message and confirms a message, RabbitMQ will reduce the corresponding count by 1, and then the consumer can continue to receive messages until the upper count limit is reached again
1. Turn off automatic response;
2. Change to manual feedback of the completed message;
2. Limit the number of channel.basicQos(n) per occurrence;
Phenomenon: those who can do more work can get more messages for processing
Here is a description of channel.basicQos(n);
For example, the consumer program calls channel.basicQos(5), and then subscribes to a queue for consumption. RabbitM will save a list of consumers. Each message sent will count the corresponding consumers. After the count reaches 5, RabbitMQ will not send messages to this consumer. After the consumer confirms that a message is processed, RabbitMQ reduces the corresponding count by 1, and the consumer can continue to receive the message until the upper limit of the count is reached again. This mechanism can be similar to the "sliding window" in TCP IP
Pre value
The message itself is sent asynchronously, so there must be more than one message on the channel at any time. In addition, the manual confirmation from the consumer is also asynchronous in nature.
Therefore, there is an unacknowledged message buffer here. Therefore, we hope that developers can limit the size of this buffer to avoid the problem of unlimited unacknowledged messages in the buffer.
This can be done by setting the "prefetch count" value using the basic.gos. Method.
This value defines the maximum number of unacknowledged messages allowed on the channel.
Once the number reaches the configured number, RabbitMQ will stop delivering more messages on the channel unless at least one unprocessed message is acknowledged,
For example, suppose there are unacknowledged messages 5, 6, 7 and 8 on the channel, and the prefetch count of the channel is set to 4, RabbitMQ. Will not deliver any messages on the channel unless at least one unacknowledged message is acked.
For example, the message tag=6 has just been acknowledged, and RabbitMQ will sense the situation and send another message.
Message response and QoS pre value have a significant impact on user throughput.
Generally, increasing prefetching will improve the speed of message delivery to consumers.
Although the automatic response transmission message rate is the best, in this case, the number of messages delivered but not processed will also increase, thus increasing the RAM consumption of consumers (random access memory). Care should be taken to use the automatic confirmation mode or manual confirmation mode with infinite preprocessing. Consumers consume a large number of messages if there is no confirmation, It will cause the memory consumption of consumers' connection nodes to become larger, so finding the appropriate pre value is a process of repeated experiments. The value is also different for different loads. The value in the range of 100 to 300 can usually provide the best throughput and will not bring too much risk to consumers.
The pre value of 1 is the most conservative. Of course, this will make the throughput very low, especially when the consumer connection delay is very serious, especially in the environment where the consumer connection waiting time is long.
For most applications, a slightly higher value will be optimal.
Release confirmation
Release confirmation principle
The producer sets the channel to confirm mode. Once the channel enters the confirm mode, all messages published on the channel will be assigned a unique ID (starting from 1). Once the message is delivered to all matching queues, the broker will send a confirmation to the producer (including the unique ID of the message), This allows the producer to know that the message has arrived at the destination queue correctly.
If the message and queue are persistent, the confirmation message will be sent after the message is written to the disk. The delivery tag field of the confirmation message returned by the broker to the producer contains the serial number of the confirmation message. In addition, the broker can also set the multiple field of basic.ack to indicate that all messages before this serial number have been processed.
The biggest advantage of confirm mode is that it is asynchronous. Once a message is published, the producer application can continue to send the next message while waiting for the channel to return the confirmation. After the message is finally confirmed, the producer application can process the confirmation message through the callback method. If RabbitMQ loses the message due to its own internal error, A nack message is sent, and the producer application can also process the nack message in the callback method
Release confirmation policy
How to open release confirmation:
Publishing confirmation is not enabled by default. If you want to enable it, you need to call the method confirmSelect. Whenever you want to use publishing confirmation, you need to call this method in channel
Channel channel = connection.createChannel(); channel.confirmSelect();
Single release confirmation
- This is a simple confirmation method. It is a synchronous confirmation publishing method, that is, after publishing a message, only it is confirmed to be published, and subsequent messages can continue to be published. The waitForConfirmsOrDie(long) method returns only when the message is confirmed. If the message is not confirmed within the specified time range, it will throw an exception.
- The biggest disadvantage of this confirmation method is that the publishing speed is particularly slow, because if the published message address is not confirmed, the publishing of all subsequent messages will be blocked. This method provides a throughput of no more than hundreds of published messages per second. Of course, this may be sufficient for some applications.
/*
* Release confirmation mode,
* 1,Single confirmation
* 2,Batch confirmation
* 3,Asynchronous batch confirmation
* */
public class ComfirmMessage {
// Number of batch messages
public static final int MESSAGE_COUNT = 1000;
public static void main(String[] args) throws Exception {
// 1. Single confirmation
// 1000 individual confirmation messages were issued, taking 567ms
ComfirmMessage.publishMessageIndividually();
}
public static void publishMessageIndividually() throws Exception {
Channel channel = RabbitMqUtils.getInstance().getChannel();
String queueName = UUID.randomUUID().toString();
//Queues in this channel are not persistent, message sharing, or automatically deleted
channel.queueDeclare(queueName,false,false,false,null);
// Open release confirmation
channel.confirmSelect();
// start time
long begin = System.currentTimeMillis();
// Batch message sending
for (int i = 0; i < MESSAGE_COUNT; i++) {
String message = i + "";
//No message persistence
channel.basicPublish("",queueName,null,message.getBytes(StandardCharsets.UTF_8));
// Release confirmation of a single message immediately
boolean flag = channel.waitForConfirms();
if (flag){
System.out.println("Message sent successfully");
}
}
// End time
long end = System.currentTimeMillis();
System.out.println("release"+MESSAGE_COUNT+"Separate confirmation messages, time consuming"+ (end - begin) + "ms");
}
}
Batch confirmation release
The above method is very slow. Compared with a single message waiting for confirmation, publishing a batch of messages first and then confirming together can greatly improve the throughput. Of course, the disadvantage of this method is that when a failure causes a problem in publishing, we don't know which message has a problem. We must save the whole batch in memory, To record important information and then republish the message. Of course, this scheme is still synchronous and blocks the release of messages.
/*
* Release confirmation mode,
* 1,Single confirmation
* 2,Batch confirmation
* 3,Asynchronous batch confirmation
* */
public class ComfirmMessage {
// Number of batch messages
public static final int MESSAGE_COUNT = 1000;
public static void main(String[] args) throws Exception {
//2. Batch confirmation
// It takes 37ms to publish 1000 batch confirmation messages
ComfirmMessage.publishMessageBatch();
}
public static void publishMessageBatch() throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
String queueName = UUID.randomUUID().toString();
channel.queueDeclare(queueName,false,false,false,null);
// Open release confirmation
channel.confirmSelect();
// start time
long begin = System.currentTimeMillis();
// Bulk confirmation message size
int batchSize = 1000;
// Batch send batch confirmation
for (int i = 0; i < MESSAGE_COUNT; i++) {
String message = i + "";
channel.basicPublish("",queueName,null,message.getBytes(StandardCharsets.UTF_8));
// When 100 messages are judged, batch confirmation is performed once
if (i%batchSize == 0){
// Confirm release
channel.waitForConfirms();
}
}
// End time
long end = System.currentTimeMillis();
System.out.println("release"+MESSAGE_COUNT+"Batch confirmation messages, time consuming"+ (end - begin) + "ms");
}
}
Asynchronous publish confirmation
Although the programming logic of asynchronous confirmation is more complex than the above two, it has the highest cost performance. It can not be said whether it is reliable or efficient. It uses callback function to achieve reliable message delivery. This middleware also ensures whether it is delivered successfully through function callback. Let's explain in detail how asynchronous confirmation is realized.
/*
* Release confirmation mode,
* 1,Single confirmation
* 2,Batch confirmation
* 3,Asynchronous batch confirmation
* */
public class ComfirmMessage {
// Number of batch messages
public static final int MESSAGE_COUNT = 1000;
public static void main(String[] args) throws Exception {
//3. Asynchronous batch confirmation
// Publish 1000 asynchronous confirmation messages, taking 36ms
ComfirmMessage.publicMessageAsync();
}
public static void publicMessageAsync() throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
String queueName = UUID.randomUUID().toString();
channel.queueDeclare(queueName,false,false,false,null);
// Open release confirmation
channel.confirmSelect();
// start time
long begin = System.currentTimeMillis();
// Message confirmation success callback function
ConfirmCallback ackCallback = (deliveryTag,multiply) -> {
System.out.println("Confirmed message:"+deliveryTag);
System.out.println("Batch confirmation: " + (multiply?"YES":"NO"));
};
// Message confirmation failure callback function
/*
* Parameter 1: tag of message
* Parameter 2: batch confirmation
* */
ConfirmCallback nackCallback = (deliveryTag, multiply) -> {
System.out.println("Unacknowledged message:"+deliveryTag);
System.out.println("Batch confirmation: " + (multiply?"YES":"NO"));
};
// Prepare a message listener to listen for which messages succeed and which messages fail
/*
* Parameter 1: which messages are successfully monitored
* Parameter 2: which messages failed to listen
* */
channel.addConfirmListener(ackCallback,nackCallback);
// Batch send messages
for (int i = 0; i < MESSAGE_COUNT; i++) {
String message = "news" + i;
channel.basicPublish("",queueName,null,message.getBytes(StandardCharsets.UTF_8));
}
// End time
long end = System.currentTimeMillis();
System.out.println("release"+MESSAGE_COUNT+"Asynchronous acknowledgement messages, time consuming"+ (end - begin) + "ms");
}
}
Because the success of message publishing and confirmation message sending are two different threads, the listener's execution is asynchronous. Therefore, it can be seen that after 1000 messages are published, the message confirmation information continues
You can use the listener as a single thread in addition to the main thread
How to handle asynchronous unacknowledged messages
The best solution is to put unconfirmed messages into a memory based queue that can be accessed by the publishing thread. For example, use the ConcurrentLinkedQueue queue to transfer messages between confirm callbacks and the publishing thread
At the same time of message publishing, all messages are recorded in a highly concurrent hash table ---- > in the success callback function of the listener, the successfully sent messages are deleted from the hash table ---- > in the failure callback function of the listener, the message of sending failure can be obtained by specifying a key
/*
* Release confirmation mode,
* 1,Single confirmation
* 2,Batch confirmation
* 3,Asynchronous batch confirmation
* */
public class ComfirmMessage {
// Number of batch messages
public static final int MESSAGE_COUNT = 1000;
public static void main(String[] args) throws Exception {
//3. Asynchronous batch confirmation
// Publish 1000 asynchronous confirmation messages, taking 36ms
ComfirmMessage.publicMessageAsync();
}
public static void publicMessageAsync() throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
String queueName = UUID.randomUUID().toString();
channel.queueDeclare(queueName,false,false,false,null);
// Open release confirmation
channel.confirmSelect();
/*
* A thread safe and orderly hash table is suitable for high concurrency
* 1,Easily associate sequence numbers with messages
* 2,Easy batch deletion, as long as the serial number is given
* 3,Support high concurrency
* */
ConcurrentSkipListMap<Long,String> outstandingConfirms = new ConcurrentSkipListMap<>();
// Message confirmation success callback function
ConfirmCallback ackCallback = (deliveryTag,multiply) -> {
// Delete the confirmed messages, and the rest are unconfirmed messages
//If the message is sent in batch, batch deletion is adopted here
if(multiply)
{
ConcurrentNavigableMap<Long, String> confiremed = outstandingConfirms.headMap(deliveryTag);
confiremed.clear();
}else {
//Otherwise, you can delete a single
outstandingConfirms.remove(deliveryTag);
}
System.out.println("Confirmed message:"+deliveryTag);
};
// Message confirmation failure callback function
/*
* Parameter 1: tag of message
* Parameter 2: batch confirmation
* */
ConfirmCallback nackCallback = (deliveryTag,multiply) -> {
// Print out the unconfirmed messages
String message = outstandingConfirms.get(deliveryTag);
System.out.println("Unacknowledged messages are:" + message +"Unacknowledged messages tag: " + deliveryTag);
};
// Prepare a message listener to listen for which messages succeed and which messages fail
/*
* Parameter 1: which messages are successfully monitored
* Parameter 2: which messages failed to listen
* */
channel.addConfirmListener(ackCallback,nackCallback);
// start time
long begin = System.currentTimeMillis();
// Batch send messages
for (int i = 0; i < MESSAGE_COUNT; i++) {
String message = "news" + i;
channel.basicPublish("",queueName,null,message.getBytes(StandardCharsets.UTF_8));
// The sum of all messages to be sent is recorded here
outstandingConfirms.put(channel.getNextPublishSeqNo(),message);
}
// End time
long end = System.currentTimeMillis();
System.out.println("release"+MESSAGE_COUNT+"Asynchronous acknowledgement messages, time consuming"+ (end - begin) + "ms");
}
}
Comparison of three release confirmation speeds
Switch
Introduction to switch
introduce
- The core idea of RabbitMQ messaging model is that messages produced by producers are never sent directly to queues. In fact, usually producers don't even know which queues these messages are delivered to.
- On the contrary, the producer can only send messages to the exchange. The work of the switch is very simple. On the one hand, it receives messages from the producer, on the other hand, it pushes them into the queue. The switch must know exactly how to process the received message. Should these messages be put in a specific queue, or should they be put in many queues, or should they be discarded. This depends on the type of switch.
type
Switches are of the following types:
direct = > route, topic, headers, fanout = > publish subscribe mode
Nameless switch:
In the previous section, we knew nothing about exchange, but we were still able to send messages to the queue. The reason why it can be implemented before is that we use the default exchange, which is identified by an empty string (").
The first parameter is the name of the switch. An empty string indicates the default or unnamed switch: messages can be routed to the queue, which is actually specified by the routingKey(bindingkey) binding key, if it exists
channel.basiPublish("","hello",null,message.getBytes())；
Temporary queue
- Whenever we connect to Rabbit, we need a new empty queue. Therefore, we can create a queue with random name, or let the server choose a random queue name for us. Secondly, once we disconnect the consumer, the queue will be deleted automatically.
- Create a temporary queue as follows:
String queueName = channel.queueDeclare().getQueue();
The created queue length is as follows:
AD stands for temporary queue
Bindings - binds the switch and the specified queue
What is binding? Binding is actually a bridge between exchange and queue. It tells us that exchange is bound to that queue. For example, the following figure tells us × Bound to Q1 and Q2
Fanout - publish subscribe mode
Fanout is a very simple type. As you can guess from the name, it broadcasts all received messages to all queues it knows. By default, there are some exchange types in the system
In short, the message in the Exchange is sent to all queues bound to the Exchange, ignoring the routingKey.
After the producer sends the message to the switch, the switch sends it to the consumer queue. If the consumer queue wants to receive the message in the switch, it needs to ensure that the switch name bound to the queue is consistent with the switch. This is the key of the broadcast mode and the coarsest premise of all subsequent modes of MQ.
such as
1) The manufacturer declares a switch (Exchange). The switch name is "fanoutLogs", the type is broadcast mode "fanout", and the message is "Now you see me";
//Parameters: switch name, switch type, broadcast mode
channel.exchangeDeclare("fanoutLogs", "fanout");
2) The consumer declares a queue Q1 and a switch (the name is "fanoutLogs" and the type is "fanout"), and finally the queue is bound to the switch;
channel.exchangeDeclare("fanoutLogs", "fanout");
//Parameters are: queue name; switch name; routingKey ignored
channel.queueBind("Q1", "fanoutLogs", "");
3) The consumer declares another queue Q2, declares a switch (the name is "phantaciLogs" and the type is "fan out"), and finally the queue is bound to the switch;
channel.exchangeDeclare("phantaciLogs", "fanout");
channel.queueBind("Q2", "phantaciLogs", "");
It should be noted that we should start the declaration consumer first and then the declaration producer. Otherwise, if we start the producer first, exchange finds that no queue is interested in it after receiving the message, and willfully discards the message
After Q1 and Q2 are started, start the producer. It can be found that Q1 with the same name as the producer Exchange normally receives messages; Q2 is also in the broadcast mode, but the switch name is different, so no messages are received.
The core idea of RabbitMQ message model: the producer will send messages to RabbitMQ exchange. One side of exchange is the producer and the other side is one or more queues. Exchange determines the life cycle of a message - send it to some queues or discard it directly.
actual combat
1. Consumers
/*
* messages receiving
* */
public class ReceiveLogs01 {
//Switch name
public static final String EXCHANGE_NAME = "logs";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Switch declaration (parameters: switch name; switch type)
channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.FANOUT);
//Temporary queue
//Declare a queue with random name. When the consumer disconnects from the queue, the queue will be deleted automatically
String queueName = channel.queueDeclare().getQueue();
//Binding switches and queues
//Parameters are: queue name; switch name; routingKey ignored
channel.queueBind(queueName,EXCHANGE_NAME,"");
System.out.println("Wait for the received message and print the received message on the screen...");
DeliverCallback deliverCallback = (consumerTag, message) -> {
System.out.println("ReceiveLogs01 The console prints the received message:" + new String(message.getBody()));
};
//Declare the messages consumed in the queue (parameters are: queue name; automatic message confirmation; callback for message delivery; consumer cancels the callback for message)
channel.basicConsume(queueName,true,deliverCallback,consumerTag -> {});
}
}
/*
* messages receiving
* */
public class ReceiveLogs02 {
//Switch name
public static final String EXCHANGE_NAME = "logs";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Switch declaration (parameters: switch name; switch type)
channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.FANOUT);
//Temporary queue
//Declare a queue with random name. When the consumer disconnects from the queue, the queue will be deleted automatically
String queueName = channel.queueDeclare().getQueue();
//Binding switches and queues
//Parameters are: queue name; switch name; routingKey ignored
channel.queueBind(queueName,EXCHANGE_NAME,"");
System.out.println("Wait for the received message and print the received message on the screen...");
DeliverCallback deliverCallback = (consumerTag, message) -> {
System.out.println("ReceiveLogs01 The console prints the received message:" + new String(message.getBody()));
};
//Declare the messages consumed in the queue (parameters are: queue name; automatic message confirmation; callback for message delivery; consumer cancels the callback for message)
channel.basicConsume(queueName,true,deliverCallback,consumerTag -> {});
}
}
2. Producers
/*
* Messaging switch
* */
public class Emitlog {
// Name of the switch
public static final String EXCHANGE_NAME = "logs";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Declare switch (parameters: switch name, switch type, broadcast mode)
channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.FANOUT);
Scanner scanner = new Scanner(System.in);
while (scanner.hasNext()){
String message = scanner.next();
//The parameters are: switch name; routingKey, ignored. In broadcast mode, the producer can declare the name and type of the switch
channel.basicPublish(EXCHANGE_NAME,"",null,message.getBytes(StandardCharsets.UTF_8));
System.out.println("Message from producer:"+ message);
}
channel.close();
}
}
Direct -- routing mode
If the producer and consumer have the same Exchange name, switch type and routing key, the consumer can successfully obtain the message.
(PS: compared with the broadcast mode in which messages can be received as long as the switch name, the direct mode adds a layer of routing key on the basis of it.)
The core idea of RabbitMQ message model: the producer will send messages to RabbitMQ exchange. One side of exchange is the producer and the other side is one or more queues. Exchange determines the life cycle of a message - send it to some queues or discard it directly.
In the above figure, we can see that X binds two queues, and the binding type is direct. The binding key of queue Q1 is orange, and the binding key of queue Q2 is green
In this case, the producer publishes the message to exchange, and the message with the binding key orange will be published to queue Q1. Messages with the binding key black or green will be published to queue Q2, and messages of other message types will be discarded.
Multiple binding
Of course, if the binding type of exchange is direct, but the key s of multiple queues it binds are the same, in this case, although the binding type is direct, it behaves a little similar to fanout, just like broadcasting, as shown in the above figure
Multiple routing key reception
By binding multiple routing keys to a consumer, the consumer can receive messages obtained by multiple routes at the same time.
If two routingkeys are bound to the consumer code [DirectWorker], the rest remain unchanged
//routingKey is info
channel.queueBind("console",EXCHANGE_NAME,"info");
//routingKey is warning
channel.queueBind("console",EXCHANGE_NAME,"warning");
actual combat
producer
public class Producer
{
// Name of the switch
public static final String EXCHANGE_NAME = "direct_logs";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Declare switch: the name of the switch and the type of switch
channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.DIRECT);
Scanner scanner = new Scanner(System.in);
while (scanner.hasNext()){
String message = scanner.next();
//Channel publishing message: the name of the switch, secret key: Info --- > sent to the specified queue, and the binary stream of the sent message
channel.basicPublish(EXCHANGE_NAME,"info",null,message.getBytes(StandardCharsets.UTF_8));
System.out.println("Message from producer:"+ message);
}
}
}
consumer
public class ReceiveLogsDirect01 {
public static final String EXCHANGE_NAME = "direct_logs";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Declare a queue
channel.queueDeclare("console",false,false,false,null);
//Binding switches and queues
//Parameter: queue name; Switch name; Key routingKey
channel.queueBind("console",EXCHANGE_NAME,"info");
channel.queueBind("console",EXCHANGE_NAME,"warning");
DeliverCallback deliverCallback = (consumerTag, message) -> {
System.out.println("ReceiveLogsDirect01 The console prints the received message:" + new String(message.getBody()));
};
channel.basicConsume("console",true,deliverCallback,consumerTag -> {});
}
}
public class ReceiveLogsDirect02 {
public static final String EXCHANGE_NAME = "direct_logs";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Declare a queue
channel.queueDeclare("disk",false,false,false,null);
//Binding switches and queues
channel.queueBind("disk",EXCHANGE_NAME,"error");
DeliverCallback deliverCallback = (consumerTag, message) -> {
System.out.println("ReceiveLogsDirect02 The console prints the received message:" + new String(message.getBody()));
};
channel.basicConsume("disk",true,deliverCallback,consumerTag -> {});
}
}
Topic - topic mode
introduce
- Routing to messages of type topic switch_ The key cannot be written arbitrarily. It must meet certain requirements. It must be a word list separated by a dot. These words can be any word, such as "stock.usd.nyse", "nyse.vmw", "quick.orange.rabbit". Of course, the word list cannot exceed 255 bytes at most.
- In this rule list, there are two substitutes:
* Can replace a word # Can replace zero or more words
Matching case
The following figure shows the binding relationship:
Q1 binds a string of three words with orange in the middle: *. Orange*
Q2 binds: the last word is a single word of rabbit: *. *. Rabbit, and the first word is multiple words of lazy: lazy#
Data reception is as follows:
- quick.orange.rabbit: received by queue Q1Q2
- quick.orange.fox: received by queue Q1
- lazy.brown.fox: received by queue Q2
- lazy.pink.rabbit: Although the two bindings of queue Q2 are satisfied, it will only be received once
- quick.orange.male.rabbit: four words do not match, and any binding will be discarded
conclusion
- When the binding key of a queue is #, the queue will receive all data, which is a bit like fanout
- If there is no # and * in the queue binding key, the queue binding type is direct
actual combat
producer
public class EmitLogTopic {
//Name of the switch
public static final String EXCHANGE_NAME = "topic_logs";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
HashMap<String, String> map = new HashMap<>();
map.put("quick.orange.rabbit","By queue Q1Q2 Received");
map.put("quick.orange.fox","By queue Q1 Received");
map.put("lazy.brown.fox","By queue Q2 Received ");
map.put("lazy.pink.rabbit","Although the queue is satisfied Q2 But will only be received once");
map.put("quick.orange.male.rabbit","If the four words do not match, any binding will be discarded");
for (Map.Entry<String, String> bindingKeyEntry : map.entrySet()) {
String routingKey = bindingKeyEntry.getKey();
String message = bindingKeyEntry.getValue();
//Message not persistent
channel.basicPublish(EXCHANGE_NAME,routingKey,null,message.getBytes(StandardCharsets.UTF_8));
System.out.println("Message sent by producer:"+ message );
}
}
}
consumer
/*
* Declare the subject switch and related queues
* Consumer C1
* */
public class ReceiveLogsTopic01 {
//Switch name
public static final String EXCHANGE_NAME = "topic_logs";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Claim switch
channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.TOPIC);
//Declaration queue
String queueName = "Q1";
channel.queueDeclare(queueName,false,false,false,null);
//Queue bundling
//Set secret key
channel.queueBind(queueName,EXCHANGE_NAME,"*.orange.*");
System.out.println("Waiting to receive message......");
//Callback function after receiving message
DeliverCallback deliverCallback = (consumerTag, message) -> {
System.out.println(new String(message.getBody()));
System.out.println("Receive queue:"+ queueName + "Binding key:" + message.getEnvelope().getRoutingKey());
};
//receive messages
channel.basicConsume(queueName,true,deliverCallback,consumerTag -> {});
}
}
/*
* Declare the subject switch and related queues
* Consumer C2
* */
public class ReceiveLogsTopic02 {
//Switch name
public static final String EXCHANGE_NAME = "topic_logs";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Claim switch
channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.TOPIC);
//Declaration queue
String queueName = "Q2";
channel.queueDeclare(queueName,false,false,false,null);
//Queue bundling
channel.queueBind(queueName,EXCHANGE_NAME,"*.*.rabbit");
channel.queueBind(queueName,EXCHANGE_NAME,"*lazy.#");
System.out.println("Waiting to receive message......");
DeliverCallback deliverCallback = (consumerTag, message) -> {
System.out.println(new String(message.getBody()));
System.out.println("Receive queue:"+ queueName + "Binding key:" + message.getEnvelope().getRoutingKey());
};
//receive messages
channel.basicConsume(queueName,true,deliverCallback,consumerTag -> {});
}
}
Dead letter queue
Concept of dead letter
- Dead letter, as the name suggests, is a message that cannot be consumed. The literal meaning can be understood as follows. Generally speaking, the producer delivers the message to the broker or directly to the queue. The consumer takes the message out of the queue for consumption, but sometimes some messages in the queue cannot be consumed for specific reasons. If such messages are not processed subsequently, It becomes a dead letter. If there is a dead letter, there will be a dead letter queue.
- Application scenario: in order to ensure that the message data of the order business is not lost, the dead letter queue mechanism of RabbitMQ needs to be used. When the message consumption is abnormal, the message is put into the dead letter queue. For example, after the user successfully places an order in the mall and clicks to pay, it will automatically become invalid if it is not paid within the specified time
Source of dead letter
- Message TTL expired
- The queue has reached the maximum length (the queue is full and can no longer add data to mq)
- The message is rejected (basic.reject or basic.nack) and request = false
How to deal with dead letter
- Discard. If it is not very important, you can choose to discard it
- Record dead letter receipt, and then conduct subsequent business analysis or processing
- Through the dead letter queue, it is processed by the application responsible for listening for dead letters
How to configure dead letter queue
- Configure the service queue and bind it to the service switch
- Configure dead letter switch and routing key for service queue
- Configure dead letter queue for dead letter switch
Note that it does not directly declare a public dead letter queue, and then all dead letter messages run to the dead letter queue by themselves. Instead, configure a dead letter switch for each service queue that needs to use dead letter. Here, the dead letter switch of the same project can share one, and then assign a separate routing key to each service queue.
With the dead letter switch and routing key, configure the dead letter queue just like configuring the service queue, and then bind it to the dead letter switch.
In other words, the dead letter queue is not a special queue, but a queue bound to the dead letter switch.
Dead letter switch is not a special switch, but a switch used to accept dead letter, so it can be any type of [Direct, Fanout, Topic].
Generally speaking, a unique routing key will be assigned to each service queue, and a dead letter queue will be configured accordingly for listening, that is, a dead letter queue will be configured for each important service queue.
Code architecture diagram
Message TTL expired
Generally speaking, after the message is generated, because the timeout is set, the message will be thrown into the dead letter queue if it is not consumed during this time.
producer
/*
* Producer code of dead letter queue
*
* */
public class Producer {
//Name of common switch
public static final String NORMAL_EXCHANGE = "normal_exchange";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
//For dead letter message, set TTL time unit as MS and 10000ms as 10s
AMQP.BasicProperties properties = new AMQP.BasicProperties().builder().expiration("10000").build();
for (int i = 0; i < 10; i++) {
String message = "info" + i;
//Release news
//Publishing to a nonexistent switch will cause an exception to the channel level protocol, which closes the channel,
//Exchange: the exchange to send the message to
//routingKey: routing KEY
//props: other properties of the message, such as routing header, etc
//Body: message body
channel.basicPublish(NORMAL_EXCHANGE,"zhangsan",properties,message.getBytes(StandardCharsets.UTF_8));
}
}
}
Properties in AMQP.BasicProperties of RabbitMq
The props parameter can give a variety of functional features to the message body.
The properties of BasicProperties are as follows
String contentType, //Type of message content String contentEncoding, //Encoding format of message content Map<String,Object> headers,//header switches can be used Integer deliveryMode,//Message persistence 1 no persistence 2 persistence Integer priority,//priority String correlationId, //Association id String replyTo,//Commonly used to name callback queues String expiration,//Set expiration time for expired messages String messageId, //Message id Date timestamp, //Timestamp of the message String type, //type String userId, //User ID String appId, //application id String clusterId //Cluster id
consumer
After startup, close 01. The consumer simulates that it cannot receive a message
/*
* Dead letter queue actual combat
* Consumer 01
* */
public class Consumer01 {
//Common switch name
public static final String NORMAL_EXCHANGE = "normal_exchange";
//Dead letter switch name
public static final String DEAD_EXCHANGE = "dead_exchange";
//Common queue name
public static final String NORMAL_QUEUE = "normal_queue";
//Dead letter queue name
public static final String DEAD_QUEUE = "dead_queue";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Declare dead letter and ordinary switch type as direct
channel.exchangeDeclare(NORMAL_EXCHANGE, BuiltinExchangeType.DIRECT);
channel.exchangeDeclare(DEAD_EXCHANGE, BuiltinExchangeType.DIRECT);
//Declare normal queue
HashMap<String, Object> arguments = new HashMap<>();
//Set the survival time for each message in the current queue. The unit is milliseconds and the value is a non negative integer
//Expiration time. The expiration time of all messages is set uniformly in the producer section. There is no need to set it here
//arguments.put("x-message-ttl",1000);
//Normal queue setting dead letter queue - the normal queue needs to communicate with the dead letter queue to ensure that the dead letter is delivered to the dead letter queue
arguments.put("x-dead-letter-exchange",DEAD_EXCHANGE);
//Set dead letter RoutingKey
arguments.put("x-dead-letter-routing-key","lisi");
//Declare dead letter and normal queue
// arguments: additional parameter settings to set some characteristics of the current queue and some characteristics of messages in the queue
channel.queueDeclare(NORMAL_QUEUE,false,false,false,arguments);
channel.queueDeclare(DEAD_QUEUE,false,false,false,null);
//Bind ordinary switches to ordinary queues
channel.queueBind(NORMAL_QUEUE,NORMAL_EXCHANGE,"zhangsan");
//Bind the switch of dead letter to the queue of dead letter
channel.queueBind(DEAD_QUEUE,DEAD_EXCHANGE,"lisi");
System.out.println("Waiting to receive message......");
DeliverCallback deliverCallback = (consumerTag, message) -> {
System.out.println("Consumer01 The message received is:" + new String(message.getBody()));
};
channel.basicConsume(NORMAL_QUEUE,true,deliverCallback,consumerTag->{});
}
}
/*
* Dead letter queue actual combat
* Consumer 02
* */
public class Consumer02 {
//Dead letter queue name
public static final String DEAD_QUEUE = "dead_queue";
//The dead letter queue only needs to be bound to the dead letter switch
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
System.out.println("Waiting to receive message......");
DeliverCallback deliverCallback = (consumerTag, message) -> {
System.out.println("Consumer02 The message received is:" + new String(message.getBody()));
};
channel.basicConsume(DEAD_QUEUE,true,deliverCallback,consumerTag->{});
}
}
Effect: after starting the producer, 10 messages are sent to NORMAL_QUEUE is then transmitted to DEAD_QUEUE, start consumer 02 and all messages are received.
Common switch binding common queue
Dead letter switch binding dead letter queue
First start the consumer 01 service, declare the general switch, queue and dead letter switch. After the queue, shut down the service, and then turn on the producer. It is found that after the message is routed to the general queue through the general switch, because no one consumes it after 10 seconds, all 10 messages become dead letters and are placed in the dead letter queue
At this time, start the service to consume the messages in the dead letter queue. After consumption, the messages in the dead letter queue are gone because they have been successfully consumed
Summary of queueDeclare() parameter in RabbitMq
Channel binding corresponding message queue function:
queueDeclare(String queue,boolean durable,boolean exclusive,boolean autoDelete,Map<String, Object>arguments)
- Parameter 1(String queue): Specifies the name of the queue. If the queue does not exist, a queue will be automatically created with the name of the incoming queue;
- Parameter 2 (Boolean drusable): defines whether the queue feature needs to be persisted. If it is false, the queue will be deleted the next time mq is started. If true, the queue will still exist at the next startup, but it should be noted that the messages in the queue will not be retained. The queue will be empty at the next startup. If you need message persistence, you can specify messageproperties.persistent when passing messages using the basicPublish function_ TEXT_ Plan, which will recover the messages in the queue the next time it is started.
- Parameter 3(boolean exclusive): whether to monopolize the queue. If true, the queue can only be bound to the current channel, and other channels cannot access the queue
- Parameter 4(boolean autoDelete): whether to delete the queue automatically. When the consumer consumes the data in the queue and disconnects from the queue, whether to delete the queue
- Parameter 5 (map < string, Object > arguments): additional parameter settings
It should be noted that a channel can bind multiple queues
Detailed explanation of the parameters of the queueDeclare method
Detailed explanation of common functions in RabbitMQ Channel class
Detailed explanation of rabbitmq channel method parameters
Dead letter queue recommended articles
[RabbitMQ] will take you through RabbitMQ dead letter queue
The queue has reached its maximum length
The message producer code removes the TTL attribute
Maximum number of messages added to the queue by consumer 01
(after startup, close the consumer to simulate that it cannot receive messages). The purpose of startup is to declare the switch and queue
//Set the limit on the number of messages stored in the normal queue
arguments.put("x-max-length",6);
Note that the original queue needs to be deleted at this time because the parameters have changed
Consumer 02 does not need to make changes
Message rejected
Consumer 01
/*
* Dead letter queue actual combat
* Consumer 01
* */
public class Consumer01 {
//Common switch name
public static final String NORMAL_EXCHANGE = "normal_exchange";
//Dead letter switch name
public static final String DEAD_EXCHANGE = "dead_exchange";
//Common queue name
public static final String NORMAL_QUEUE = "normal_queue";
//Dead letter queue name
public static final String DEAD_QUEUE = "dead_queue";
public static void main(String[] args) throws Exception{
Channel channel = RabbitMqUtils.getInstance().getChannel();
//Declare dead letter and ordinary switch type as direct
channel.exchangeDeclare(NORMAL_EXCHANGE, BuiltinExchangeType.DIRECT);
channel.exchangeDeclare(DEAD_EXCHANGE, BuiltinExchangeType.DIRECT);
//Declare normal queue
HashMap<String, Object> arguments = new HashMap<>();
//Set the survival time for each message in the current queue. The unit is milliseconds and the value is a non negative integer
//Expiration time. The expiration time of all messages is set uniformly in the producer section. There is no need to set it here
//arguments.put("x-message-ttl",1000);
//Set the limit on the number of messages stored in the normal queue
//arguments.put("x-max-length",6);
//Normal queue setting dead letter queue - the normal queue needs to communicate with the dead letter queue to ensure that the dead letter is delivered to the dead letter queue
arguments.put("x-dead-letter-exchange",DEAD_EXCHANGE);
//Set dead letter RoutingKey
arguments.put("x-dead-letter-routing-key","lisi");
//Declare dead letter and normal queue
// arguments: additional parameter settings to set some characteristics of the current queue and some characteristics of messages in the queue
channel.queueDeclare(NORMAL_QUEUE,false,false,false,arguments);
channel.queueDeclare(DEAD_QUEUE,false,false,false,null);
//Bind ordinary switches to ordinary queues
channel.queueBind(NORMAL_QUEUE,NORMAL_EXCHANGE,"zhangsan");
//Bind the switch of dead letter to the queue of dead letter
channel.queueBind(DEAD_QUEUE,DEAD_EXCHANGE,"lisi");
System.out.println("Waiting to receive message......");
DeliverCallback deliverCallback = (consumerTag, message) ->
{
String msg= new String(message.getBody());
if(msg.equals("info5")){
System.out.println("Consumer01 Message received" + msg + "And refuse to sign the message");
//If the request is set to false, it means to refuse to re queue the queue. If dead letter is configured, the switch will send it to the dead letter queue
channel.basicReject(message.getEnvelope().getDeliveryTag(), false);
}else {
//Others that are not rejected shall be answered manually
System.out.println("Consumer01 Message received"+msg);
channel.basicAck(message.getEnvelope().getDeliveryTag(), false);
}
};
//Manual start here
channel.basicConsume(NORMAL_QUEUE,false,deliverCallback,consumerTag->{});
}
}
Consumer 02 code unchanged
No change in producer code
