How to avoid repeated consumption of message queues: Redis solves the problem of repeated consumption

Repeated consumption:

In order to solve the problem of message loss caused by various reasons on the consumer side, we all know that the root cause is the automatic ack mechanism of RabbitMQ. Therefore, in order to avoid the above problems, we will select Manual ack to ensure that messages will not be lost for some reasons.

However, there is also a problem: if you forget the ACK, or the consumer fails to give RabbitMQ the corresponding ack for various reasons, and you can't confirm that the message has been consumed, the message that has not been "constrained" may be consumed by another consumer, resulting in repeated consumption

If you are adding or performing some non idempotent operations, such as fee deduction, you are finished

Redis seems to be a better solution to the problem of repeated consumption:

The idea is as follows:

Before consumers consume a message, first put the id of the message into Redis

The clearest way: when the id is set to 0, the business is being executed, and when the id is 1, the business is successfully executed and the consumption is completed

If the manual ack fails, when RabbitMQ gives the message to other consumers, execute setnx first. If the key already exists, get its value. If the value is 0, that is, the current consumer will do nothing before the current message is consumed. If the value is 1, the ACK operation will be performed directly.

Of course, there will be a very serious hidden danger:

If the first consumer has a deadlock problem during business execution, it will be unable to obtain the resource of the key

Therefore, when putting the message id into Redis, we should set a lifetime, or expiration time, to avoid deadlock in extreme cases.


I don't have much to say. Let's test and practice first

1, Test dependent lead



  2, Before testing, you use a previously written tool class to create a connection, so you don't have to configure the connection information all the time

import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;

public class RabbitMQClient {

    public static Connection getConnection(){
        // Create Connection factory
        ConnectionFactory factory = new ConnectionFactory();

        // Create Connection
        Connection conn = null;
        try {
            conn = factory.newConnection();
        } catch (Exception e) {
        // Return to Connection
        return conn;

  3, Producer code implementation

import java.util.UUID;

public class Publisher {
    public void publish() throws Exception {
        //Create a message queuing connection using a tool class
        Connection connection = RabbitMQClient.getConnection();
        //Create Channel
        Channel channel = connection.createChannel();
        //Specify routing rules and message properties
        AMQP.BasicProperties properties = new AMQP.BasicProperties().builder()
                .deliveryMode(1)  //0: message not persistent 1: Message persistent
                .messageId(UUID.randomUUID().toString())   //Generate random information id
        String msg = "This is MQ Test!"; //Message body
        // Parameter 1: Exchange: specify exchange, use '', and use the default switch
        // Parameter 2: routingKey: Specifies the routing rule and uses the specific queue name.
        // Parameter 3: mandatory: if exchange cannot find a qualified queue according to its type and message routeKey, it will return the message to the producer.
        // Parameter 4: BasicProperties: Specifies the properties carried by the delivered message.
        // Parameter 5 byte []: Specifies the specific message to be published

        System.out.println("The producer released the message successfully!");

4, Consumer code implementation

import com.yj.utils.RabbitMQClient;
import com.rabbitmq.client.*;
import org.junit.Test;
import redis.clients.jedis.Jedis;


public class Consumer {
    public void consume() throws Exception {
        Connection connection = RabbitMQClient.getConnection();
        Channel channel = connection.createChannel();

        //Declaration queue
        //Parameter 1: queue - specifies the name of the queue
        //Parameter 2: durable - whether the current queue needs to be persisted (true. If false is set, all queues will be deleted after MQ restarts)
        //Parameter 3: exclusive - exclusive (conn.close() the current queue will be automatically deleted, and the current queue can only be consumed by one consumer)
        //Parameter 4: autoDelete - if there are no consumers in this queue, the queue will be deleted automatically
        //Parameter 5: arguments - specifies additional information about the current queue

        //Custom listener listening queue
        DefaultConsumer consumer = new DefaultConsumer(channel){
            public void handleDelivery(String consumerTag, Envelope envelope,
                                       AMQP.BasicProperties properties, byte[] body) throws IOException {
                //1. Connect to redis
                Jedis jedis = new Jedis("", 6379);
                //2. Get the message id in the queue
                String messageId = properties.getMessageId();
                //3. setnx to redis, value is 0 by default, and the expiration time is set to 10s
                String result = jedis.set(messageId, "0", "NX", "EX", 10);

                if (result != null && result.equalsIgnoreCase("OK")){
                    System.out.println("Received message:" + new String(body,"UTF-8"));

                    //4. If it is successfully added, the consumption is successful, and the message id is set, indicating that the business is being executed
                    //1 is not set here. 6868 in order to stand out in the test database,

                    //5. Manual ack
                    //deliveryTag: the index of the message;
                    //multiple: batch. true: all messages less than deliveryTag will be ack ed at one time.

                }else {
                    //6. If setnx in 2 fails, get the value corresponding to the key. If it is 0, do nothing. If it is 1, manually ack
                    String s = jedis.get(messageId);
                    if ("1".equalsIgnoreCase(s)){

        System.out.println("Consumer starts listening to queue!");;//Let the program not stop, only the keyboard input data will go down

        // Release resources

5, A simple test



Test successful!

  Finally: it seems feasible+_+

Tags: Java RabbitMQ Distribution

Posted on Thu, 25 Nov 2021 13:49:51 -0500 by marty68