redis implementation queue

Redis is designed for caching, but it can be used for message queuing due to its own characteristics.

It has several blocking APIs that can be used. It is these blocking APIs that enable it to do message queuing;
In addition, other features of message queuing, such as FIFO (first in, first out), are also easy to implement. Only a list object is needed to fetch data from the beginning and plug data from the tail;
Redis can do message queuing thanks to its list object blpop brpop interface and some interfaces of Pub/Sub (publish / subscribe). They are blocking versions, so they can be used to do message queuing. (List : lpush / rpop)

Mode 1: producer consumer mode


1. Use the list structure as the queue, rpush produces messages, and lpop consumes messages. When there is no message in lpop, sleep appropriately and try again later.
Or, instead of sleep, use the blpop command directly. When there is no message, it will block until the message arrives. But redis has no akc function

2. Code:

@Component
@RequestMapping("/RedisApplication")
public class RedisApplication {

    @Autowired
    private RedisTemplate redisTemplate;

    @RequestMapping(value = "/testFIFO")
    public void testFIFO() throws InterruptedException {
        System.out.println("---------------Start queue--------------");
        for (int i = 0; i < 5; i++) {
            String arg = "key" + i;
            redisTemplate.opsForList().leftPush("FIFOKEY", arg);
        }
        System.out.println("----------------Put in queue stop------------");
        while (true) {
            Object outKey = redisTemplate.opsForList().rightPop("FIFOKEY");
            if (outKey != null) {
                System.out.println(outKey);
            } else {
                Thread.sleep(500);
            }
        }
    }
}

 

Mode 2: publish subscriber mode


Using the pub/sub topic subscriber mode, you can implement a 1:N message queue.

Disadvantage: when the consumer is offline, the production message will be lost. In this scenario, MQ is recommended.

Code:

Requested interface: the subscriber, using the redis queue, placed in the queue.

package com.airboot.bootdemo.controller;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;


@RestController
@RequestMapping("/SentRedisController")
public class SentRedisController {

    @Autowired
    private RedisTemplate redisTemplate;

    /**
     * redis Producer testing
     * @param data
     * @return
     */
    @GetMapping("/send1")
    String send1(String data) {
        redisTemplate.convertAndSend("testkafka", data);
        return "success";
    }
    /**
     * redis Producer testing
     * @param data
     * @return
     */
    @GetMapping("/send2")
    String send2(String data) {
        redisTemplate.convertAndSend("testkafka1", data);
        return "success";
    }
}

Configure the listener. listen queue

package com.airboot.bootdemo.config;

import com.airboot.bootdemo.controller.RedisSubscriber;
import com.airboot.bootdemo.controller.RedisSubscriberTwo;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.listener.PatternTopic;
import org.springframework.data.redis.listener.RedisMessageListenerContainer;
import org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;


@Configuration
public class RedisConfig {

    @Autowired
    private RedisTemplate redisTemplate;

   //Serialize general settings
    @Bean
    public RedisTemplate redisTemplateInit() {
        Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);
        // Set the instantiation object of the serialized Key
        redisTemplate.setKeySerializer(new StringRedisSerializer());
        // Set the instantiation object of the serialized Value
        redisTemplate.setValueSerializer(new GenericJackson2JsonRedisSerializer());
        redisTemplate.setHashKeySerializer(jackson2JsonRedisSerializer);
        redisTemplate.setHashValueSerializer(jackson2JsonRedisSerializer);
        return redisTemplate;
    }

    //Configure the listening configuration to enable the class RedisSubscriber RedisSubscriberTwo to listen to testkafka1, 
    //testkafka these two queues
    @Bean
    RedisMessageListenerContainer container(RedisConnectionFactory connectionFactory,
                                            RedisSubscriber listenerAdapter,
                                            RedisSubscriberTwo listenerAdapter2){
        RedisMessageListenerContainer container = new RedisMessageListenerContainer();
        container.setConnectionFactory(connectionFactory);
        //Subscribed to a channel called chat
        container.addMessageListener(listenerAdapter, new PatternTopic("testkafka"));
        container.addMessageListener(listenerAdapter, new PatternTopic("testkafka1"));//Configure subscriptions to subscribe to
        container.addMessageListener(listenerAdapter2, new PatternTopic("testkafka"));//Configure subscriptions to subscribe to
        //This container can add multiple messagelisteners
        return container;
    }

}

Multiple consumers can use the above configuration to listen to multiple classes

package com.airboot.bootdemo.controller;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.connection.Message;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.listener.adapter.MessageListenerAdapter;
import org.springframework.stereotype.Component;

@Component
public class RedisSubscriberTwo extends MessageListenerAdapter {

    @Autowired
    private RedisTemplate<String, String> redisTemplate;

    @Override
    public void onMessage(Message message, byte[] bytes) {
        System.out.println(message);
        byte[] body = message.getBody();
        byte[] channel = message.getChannel();
        String msg = redisTemplate.getStringSerializer().deserialize(body);
        String topic = redisTemplate.getStringSerializer().deserialize(channel);
        System.out.println("Listen to topic For 2" + topic + "News:" + msg);
    }
}


package com.airboot.bootdemo.controller;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.connection.Message;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.listener.adapter.MessageListenerAdapter;
import org.springframework.stereotype.Component;

@Component
public class RedisSubscriber extends MessageListenerAdapter {
    //
    @Autowired
    private RedisTemplate<String, String> redisTemplate;

    @Override
    public void onMessage(Message message, byte[] bytes) {
        System.out.println(message);
        byte[] body = message.getBody();
        byte[] channel = message.getChannel();
        String msg = redisTemplate.getStringSerializer().deserialize(body);
        String topic = redisTemplate.getStringSerializer().deserialize(channel);
        System.out.println("Listen to topic by" + topic + "News:" + msg);
    }
}

Mode 3: delay queue

In the above example, we have a simple message queue. Let's continue to think about a realistic scenario, assuming that these are some game products, which need to add the "delayed sales" feature, and can start processing these game product data at some time in the future. To implement this delay feature, we need to modify the implementation of the existing queue:

  1. The message data contains the execution time of the delayed processing message. If the working process finds that the execution time of the message has not yet reached, it will push the message data into the queue again after a short wait. (delay sending message)
  2. Use the ordered set to store the message data that needs to be consumed later. Set the execution time of the task to the score. Start a working process to find out whether there are tasks that can be executed immediately in the ordered set. If so, remove the message from the ordered set and consume it.

Code:

package com.airboot.bootdemo.utils;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.TypeReference;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.ZSetOperations;

import java.lang.reflect.Type;
import java.util.Set;
import java.util.UUID;


public class RedisDelayQueue<T> {


    private String queueKey;

    public RedisTemplate redisTemplate;

    // TypeReference is required when generic types exist in fastjson serialized objects
    private Type TaskType = new TypeReference<TaskItem<T>>() {
    }.getType();

    public RedisDelayQueue(RedisTemplate redisTemplate, String queueKey) {
        this.queueKey = queueKey;
        this.redisTemplate = redisTemplate;
    }

    static class TaskItem<T> {
        public String id;
        public T msg;
    }

    public void delay(T msg) {
        TaskItem<T> item = new TaskItem<T>();
        //Assign unique uuid
        item.id = UUID.randomUUID().toString();
        item.msg = msg;
        //fastjson serialization
        String s = JSON.toJSONString(item);
        ZSetOperations operations = redisTemplate.opsForZSet();
        //Insert the delay queue and try again in 5s
        operations.add(queueKey, s, System.currentTimeMillis() + 5000);
    }

    public void loop() {
        while (!Thread.interrupted()) {
            //Just take one.
            Set<String> values = redisTemplate.opsForZSet().rangeByScore(queueKey, 0, System.currentTimeMillis(), 0, 1);
            if (values.isEmpty()) {
                try {
                    //Break will continue
                    Thread.sleep(500);
                } catch (InterruptedException e) {
                    break;
                }
                continue;
            }
            String s = values.iterator().next();
            if (redisTemplate.opsForZSet().remove(queueKey, s) > 0) {
                //Multiple processes call at the same time, only one will remove successfully
                TaskItem<T> task = JSON.parseObject(s, TaskType);
                //Execute business logic
                handleTask(task.msg);
            }
        }
    }

    private void handleTask(T msg) {
        System.out.println(msg);
    }
}

call

@RequestMapping("/testDelayQueue")
    public void testDelayQueue() {
        RedisDelayQueue queue = new RedisDelayQueue(redisTemplate, "DelayQueue");
        Thread producer = new Thread() {

            @Override
            public void run() {
                for (int i = 0; i < 10; i++) {
                    queue.delay("DelayQueue" + i);
                }
            }

        };
        Thread consumer = new Thread() {

            @Override
            public void run() {
                queue.loop();
            }

        };
        producer.start();
        consumer.start();
        try {
            producer.join();
            Thread.sleep(6000);
            consumer.interrupt();
            consumer.join();
        } catch (InterruptedException e) {
        }
    }

Principle:

There are several statements:

        Put it into redis, the weight is 5 seconds after the current time
        operations.add(queueKey, s, System.currentTimeMillis() + 5000);
//Get the first data in the set whose time is less than the current time, so you get the above data after 5 seconds
Set<String> values = redisTemplate.opsForZSet().rangeByScore(queueKey, 0, System.currentTimeMillis(), 0, 1);
Remove the data found above and imitate the data to be consumed
redisTemplate.opsForZSet().remove(queueKey, s)

 

103 original articles published, praised 8, visited 10000+
Private letter follow

Tags: Redis JSON Java less

Posted on Mon, 13 Jan 2020 08:31:25 -0500 by phreek