Redis -- delay queue

> rpush my-queue apple banana pear (integer) 3 > llen my-queue (integer) 3 > lpop my-queue "apple" > llen my-queue (integer) 2 > lpop my-queue "banana" > llen my-queue (integer) 1 > lpop my-queue "pear" > llen my-queue (integer) 0 > lpop my-queue (nil)

Empty queue

1. If the queue is empty, the client will fall into the loop of pop. Empty polling not only increases the CPU of the client, but also the QPS of Redis
2. If there are dozens of empty polling clients, Redis's slow query will increase significantly. You can try to let the client thread sleep 1s
3. But sleep can cause the delay of messages to increase. You can use blpop/brpop (blocking, blocking read)
4. When there is no data in the queue, the blocking read will immediately enter the sleep state. Once there is data coming, it will be awakened immediately, and the message delay is almost 0

idle connection

1. If the thread keeps blocking there, the client connection of Redis becomes idle connection
2. If the server is idle for a long time, it will disconnect actively to reduce idle resource consumption. At this time, blpop/brpop will throw an exception

Lock conflict handling

1. A strategy to deal with the failure of locking in distributed system
2. Throw an exception directly and notify the user to try again later
3. sleep and try again
4. Transfer the request to the delay queue and try again later
5. Throw exception
6. This method is more suitable for requests directly initiated by users
7. sleep
8. sleep will block the current message processing thread, which will delay the subsequent message processing of the queue
9. If the collision is frequent, the sleep scheme is not suitable
10. Delay queue
11. It is more suitable for asynchronous message processing scenarios, which can avoid conflicts by transferring the current conflicting requests to another queue for later processing

Delay queue

1. Delay queue can be realized through zset of Redis
2. Serialize the message into a string as the value of zet, and regard the expiration processing time of the message as the score
3. Then multithread polls zset to get the expired tasks for processing
4. Multithreading is to ensure availability, but at the same time, concurrent security should be considered to ensure that tasks cannot be executed multiple times

public class RedisDelayingQueue<T> { @Data @AllArgsConstructor @NoArgsConstructor private static class TaskItem<T> { private String id; private T msg; } private Type taskType = new TypeReference<TaskItem<T>>() { }.getType(); private Jedis jedis; private String queueKey; public RedisDelayingQueue(Jedis jedis, String queueKey) { this.jedis = jedis; this.queueKey = queueKey; } public void delay(T msg) { TaskItem<T> task = new TaskItem<>(UUID.randomUUID().toString(), msg); jedis.zadd(queueKey, System.currentTimeMillis() + 5000, JSON.toJSONString(task)); } public void loop() { // It can be further optimized. zrangeByScore and zrem can be moved to Redis server through Lua script for atomization operation to reduce the waste of resources caused by seizing failure while (!Thread.interrupted()) { // Just take one. Set<String> values = jedis.zrangeByScore(queueKey, 0, System.currentTimeMillis(), 0, 1); if (values.isEmpty()) { try { Thread.sleep(500); } catch (InterruptedException e) { break; } continue; } String s = values.iterator().next(); if (jedis.zrem(queueKey, s) > 0) { // zrem is the key to multithreading and multiprocessing TaskItem<T> task = JSON.parseObject(s, taskType); this.handleMsg(task.msg); } } } private void handleMsg(T msg) { try { System.out.println(msg); } catch (Throwable ignored) { // Be sure to catch exceptions to avoid loop exception exit due to individual task processing problems } } public static void main(String[] args) { final RedisDelayingQueue<String> queue = new RedisDelayingQueue<>(new Jedis("localhost", 16379), "q-demo"); Thread producer = new Thread() { @Override public void run() { for (int i = 0; i < 10; i++) { queue.delay("zhongmingmao" + i); } } }; Thread consumer = new Thread() { @Override public void run() { queue.loop(); } }; producer.start(); consumer.start(); try { producer.join(); Thread.sleep(6000); consumer.interrupt(); consumer.join(); } catch (InterruptedException ignored) { } } }

6 November 2019, 04:18 | Views: 7924