Play redis delay message queue

In the previous article, the redis based list implemented a simple message queue: Play redis simple message queue

Source address Using demo

Product manager often said that we should not only have X function, but also Y function, so that customers can be more satisfied. Similarly, it is not enough to have a simple message queue, and a delay message queue is needed to be a complete message queue.

Look at the command of redis. Looking at it, the sorted set is a very useful command. It can be used to make a delayed message queue

redis ordered set

redis is an ordered collection. Each element is associated with a score of double type. redis sorts the members of the collection from small to large by scores.
Members of an ordered set are unique, but scores can be repeated.

Simple operation

Add data

127.0.0.1:6379> ZADD testSet1 5 a
(integer) 1
127.0.0.1:6379> ZADD testSet1 1 b 8 c 7 d
(integer) 3

read

127.0.0.1:6379> ZRANGEBYSCORE testSet1 0 3
1) "b"
127.0.0.1:6379> ZRANGEBYSCORE testSet1 0 5
1) "b"
2) "a"

You can also type the score

127.0.0.1:6379> ZRANGEBYSCORE testSet1 -inf 5 WITHSCORES
1) "b"
2) "1"
3) "a"
4) "5"

Find out all the data

127.0.0.1:6379> ZRANGEBYSCORE testSet1 -inf inf
1) "b"
2) "a"
3) "d"
4) "c"

Delete data

ZREMRANGEBYSCORE testSet1 0 2

Implementation of delay queue

The general idea is very simple, that is, the score of each value saves time, that is, when adding an element, its score is the current time + delay time. When the data is obtained by rotation, the specific delay message is to find the data item less than or equal to the current time.

Another problem is that zrangebycore is different from the list pop, which takes out elements and deletes them in the list. Zrangebycore will only retrieve the data and will not delete it from the sorted set. Solution 1: using redis transaction, first extract the data from ZRANGEBYSCORE, and then delete the data with zrengebyscore.

Specific implementation code

Add a delay message. The parameter delay is how long we want to delay:

func (p *Producer) PublishDelayMsg(topicName string, body []byte, delay time.Duration) error {
    if delay <= 0 {
        return errors.New("delay need great than zero")
    }
    tm := time.Now().Add(delay)
    msg := NewMessage("", body)
    msg.DelayTime = tm.Unix()

    sendData, _ := json.Marshal(msg)
    return p.redisCmd.ZAdd(topicName+zsetSuffix, redis.Z{Score: float64(tm.Unix()), Member: string(sendData)}).Err()
}

Use, for example, we think about one second to deal with it

producer.PublishDelayMsg(topicName, body, time.Second)

Read and process messages
This is relatively simple. In a ticker, the data less than or equal to the current time is read circularly:

func (s *consumer) startGetDelayMessage() {
    go func() {
        ticker := time.NewTicker(s.options.RateLimitPeriod)
        defer func() {
            log.Println("stop get delay message.")
            ticker.Stop()
        }()
        topicName := s.topicName + zsetSuffix
        for {
            currentTime := time.Now().Unix()
            select {
            case <-s.ctx.Done():
                log.Printf("context Done msg: %#v \n", s.ctx.Err())
                return
            case <-ticker.C:
                var valuesCmd *redis.ZSliceCmd
                _, err := s.redisCmd.TxPipelined(func(pip redis.Pipeliner) error {
                    valuesCmd = pip.ZRangeWithScores(topicName, 0, currentTime)
                    pip.ZRemRangeByScore(topicName, "0", strconv.FormatInt(currentTime, 10))
                    return nil
                })
                if err != nil {
                    log.Printf("zset pip error: %#v \n", err)
                    continue
                }
                rev := valuesCmd.Val()
                for _, revBody := range rev {
                    msg := &Message{}
                    json.Unmarshal([]byte(revBody.Member.(string)), msg)
                    if s.handler != nil {
                        s.handler.HandleMessage(msg)
                    }
                }
            }
        }
    }()
}

Tags: Database Redis pip Unix less

Posted on Wed, 06 May 2020 01:05:18 -0400 by tomz0r