Redis

Redis is short for Remote Dictionary Service; It is also a Remote Dictionary Service;

Redis is a memory database, KV database and data structure database;

Redis is widely used, such as Twitter, Blizzard Entertainment, Github, Stack Overflow, Tencent, Alibaba, JD, Huawei, Sina Weibo, etc. many small and medium-sized companies are also using it;

Redis command view: http://redis.cn/commands.html

application

Record the number of friends' circle likes, comments and clicks( hash)
Record the list of circle of friends (sort), so as to quickly display the circle of friends( zset)
Record the title, abstract, author and cover of the article for list page display( hash)
Record like users in the circle of friends ID List, comments ID List for displaying and de counting( zset)
Cache hotspot data to reduce database pressure( hash)
If you talk to your circle of friends ID Is an integer id，Available redis To assign a circle of friends id((counter)( string)
Through collection( set)The intersection and union difference set operation is used to record the friend relationship( set)
In the game business, the record of each game is stored( list)

Installation and compilation

git clone https://gitee.com/mirrors/redis.git -b 6.2
cd redis
make
make test
make install
# The default installation is / usr/local/bin
# Redis server is a server program
# Redis cli is a client program

start-up

mkdir redis-data
# Copy redis.conf in the redis folder to redis data
# Modify redis.conf
# requirepass change password 123456
# daemonize yes
cd redis-data
redis-server redis.conf
# Access redis server through redis cli
redis-cli -h 127.0.0.1 -a 123456

Know Redis

redis storage structure  

value code in redis

string 

Character array. The string is a dynamic string raw. When the length of the string is less than 1M, the capacity will be doubled; if it exceeds 1M, only 1M will be expanded each time; the maximum length of the string is 512M;

Note: redis string is a binary security string. It can store binary data such as pictures and binary protocols;

Basic command

# Set the value of the key
SET key val
# Get value of key
GET key
# Perform an atomic plus one operation
INCR key
# Performs an atomic plus an integer operation
INCRBY key increment
# Perform atomic minus one
DECR key
# Performs the atomic subtraction of an integer
DECRBY key decrement
# If the key does not exist, it is equivalent to the SET command. When the key exists, nothing is done
SETNX key value
# Delete key val key value pair
DEL key
# Set or clear the bit value of the value (string) of the key at offset.
SETBIT key offset value
# Returns the bit value of the string corresponding to the key at offset
GETBIT key offset
# Count the number of bit s whose string is set to 1
BITCOUNT key

storage structure

If the string length is less than or equal to 20 and can be converted into an integer, int is used for storage;

If the string length is less than or equal to 44, embstr is used for storage;

If the string length is greater than 44, raw storage is used;

OBJECT encoding +key   You can see what the storage structure is

  interview

Why is the redis string storage less than or equal to 44 bytes of embstr type, while over 44 is raw type;

typedef struct redisObject {
unsigned type:4;
unsigned encoding:4;
unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or
* LFU data (least significant 8 bits
frequency
* and most significant 16 bits access time).
*/
int refcount;
void *ptr;
} robj;      redis be-all key value Are described with this structure
struct __attribute__ ((__packed__)) sdshdr8 {
uint8_t len; /* used */
uint8_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
#define OBJ_ENCODING_EMBSTR_SIZE_LIMIT 44
robj *createStringObject(const char *ptr, size_t len) {
if (len <= OBJ_ENCODING_EMBSTR_SIZE_LIMIT)
return createEmbeddedStringObject(ptr,len);
else
return createRawStringObject(ptr,len);
}

redisObject takes 16 bytes; sdshdr8 takes 3+x+1 bytes (1 is added after char buf [] to reserve one \ 0);

The redis memory allocator considers that a string larger than 64 bytes is a large string; therefore, the size of the small string is 64 - 16 - 3 - 1 = 44;

application

Object storage

SET role:10001 '{["name"]:"mark",["sex"]:"male",["age"]:30}'

GET role:10001

accumulator

# Add 1 to the total number of readings                 
incr reads                  //If there is no reads, first create reads = 0, and then call incr
# Cumulative plus 100
incrby reads 100

Distributed lock

# Lock
setnx lock 1
# Release lock
del lock

Bit operation

# Monthly check-in function 10001 user ID 202106 check-in in June 2021 the first day of June
setbit sign:10001:202106 1 1
# Calculate the attendance in June 2021
bitcount sign:10001:202106
# Get the check-in status on the second day of June 2021. 1 has checked in and 0 has not checked in
getbit sign:10001:202106 2

list

The two-way linked list is implemented. The time complexity of the first and last operations (deletion and addition) of the list is O(1); the time complexity of finding intermediate elements is O(n); the basis for whether the data in the list is compressed:           (if the data is too large, gzip compression will be performed.)

1. The element length is less than 48 and is not compressed;

2. The length difference of elements before and after compression shall not exceed 8, and they shall not be compressed;

Basic command

# Queue one or more elements from the left side of the queue
LPUSH key value [value ...]
# Pop an element from the left side of the queue
LPOP key
# Queue one or more elements from the right side of the queue
RPUSH key value [value ...]
# Pop an element from the right side of the queue
RPOP key
# Returns the element 0 between start and end from the queue. 1, 2 - 1 is the last - 2 is the penultimate - 3
LRANGE key start end
# Remove the element with the value of the previous count from the list stored in the key
LREM key count value
# It is a blocking version of RPOP, because this command blocks the connection when a given list cannot pop up any elements
BRPOP key timeout

storage structure

/* Minimum ziplist size in bytes for attempting compression. */
#define MIN_COMPRESS_BYTES 48
/* quicklistNode is a 32 byte struct describing a ziplist for a quicklist.
* We use bit fields keep the quicklistNode at 32 bytes.
* count: 16 bits, max 65536 (max zl bytes is 65k, so max count actually <
32k).
* encoding: 2 bits, RAW=1, LZF=2.
* container: 2 bits, NONE=1, ZIPLIST=2.
* recompress: 1 bit, bool, true if node is temporary decompressed for
usage.
* attempted_compress: 1 bit, boolean, used for verifying during testing.
* extra: 10 bits, free for future use; pads out the remainder of 32 bits */
typedef struct quicklistNode {
struct quicklistNode *prev;
struct quicklistNode *next;
unsigned char *zl;
unsigned int sz; /* ziplist size in bytes */
unsigned int count : 16; /* count of items in ziplist */
unsigned int encoding : 2; /* RAW==1 or LZF==2 */
unsigned int container : 2; /* NONE==1 or ZIPLIST==2 */
unsigned int recompress : 1; /* was this node previous compressed? */
unsigned int attempted_compress : 1; /* node can't compress; too small
*/
unsigned int extra : 10; /* more bits to steal for future usage */
} quicklistNode;
typedef struct quicklist {
quicklistNode *head;
quicklistNode *tail;
unsigned long count; /* total count of all entries in all
ziplists */
unsigned long len; /* number of quicklistNodes */
int fill : QL_FILL_BITS; /* fill factor for individual
nodes */
unsigned int compress : QL_COMP_BITS; /* depth of end nodes not to
compress;0=off */
unsigned int bookmark_count: QL_BM_BITS;
quicklistBookmark bookmarks[];
} quicklist;

application

Stack (FIFO)

LPUSH + LPOP
# perhaps
RPUSH + RPOP

Queue (FIFO)

LPUSH + RPOP
# perhaps
RPUSH + LPOP

blocking queue

LPUSH + BRPOP  +timeout
# perhaps
RPUSH + BLPOP  +timeout

Asynchronous message queue

The operation is the same as the queue, but between different systems;

Get fixed window record (record)

 

# In some business scenarios, you need to obtain a fixed number of records; for example, you need to obtain the last 50 records; these records need to be inserted first
 Return in sequence after;
lpush says '{["name"]:"xiaoming", ["text"]:"Happy children's Day!",
["picture"]:["url://image-20210601172741434.jpg", "url://image20210601172741435.jpg"], timestamp = 1231231230}'
lpush says '{["name"]:"xiaoli", ["text"]:"Happy children's Day!",
["picture"]:["url://image-20210601172742434.jpg", "url://image20210601172741436.jpg"], timestamp = 1231231231}'
lpush says '{["name"]:"xiaohua]", ["text"]:"Happy children's Day!",
["picture"]:["url://image-20210601172743434.jpg", "url://image20210601172741437.jpg"], timestamp = 1231231232}'
lpush says '{["name"]:"xiaoxin]", ["text"]:"Everything is just for the better you",
["picture"]:["url://image-20210601172744434.jpg", "url://image20210601172741438.jpg"], timestamp = 1231231233}'
lpush says '{["name"]:"xiaoai]", ["text"]:"hello 0Voice! hello
to better self", ["picture"]:["url://image-20210601172745439.jpg",
"url://image-20210601172741435.jpg"], timestamp = 1231231234}'
lpush says '{["name"]:"xiaomei", ["text"]:"2021 This year's students are awesome!",
["picture"]:["url://image-20210601172745434.jpg", "url://image20210601172741440.jpg"], timestamp = 1231231235}'
# Cut the last 5 records
ltrim says 0 4
lrange says 0 -1

The atomicity of commands needs to be guaranteed in actual projects, so lua scripts or pipeline commands are generally used;

-- redis lua script
local record = KEYS[1]
redis.call("LPUSH", "says", record)
redis.call("LTRIM", "says", 0, 4)

hash

Hash table, which contains this data structure in many high-level languages; c++ unordered_map quickly indexes value through key;

Basic command

# Get the value corresponding to the field in the key hash
HGET key field
# Set the value corresponding to the field in the key corresponding hash
HSET key field value
# Set multiple hash key value pairs
HMSET key field1 value1 field2 value2 ... fieldn valuen
# Gets the value of multiple field s
HMGET key field1 field2 ... fieldn
# Add an integer value to the value corresponding to the field in the key corresponding hash
HINCRBY key field increment
# Get the number of key value pairs in the hash corresponding to the key
HLEN key
# Delete the key value pair of the hash corresponding to the key. The key is field
HDEL key field

storage structure

If the number of nodes is greater than 512 (hash Max ziplist entries) or the length of all strings is greater than 64 (hash Max ziplistvalue), use dict;

If the number of nodes is less than or equal to 512 and the length of a string is less than 64, use ziplost to implement it;

application

Storage object

hmset hash:10001 name mark age 18 sex male
# Compare with string
set hash:10001 '{["name"]:"mark",["sex"]:"male",["age"]:18}'
# Suppose the age of the modified mark is 19
# hash: 
hset hash:10001 age 19
# string:
get role:10001
# Call json to decrypt the obtained string, take out the field and modify the age value
# Then call json encryption
set role:10001 '{["name"]:"mark",["sex"]:"male",["age"]:19}'

Shopping Cart

# Use user id as key
# Commodity id as field
# Quantity of goods as value
# Note: these items are displayed in the order we add them;

# Add item:
    hset MyCart:10001 40001 1
    lpush MyItem:10001 40001
# Increase quantity:
    hincrby MyCart:10001 40001 1
    hincrby MyCart:10001 40001 -1 // Reduce quantity 1
# Show all item quantities:
    hlen MyCart:10001
# Delete item:
    hdel MyCart:10001 40001
    lrem MyItem:10001 1 40001
# Get all items:
    lrange MyItem:10001
# 40001 40002 40003
    hget MyCart:10001 40001
    hget MyCart:10001 40002
    hget MyCart:10001 40003

set

Set; used to store unique fields, which does not require order;

Basic command

# Add one or more specified member elements to the key of the collection
SADD key member [member ...]
# Calculate the number of collection elements
SCARD key
# SMEMBERS key
SMEMBERS key
# Returns whether the member member is a member of the stored collection key
SISMEMBER key member
# Randomly return one or more elements in the key set without deleting them
SRANDMEMBER key [count]
# Removes and returns one or more random elements from the collection stored in the key
SPOP key [count]
# Returns the element of the difference between a set and a given set
SDIFF key [key ...]
# Returns the intersection of members of all specified sets
SINTER key [key ...]
# Returns the of a given collection and all members of the collection
SUNION key [key ...]

storage structure

If all elements are integers and the number of nodes is less than or equal to 512 (set Max int set entries), the integer array is used for storage;

If one of the elements is not an integer or the number of nodes is greater than 512, the dictionary is used for storage;

application

luck draw

# Add raffle user
sadd Award:1 10001 10002 10003 10004 10005 10006
sadd Award:1 10009
# View all lottery users
smembers Award:1
# Extract multiple lucky users
srandmember Award:1 10
# How to select one first prize, two second prizes and three third prizes?

Common concern

sadd follow:A mark king darren mole vico
sadd follow:C mark king darren
sinter follow:A follow:C

Recommend friends

sadd follow:A mark king darren mole vico
sadd follow:C mark king darren
# C possible acquaintances:
sdiff follow:A follow:C

zset

Ordered set; Used to realize ranking; It is an ordered structure;

Basic command

# Added to the sorted set with key
ZADD key [NX|XX] [CH] [INCR] score member [score member ...]
# Delete the key value pair of member from the ordered set with key as key
ZREM key member [member ...]
# Returns the score value of the member in the ordered set key
ZSCORE key member
# Add increment to the score value of the member of the ordered set key
ZINCRBY key increment member
# Returns the number of ordered set elements of the key
ZCARD key
# Returns the ranking of member s in the ordered set key
ZRANK key member
# Returns the specified range of elements stored in the ordered collection key
ZRANGE key start stop [WITHSCORES]
# Returns the members in the specified interval in the ordered set key (in reverse order)
ZREVRANGE key start stop [WITHSCORES]

storage structure

If the number of nodes is greater than 128 or the length of a string is greater than 64, a skip list is used;

If the number of nodes is less than or equal to 128 (Zset Max ziplost entries) and the length of all strings is less than or equal to 64 (Zset maxziplost value), ziplost storage is used;

application

Baidu hot list

 

# Click news:
zincrby hot:20210203 1 10001
zincrby hot:20210203 1 10002
zincrby hot:20210203 1 10003
zincrby hot:20210203 1 10004
zincrby hot:20210203 1 10005
zincrby hot:20210203 1 10006
zincrby hot:20210203 1 10007
zincrby hot:20210203 1 10008
zincrby hot:20210203 1 10009
zincrby hot:20210203 1 10010
# Get leaderboard:
zrevrange hot:20210203 0 9 withscores

Delay queue

Serialize the message into a string as the member of zset; The expiration processing time of this message is regarded as a score, and then multiple threads poll zset to obtain the expired tasks for processing.

def delay(msg):
    msg.id = str(uuid.uuid4()) #Ensure member uniqueness
    value = json.dumps(msg)
    retry_ts = time.time() + 5 # Retry after 5s
    redis.zadd("delay-queue", retry_ts, value)
# Use connection pool
def loop():
    while True:
        values = redis.zrangebyscore("delay-queue", 0, time.time(), start=0,
        num=1)
        if not values:
            time.sleep(1)
            continue
        value = values[0]
        success = redis.zrem("delay-queue", value)
        if success:
            msg = json.loads(value)
            handle_msg(msg)
# Disadvantages: loop is a multi-threaded competition. Both threads get data from zrangebyscore, but zrem succeeds and loses
 Defeat,
# Optimization: to avoid unnecessary operations, you can use lua script atom to execute these two commands
# Solution: funnel current limiting

  The producer hash es the scheduled tasks to different redis entities, and assigns a dispatcher process to each redis entity to periodically obtain the timeout events in redis and publish them to different consumers;

Time window current limiting

The system limits that a user's behavior can only occur N times in a specified time;

# Specify user_ A behavior action of ID can only occur for the number of times during a specific time period
max_count
def is_action_allowed(userid, action, period, max_count):
    key = 'hist:%s:%s' % (userid, action)
    now_ts = int(time.time()*1000) # Millisecond timestamp
    with client.pipeline() as pipe:
        # Record behavior
        pipe.zadd(key, now_ts, now_ts)
        # Remove the behavior records before the time window, and the rest are in the time window
        pipe.zremrangebyscore(key, 0, now_ts - period * 1000)
        # Gets the number of behaviors in the time window
        pipe.zcard(key)
        # Set the expiration time to prevent cold users from continuously occupying memory. The length of the time window is + 1 second
        pipe.expire(key, period + 1)
        _,_,current_count,_ = pipe.execute()
    return current_count <= max_count
can_reply = is_action_allowed(10001, "replay", 60, 5)
if can_reply:
    do_reply()
else:
    raise ActionThresholdOverflow()
# Maintain a time window, clear all records outside the window, and only keep records in the window;
# Disadvantages: the data in all time windows are recorded. If this amount is large, it is not suitable for such current limiting;