Redis
Redis is short for Remote Dictionary Service; It is also a Remote Dictionary Service;
Redis is a memory database, KV database and data structure database;
Redis is widely used, such as Twitter, Blizzard Entertainment, Github, Stack Overflow, Tencent, Alibaba, JD, Huawei, Sina Weibo, etc. many small and medium-sized companies are also using it;
Redis command view: http://redis.cn/commands.html
application
Record the number of friends' circle likes, comments and clicks( hash) Record the list of circle of friends (sort), so as to quickly display the circle of friends( zset) Record the title, abstract, author and cover of the article for list page display( hash) Record like users in the circle of friends ID List, comments ID List for displaying and de counting( zset) Cache hotspot data to reduce database pressure( hash) If you talk to your circle of friends ID Is an integer id,Available redis To assign a circle of friends id((counter)( string) Through collection( set)The intersection and union difference set operation is used to record the friend relationship( set) In the game business, the record of each game is stored( list)
Installation and compilation
git clone https://gitee.com/mirrors/redis.git -b 6.2 cd redis make make test make install # The default installation is / usr/local/bin # Redis server is a server program # Redis cli is a client program
start-up
mkdir redis-data # Copy redis.conf in the redis folder to redis data # Modify redis.conf # requirepass change password 123456 # daemonize yes cd redis-data redis-server redis.conf # Access redis server through redis cli redis-cli -h 127.0.0.1 -a 123456
Know Redis
redis storage structure
value code in redis
string
Character array. The string is a dynamic string raw. When the length of the string is less than 1M, the capacity will be doubled; if it exceeds 1M, only 1M will be expanded each time; the maximum length of the string is 512M;
Note: redis string is a binary security string. It can store binary data such as pictures and binary protocols;
Basic command
# Set the value of the key SET key val # Get value of key GET key # Perform an atomic plus one operation INCR key # Performs an atomic plus an integer operation INCRBY key increment # Perform atomic minus one DECR key # Performs the atomic subtraction of an integer DECRBY key decrement # If the key does not exist, it is equivalent to the SET command. When the key exists, nothing is done SETNX key value # Delete key val key value pair DEL key # Set or clear the bit value of the value (string) of the key at offset. SETBIT key offset value # Returns the bit value of the string corresponding to the key at offset GETBIT key offset # Count the number of bit s whose string is set to 1 BITCOUNT key
storage structure
If the string length is less than or equal to 20 and can be converted into an integer, int is used for storage;
If the string length is less than or equal to 44, embstr is used for storage;
If the string length is greater than 44, raw storage is used;
OBJECT encoding +key You can see what the storage structure is
interview
Why is the redis string storage less than or equal to 44 bytes of embstr type, while over 44 is raw type;
typedef struct redisObject { unsigned type:4; unsigned encoding:4; unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or * LFU data (least significant 8 bits frequency * and most significant 16 bits access time). */ int refcount; void *ptr; } robj; redis be-all key value Are described with this structure struct __attribute__ ((__packed__)) sdshdr8 { uint8_t len; /* used */ uint8_t alloc; /* excluding the header and null terminator */ unsigned char flags; /* 3 lsb of type, 5 unused bits */ char buf[]; }; #define OBJ_ENCODING_EMBSTR_SIZE_LIMIT 44 robj *createStringObject(const char *ptr, size_t len) { if (len <= OBJ_ENCODING_EMBSTR_SIZE_LIMIT) return createEmbeddedStringObject(ptr,len); else return createRawStringObject(ptr,len); }
redisObject takes 16 bytes; sdshdr8 takes 3+x+1 bytes (1 is added after char buf [] to reserve one \ 0);
The redis memory allocator considers that a string larger than 64 bytes is a large string; therefore, the size of the small string is 64 - 16 - 3 - 1 = 44;
application
Object storage
SET role:10001 '{["name"]:"mark",["sex"]:"male",["age"]:30}'
GET role:10001
accumulator
# Add 1 to the total number of readings incr reads //If there is no reads, first create reads = 0, and then call incr # Cumulative plus 100 incrby reads 100
Distributed lock
# Lock setnx lock 1 # Release lock del lock
Bit operation
# Monthly check-in function 10001 user ID 202106 check-in in June 2021 the first day of June setbit sign:10001:202106 1 1 # Calculate the attendance in June 2021 bitcount sign:10001:202106 # Get the check-in status on the second day of June 2021. 1 has checked in and 0 has not checked in getbit sign:10001:202106 2
list
The two-way linked list is implemented. The time complexity of the first and last operations (deletion and addition) of the list is O(1); the time complexity of finding intermediate elements is O(n); the basis for whether the data in the list is compressed: (if the data is too large, gzip compression will be performed.)
1. The element length is less than 48 and is not compressed;
2. The length difference of elements before and after compression shall not exceed 8, and they shall not be compressed;
Basic command
# Queue one or more elements from the left side of the queue LPUSH key value [value ...] # Pop an element from the left side of the queue LPOP key # Queue one or more elements from the right side of the queue RPUSH key value [value ...] # Pop an element from the right side of the queue RPOP key # Returns the element 0 between start and end from the queue. 1, 2 - 1 is the last - 2 is the penultimate - 3 LRANGE key start end # Remove the element with the value of the previous count from the list stored in the key LREM key count value # It is a blocking version of RPOP, because this command blocks the connection when a given list cannot pop up any elements BRPOP key timeout
storage structure
/* Minimum ziplist size in bytes for attempting compression. */ #define MIN_COMPRESS_BYTES 48 /* quicklistNode is a 32 byte struct describing a ziplist for a quicklist. * We use bit fields keep the quicklistNode at 32 bytes. * count: 16 bits, max 65536 (max zl bytes is 65k, so max count actually < 32k). * encoding: 2 bits, RAW=1, LZF=2. * container: 2 bits, NONE=1, ZIPLIST=2. * recompress: 1 bit, bool, true if node is temporary decompressed for usage. * attempted_compress: 1 bit, boolean, used for verifying during testing. * extra: 10 bits, free for future use; pads out the remainder of 32 bits */ typedef struct quicklistNode { struct quicklistNode *prev; struct quicklistNode *next; unsigned char *zl; unsigned int sz; /* ziplist size in bytes */ unsigned int count : 16; /* count of items in ziplist */ unsigned int encoding : 2; /* RAW==1 or LZF==2 */ unsigned int container : 2; /* NONE==1 or ZIPLIST==2 */ unsigned int recompress : 1; /* was this node previous compressed? */ unsigned int attempted_compress : 1; /* node can't compress; too small */ unsigned int extra : 10; /* more bits to steal for future usage */ } quicklistNode; typedef struct quicklist { quicklistNode *head; quicklistNode *tail; unsigned long count; /* total count of all entries in all ziplists */ unsigned long len; /* number of quicklistNodes */ int fill : QL_FILL_BITS; /* fill factor for individual nodes */ unsigned int compress : QL_COMP_BITS; /* depth of end nodes not to compress;0=off */ unsigned int bookmark_count: QL_BM_BITS; quicklistBookmark bookmarks[]; } quicklist;
application
Stack (FIFO)
LPUSH + LPOP # perhaps RPUSH + RPOP
Queue (FIFO)
LPUSH + RPOP # perhaps RPUSH + LPOP
blocking queue
LPUSH + BRPOP +timeout # perhaps RPUSH + BLPOP +timeout
Asynchronous message queue
The operation is the same as the queue, but between different systems;
Get fixed window record (record)
# In some business scenarios, you need to obtain a fixed number of records; for example, you need to obtain the last 50 records; these records need to be inserted first Return in sequence after; lpush says '{["name"]:"xiaoming", ["text"]:"Happy children's Day!", ["picture"]:["url://image-20210601172741434.jpg", "url://image20210601172741435.jpg"], timestamp = 1231231230}' lpush says '{["name"]:"xiaoli", ["text"]:"Happy children's Day!", ["picture"]:["url://image-20210601172742434.jpg", "url://image20210601172741436.jpg"], timestamp = 1231231231}' lpush says '{["name"]:"xiaohua]", ["text"]:"Happy children's Day!", ["picture"]:["url://image-20210601172743434.jpg", "url://image20210601172741437.jpg"], timestamp = 1231231232}' lpush says '{["name"]:"xiaoxin]", ["text"]:"Everything is just for the better you", ["picture"]:["url://image-20210601172744434.jpg", "url://image20210601172741438.jpg"], timestamp = 1231231233}' lpush says '{["name"]:"xiaoai]", ["text"]:"hello 0Voice! hello to better self", ["picture"]:["url://image-20210601172745439.jpg", "url://image-20210601172741435.jpg"], timestamp = 1231231234}' lpush says '{["name"]:"xiaomei", ["text"]:"2021 This year's students are awesome!", ["picture"]:["url://image-20210601172745434.jpg", "url://image20210601172741440.jpg"], timestamp = 1231231235}' # Cut the last 5 records ltrim says 0 4 lrange says 0 -1
The atomicity of commands needs to be guaranteed in actual projects, so lua scripts or pipeline commands are generally used;
-- redis lua script local record = KEYS[1] redis.call("LPUSH", "says", record) redis.call("LTRIM", "says", 0, 4)
hash
Hash table, which contains this data structure in many high-level languages; c++ unordered_map quickly indexes value through key;
Basic command
# Get the value corresponding to the field in the key hash HGET key field # Set the value corresponding to the field in the key corresponding hash HSET key field value # Set multiple hash key value pairs HMSET key field1 value1 field2 value2 ... fieldn valuen # Gets the value of multiple field s HMGET key field1 field2 ... fieldn # Add an integer value to the value corresponding to the field in the key corresponding hash HINCRBY key field increment # Get the number of key value pairs in the hash corresponding to the key HLEN key # Delete the key value pair of the hash corresponding to the key. The key is field HDEL key field
storage structure
If the number of nodes is greater than 512 (hash Max ziplist entries) or the length of all strings is greater than 64 (hash Max ziplistvalue), use dict;
If the number of nodes is less than or equal to 512 and the length of a string is less than 64, use ziplost to implement it;
application
Storage object
hmset hash:10001 name mark age 18 sex male # Compare with string set hash:10001 '{["name"]:"mark",["sex"]:"male",["age"]:18}' # Suppose the age of the modified mark is 19 # hash: hset hash:10001 age 19 # string: get role:10001 # Call json to decrypt the obtained string, take out the field and modify the age value # Then call json encryption set role:10001 '{["name"]:"mark",["sex"]:"male",["age"]:19}'
Shopping Cart
# Use user id as key # Commodity id as field # Quantity of goods as value # Note: these items are displayed in the order we add them; # Add item: hset MyCart:10001 40001 1 lpush MyItem:10001 40001 # Increase quantity: hincrby MyCart:10001 40001 1 hincrby MyCart:10001 40001 -1 // Reduce quantity 1 # Show all item quantities: hlen MyCart:10001 # Delete item: hdel MyCart:10001 40001 lrem MyItem:10001 1 40001 # Get all items: lrange MyItem:10001 # 40001 40002 40003 hget MyCart:10001 40001 hget MyCart:10001 40002 hget MyCart:10001 40003
set
Set; used to store unique fields, which does not require order;
Basic command
# Add one or more specified member elements to the key of the collection SADD key member [member ...] # Calculate the number of collection elements SCARD key # SMEMBERS key SMEMBERS key # Returns whether the member member is a member of the stored collection key SISMEMBER key member # Randomly return one or more elements in the key set without deleting them SRANDMEMBER key [count] # Removes and returns one or more random elements from the collection stored in the key SPOP key [count] # Returns the element of the difference between a set and a given set SDIFF key [key ...] # Returns the intersection of members of all specified sets SINTER key [key ...] # Returns the of a given collection and all members of the collection SUNION key [key ...]
storage structure
If all elements are integers and the number of nodes is less than or equal to 512 (set Max int set entries), the integer array is used for storage;
If one of the elements is not an integer or the number of nodes is greater than 512, the dictionary is used for storage;
application
luck draw
# Add raffle user sadd Award:1 10001 10002 10003 10004 10005 10006 sadd Award:1 10009 # View all lottery users smembers Award:1 # Extract multiple lucky users srandmember Award:1 10 # How to select one first prize, two second prizes and three third prizes?
Common concern
sadd follow:A mark king darren mole vico sadd follow:C mark king darren sinter follow:A follow:C
Recommend friends
sadd follow:A mark king darren mole vico sadd follow:C mark king darren # C possible acquaintances: sdiff follow:A follow:C
zset
Ordered set; Used to realize ranking; It is an ordered structure;
Basic command
# Added to the sorted set with key ZADD key [NX|XX] [CH] [INCR] score member [score member ...] # Delete the key value pair of member from the ordered set with key as key ZREM key member [member ...] # Returns the score value of the member in the ordered set key ZSCORE key member # Add increment to the score value of the member of the ordered set key ZINCRBY key increment member # Returns the number of ordered set elements of the key ZCARD key # Returns the ranking of member s in the ordered set key ZRANK key member # Returns the specified range of elements stored in the ordered collection key ZRANGE key start stop [WITHSCORES] # Returns the members in the specified interval in the ordered set key (in reverse order) ZREVRANGE key start stop [WITHSCORES]
storage structure
If the number of nodes is greater than 128 or the length of a string is greater than 64, a skip list is used;
If the number of nodes is less than or equal to 128 (Zset Max ziplost entries) and the length of all strings is less than or equal to 64 (Zset maxziplost value), ziplost storage is used;
application
Baidu hot list
# Click news: zincrby hot:20210203 1 10001 zincrby hot:20210203 1 10002 zincrby hot:20210203 1 10003 zincrby hot:20210203 1 10004 zincrby hot:20210203 1 10005 zincrby hot:20210203 1 10006 zincrby hot:20210203 1 10007 zincrby hot:20210203 1 10008 zincrby hot:20210203 1 10009 zincrby hot:20210203 1 10010 # Get leaderboard: zrevrange hot:20210203 0 9 withscores
Delay queue
Serialize the message into a string as the member of zset; The expiration processing time of this message is regarded as a score, and then multiple threads poll zset to obtain the expired tasks for processing.
def delay(msg): msg.id = str(uuid.uuid4()) #Ensure member uniqueness value = json.dumps(msg) retry_ts = time.time() + 5 # Retry after 5s redis.zadd("delay-queue", retry_ts, value) # Use connection pool def loop(): while True: values = redis.zrangebyscore("delay-queue", 0, time.time(), start=0, num=1) if not values: time.sleep(1) continue value = values[0] success = redis.zrem("delay-queue", value) if success: msg = json.loads(value) handle_msg(msg) # Disadvantages: loop is a multi-threaded competition. Both threads get data from zrangebyscore, but zrem succeeds and loses Defeat, # Optimization: to avoid unnecessary operations, you can use lua script atom to execute these two commands # Solution: funnel current limiting
The producer hash es the scheduled tasks to different redis entities, and assigns a dispatcher process to each redis entity to periodically obtain the timeout events in redis and publish them to different consumers;
Time window current limiting
The system limits that a user's behavior can only occur N times in a specified time;
# Specify user_ A behavior action of ID can only occur for the number of times during a specific time period max_count def is_action_allowed(userid, action, period, max_count): key = 'hist:%s:%s' % (userid, action) now_ts = int(time.time()*1000) # Millisecond timestamp with client.pipeline() as pipe: # Record behavior pipe.zadd(key, now_ts, now_ts) # Remove the behavior records before the time window, and the rest are in the time window pipe.zremrangebyscore(key, 0, now_ts - period * 1000) # Gets the number of behaviors in the time window pipe.zcard(key) # Set the expiration time to prevent cold users from continuously occupying memory. The length of the time window is + 1 second pipe.expire(key, period + 1) _,_,current_count,_ = pipe.execute() return current_count <= max_count can_reply = is_action_allowed(10001, "replay", 60, 5) if can_reply: do_reply() else: raise ActionThresholdOverflow() # Maintain a time window, clear all records outside the window, and only keep records in the window; # Disadvantages: the data in all time windows are recorded. If this amount is large, it is not suitable for such current limiting;