redis programming for python learning

Redis introduction

The abbreviation of Redis(Remote Dictionary Server) remote dictionary data service, which is provided by Italy
What we developed is a memory cache database. It is written in ANSI C language, supports network, and can be based on
It can also store log and key value databases that can be persisted, and provide API s in multiple languages and languages
Non relational database of API.

Comparison between Redis and Memcache?

Redis installation

  1. First, download the Redis compressed package on the official website, address http://redis.io/download
  2. Compress the package to decompress and compile
    #tar xzf redis-x.x.x.tar.gz
    #cd redis-x.x.x/
    #make && make install
    #sh utils/install_server.sh
    Redis deployment
  3. Execute the Redis server command to start the Redis service
    redis-server
  4. Client redisClient
    redis-cli
    Log on to redis

    set 'a' '123'

  5. After adding the key value, it is found that a file named dump.rdb is created in the current running directory
    Pieces for persistent data storage

    Redis built in data type

    • string types are binary safe. It can contain any data (eg: jpg picture or serialized object).
    From the perspective of internal implementation, string can be regarded as a byte array, with the maximum limit of 1G bytes.
    The hash type is a mapping table of field and value of string type. Its addition and deletion operations are
    O (1) (average).
    • the list type is a two-way linked list of string type. The maximum length is (2 32).
    • set type is an unordered set of string type implemented by hash table. Add, delete, and find copies of
    The heterozygosity is O(1). The maximum length is (2 32).
    Redis built in instructions: http://doc.redisfans.com/

    Redis application scenario

    Redis persistence

    Redis is a memory database, which directly saves the content to the hard disk with traditional MySQL,Oracle and other relational databases
    Compared with the traditional database, the read-write efficiency of the memory database is much faster (the read-write efficiency of the memory is far greater than that of the hard disk
    Efficiency). But saving in memory also brings a disadvantage. Once the power is off or down, the
    All data will be lost.
    Goal: persistence is to write the data in memory to the disk, so as to prevent the memory data loss due to service failure.
    Redis provides two persistence methods: RDB (default) and AOF
    Redis persistence: RDB
    RDB is a method used by Redis for persistence. It writes the snapshot of the data set in the current memory
    Disk, that is, Snapshot snapshot (all key value pair data in the database). Recovery will be fast
    Read the file directly into memory.

    Redis persistence: RDB

    We can configure redis to take snapshots automatically if more than m key s are modified in n seconds. The following is the default snapshot protection
    Storage configuration:
    dbfilename dump.rdb
    save 900 1
    save 300 10
    #The name of the persistent storage file is dump.rdb
    #If more than one key is modified in 900 seconds, snapshot saving is initiated
    #If more than 10 key s are modified in 300 seconds, snapshot protection will be initiated
    Deposit
    Save 60 10000 ා 600 seconds. If more than 10000 key s are modified, initiate snapshot saving
    Redis persistence: AOF
    • RDB persistent storage is backed up once in a certain period of time, if
    If redis fails unexpectedly, it will lose the last snapshot
    All modifications of (data loss).
    How it works: when redis is restarted, it will execute the file again
    Saved write commands to rebuild the contents of the entire database in memory.

    Redis persistence: AOF

    Tell redis through the configuration file when we want to force the os to write to disk through the fsync function. There are three
    One way is as follows (the default is fsync once per second). Compress / optimize the command through BGREWRITEAOF instruction
    //Enable aof persistence mode
    appendonly yes
    //Write to disk immediately after receiving the write command, which is the slowest, but guarantees complete persistence
    #appendfsync always
    //Write to disk every second, making a good compromise between performance and persistence
    appendfsync everysec
    //Fully dependent on os, best performance, no guarantee of persistence
    #appendfsync no
    Configuration: modifying redis configuration file
    vim /etc/redis/6379.conf
    appendonly yes modify appendonly to yes
    CD / var / lib / redis / 6379 / ා enter the directory to generate aof files
    ls
    appendonly.aof dump.rdb
    python inserts data in batches and finds that the file size of appendonly.aof changes

import  redis
redis_client = redis.StrictRedis(host='localhost',port=6379)
redis_client.mset({'name':'westos','age':10})
name = redis_client.get('name').decode('utf-8')
name_len = redis_client.strlen('name')
print('User name:',name,'User name length:',name_len)
age_add = redis_client.incr('age')
for i in range(1000):
    redis_client.incr('age')

Redis architecture mode

Single version
Features: simple
Problems: 1. Limited memory capacity; 2. Limited processing capacity; 3. High availability.

Master slave replication

Master slave replication
Redis's replication function allows users to create any number of the
A replica of a server, in which the replicated server is the master and created through replication
The server replica of is slave. As long as the network connection between the master and slave servers is normal, the master and slave
The two servers will have the same data, and the master server will always synchronize the data updates on itself to the
From the server, so that the data of the master and slave servers are always the same.
Characteristic:
1. master/slave role
2. Same master/slave data
3. Reduce the reading pressure of master in transfer from database
Question:

  1. High availability is not guaranteed
  2. Failed to solve the pressure of master writing
    Python and Redis


    encapsulation
    • the connection to the redis server is consistent
    • encapsulate reading and writing of string type


"""

01 redis data structure.py

import redis

redis_client = redis.StrictRedis(host='172.25.254.18', port=6379)

#Operation on string
"""
get
mget
set 
mset
incr
incrby
decr 
decrby
strlen
"""
redis_client.mset({'name':'westos', 'age':10})
name = redis_client.get('name').decode('utf-8')
name_len = redis_client.strlen('name')
print("User name: ", name, "User name length: ", name_len)
age_add = redis_client.incr('age')
print("The current age is: ", redis_client.get('age'))
for i in range(10000):
    redis_client.incr('age')

Limit the number of visits in a certain period of time. For example, the function of our login can obtain the verification code from the mobile phone, but the third party we send the verification code to uses how much is it,
You can't let him click all the time and send messages all the time. Even if the front-end js has been verified, it will be troublesome for some people to use fiddler to intercept and bypass the front-end,
At this time, you can use the incr command of redis and expire to make a solution

  1. The user requested our SMS verification code service, and the corresponding key (phone number) of the user
  2. If there is one in the cache, the verification code has been requested. For example, if my system parameter is set to send a SMS at least once in 60 seconds, it means that I have requested it again within 60 seconds, and return to False to block it
  3. Otherwise, it is the first time to request the SMS verification code or wait for the verification code of 60 requests again, send the verification code, put it into the redis cache and set the expiration time of 60 seconds
import time

def send_msg(phone):
    """Simulate sending verification code to mobile phone"""
    import string
    import random
    #['4', '1', '5', '9']
    nums_list = random.sample(string.digits, 4)
    nums_code = "".join(nums_list)
    print("%s Verification Code:%s[company]" %(phone, nums_code))
    return  nums_code

def phone_code(phone):
    """
    //The phone verification code is simulated and can be sent again after 3 seconds
    :param phone:
    :return:
    """
    import redis
    client = redis.StrictRedis()
    #You can query from the cache that the verification code has been sent.
    if client.get(phone):
        print("The verification code is sent frequently. Please try again later")
        return  False
    else:
        code = send_msg(phone)
        client.set(phone, code, 3)
        print("Send verification code successfully")

if __name__ == '__main__':
    print("First send")
    phone_code('110')
    print("Second send")
    phone_code('110')
    time.sleep(4)
    print("Third send")
    phone_code('110')"""
IP Restrict frequent access:
    //If you are running HTTP service and want to limit the access frequency of HTTP, you can use some stable tools,
    //For example: github.com/dip/tollbooth. However, if the application you build is relatively simple, you can also implement it yourself.
IP Restricted rules:
    //No more than 60 visits per minute.
"""

def ip_limit(IP):
    """
    //No more than 60 visits per minute.
    key: value = IP:count
    :param IP:
    :return:
    """
    import redis
    client = redis.StrictRedis()
    #1) . judge whether the server has been accessed;
    if client.exists(IP):
        count = client.get(IP).decode('utf-8')
        if int(count) >= 60:
            print("%s Frequent visits" %(IP))
        else:
            client.incr(IP)
            print("Visit+1")
    else:
        client.set(IP, 1, 60)
        print("%s First visit" %(IP))

if __name__ == '__main__':
    ip_limit('127.0.0.1')
    for i in range(100):
        ip_limit('172.25.254.197')

"""

02 redis data structure list.py

Redis: Memory cache database, message queuing(kafka,RabbitMQ)
#How to realize the design of snatching red packets in high concurrency scenarios.

import  queue

class RedisQueue(object):
    """On the left is the team head, and on the right is the message queue at the end"""
    def __init__(self, name, **conf):
        import redis
        self.__client = redis.Redis(**conf)
        self.key = name

    def qsize(self):
        return  self.__client.llen(self.key)

    def put(self, item):
        """Team entry operation"""
        self.__client.rpush(self.key, item)

    def get(self, timeout=5):
        """Get the counterpart. If it doesn't arrive, the waiting time is timeout"""
        item = self.__client.blpop(self.key, timeout=timeout)
        return  item

    def get_nowait(self):
        item = self.__client.blpop(self.key)
        return  item

if __name__ == '__main__':
    q = RedisQueue('scores', host='172.25.254.18', port=6380)
    for i in range(10):
        q.put(i)

    while True:
        result = q.get(timeout=100)
        print(result)
        if not result:
            break

06 sets and ordered sets.py

//Project practice: distributed cache service practice: Redis realizes leaderboard function
//The function of list is often used in web pages and apps to display a list of key values in descending order. When the concurrency of operations and queries is large,
//Using traditional database will encounter performance bottleneck, resulting in large delay. Using Redis, a distributed caching service, you can implement a
//The function of product hot list. Its advantages are:
    - The data is stored in the cache, and the speed of reading and writing is very fast.
    - Provide string( String),Linked list ( List),Set ( Set),Hash ( Hash)And so on.

"""
import random

def productSalesRankDemo():
    import redis
    import uuid
    PRODUCT_KINDS = 30
    host = "127.0.0.1"
    port = 6379
    password = ''
    key = "Top selling list"
    client = redis.StrictRedis(host=host, port=port, password=password)
    if not client.exists(key):
        products = {}
        for count in range(PRODUCT_KINDS):
            product_name = 'product-' + str(uuid.uuid4())
            product_sale = random.randint(1000, 20000)
            products[product_name] = product_sale
        client.zadd(key, products)

productSalesRankDemo()"""

07 pipeline optimizer.py

import redis

redis_client = redis.StrictRedis(host='172.25.254.18', port=6380)
pipe = redis_client.pipeline()
pipe.set('name', 'westos')
pipe.set('name', 'westos')
pipe.set('name', 'westos')
pipe.set('name', 'westos')
pipe.set('name', 'westos')
pipe.execute()
"""

08 send mail.py

#Set the suffix of server, user name, password and mailbox
import smtplib
from email.mime.text import MIMEText
from email.utils import formataddr

smtp_server = "smtp.163.com"
from_username = 'Technology Center'
mail_user = "xc@163.com"
#Is it the authorization to open smtp? It's not the real password.
mail_password = "westos123"
#Prefix of message subject
mail_prefix = "[Operation and Development Department]-"

def send_email(to_addrs, subject, msg):
    try:
        # Make MIME encapsulation for the text information to be sent
        msg = MIMEText(msg)
        #Format sender name
        msg['From'] = formataddr([from_username, mail_user])
        msg['To'] = to_addrs
        msg['Subject'] = mail_prefix + subject

        #1. Instantiate the smtp object
        server = smtplib.SMTP()
        #2. Connect to mail server
        server.connect(smtp_server)
        #3. login
        server.login(mail_user, mail_password)
        #4. Send email content
        server.sendmail(mail_user, to_addrs, msg.as_string())
        #5. Close the connection
        server.quit()
    except Exception as  e:
        print(str(e))
        return False
    else:
        return True

if __name__ == '__main__':
    send_email('976131@qq.com','Winter vacation operation', 'Please submit before March 2')
    print("Send successfully.....")

Tags: Python Redis Database snapshot network

Posted on Thu, 16 Jan 2020 10:02:55 -0500 by thebay