Non relational redis database

Article directory

redis introduction

NoSQL(NoSQL = Not Only SQL), which means "not only SQL", generally refers to non relational databases. The production of NoSQL database is to solve the challenges brought by large-scale data collection and multiple data types, especially the application problems of big data, including the storage of large-scale data.
The abbreviation of Redis(Remote Dictionary Server) remote dictionary data service is a memory cache database developed by Italians. It is a log database and key value database written in ANSI C language, supporting network, memory based and persistent. It also provides multi language API and non relational database of multi language API.
Data loading speed: hard disk < memory < cache
There are two types of caching:
• page caching is often used in CMS (content management system).
Data cache is often used in the specific data of the page.
If the data will not change in a short period of time, and is frequently accessed, in order to improve the user's request speed and reduce the load of the website, put the data on a medium with faster reading speed, called data cache. The media can be files, databases, and memory. Memory is often used for data caching.
Comparison between Redis and Memcache:

Redis persistence:
Redis is a memory database. Compared with traditional MySQL, Oracle and other relational databases, which directly save the content to the hard disk, the read-write efficiency of the memory database is much faster than that of the traditional database (the read-write efficiency of the memory is far greater than that of the hard disk). But saving in memory also brings a disadvantage. Once the power is off or down, all the data in the memory database will be lost.
Goal: persistence is to write the data in memory to the disk, so as to prevent the memory data loss due to service failure. Redis provides two types of persistence: RDB (default) and AOF.
RDB:
RDB is a method used by redis for persistence. It writes the data set Snapshot in the current memory to disk, that is, Snapshot snapshot Snapshot (all key value pairs of data in the database). During recovery, the Snapshot file is read directly into memory. We can configure redis to automatically take a Snapshot if more than m keys are modified in n seconds. The following is the default Snapshot saving configuration:
dbfilename dump.rdb the name of the persistent storage file is dump.rdb
Save 900 1 ා if more than one key is modified in 900 seconds, initiate snapshot saving
Save 300 10 ා 300 seconds. If more than 10 key s are modified, initiate snapshot saving
Save 60 10000 ා 600 seconds. If more than 10000 key s are modified, initiate snapshot saving
AOF:
• RDB persistent storage is backed up once in a certain period of time. If redis fails unexpectedly, all the changes after the last snapshot (data loss) will be lost. • working principle: when redis is restarted, the contents of the entire database will be rebuilt in memory by executing the write command saved in the file again.

Tell redis through the configuration file when we want to force the os to write to disk through the fsync function. There are three ways:
(default: fsync once per second), compress / optimize the command through BGREWRITEAOF instruction
appendonly yes / / enable aof persistence mode / / write to the disk immediately after receiving the write command, which is the slowest, but guarantees full persistence
appendfsync always / / writes to disk every second, making a good compromise between performance and persistence
appendfsync everysec / / fully dependent on os, best performance, no guarantee of persistence
# appendfsync no
Redis

Redis architecture mode:
Master slave replication
The replication function of Redis allows users to create any number of replicas of the server according to a Redis server, including the replicated server as the master server and the server replica created through replication as the slave server. As long as the network connection between the master and slave servers is normal, the master and slave servers will have the same data, and the master server will always synchronize the data updates on itself to the slave server, so as to ensure the same data of the master and slave servers.
Features: 1. master/slave role 2. The master/slave data is the same 3. Reduce the reading pressure of the master in the transfer from database
Problem: 1. High availability is not guaranteed 2. The pressure of master writing is not solved

Advantages of redis:

  • In the data storage memory, do the cache operation, first find the data in the cache, the cache does not go to the hard disk to find.

  • Data persistence: search and view in memory, and the hard disk also saves a copy.

  • Support multiple data types, master-slave mode, high availability, value storage 1G maximum

redis installation

  • Installation under linux (with redis installation package):
tar  xf Installation package
cd redis-5.0.5
make #Compile
make install #install
cd utils/
./install_server.sh #Execution script
ps -A | grep redis
redis-cli #The client logs in to redis service port 6379
# (at this time, the redis server has been brought out. If it is not opened, the redis server will start the redis service.)
  • Installation under windows (with redis installation package):
    1) Extract the installation package to the specified directory and add it to the Path of the system environment variable
    2) In windows power shell, execute under the directory of the unzipped installation package:
#Deploy redis as a service under windows 
.\redis-server --service-install redis.windows.conf  
#Start redis temporary server
redis-server redis.windows.conf

In another shell:

#Client access redis server
redis-cli.exe -h 127.0.0.1 -p 6379 
  • pycharm connection redis
    In Terminal:
    pip install redis
    import redis

Redis built in data type


redis application scenario

Redis string common operations

set age 2 # String name is age value value value is 2
get age # "2" view the value value corresponding to age
set name fentiao
getrange name 0 3 #Get the value "fent" with the value index of 0-3 corresponding to the name key
setrange name 2 westos # Insert the character westos in the value value value fentao corresponding to the name key with the index of 2 places
append name -westos #Append - westos to the value value of fentiao corresponding to the name key
mget #Simultaneous acquisition
mset #Simultaneous setting
strlen name #Get the length of value corresponding to the name
incr  age #Increase age value by 1
get age #See
incrgy age 10 #Age increases by 10 at a time
decr age #Age reduction 1
decrby age 10  #Age reduction 10
getset name westos #If westos has it, get it or set it
get name #See

Small case: python implements the connection operation of redis database

import redis
redis_client=redis.StrictRedis(host='127.0.0.1',port=6379) #Instantiate the object client and log in to redis
redis_client.mset({'name':'westos','age':10}) #Set key value pair at the same time
name=redis_client.get('name').decode('utf-8')#Change the value value corresponding to the name key to bits type to string type
name_len=redis_client.strlen('name') #Find the length of the value value corresponding to the name key
print('user name:  ',name,'User name length:',name_len)
redis_client.incr('age') #Increase the age by one year
print('age: ',redis_client.get('age').decode('utf-8'))#Change the value value corresponding to age key to bits type to string type


#Execution result:
user name:   westos User name length: 6
age:  11

Small case: generate the verification code sent to the user. The verification code is limited to generate once within 3s, and cannot be acquired for the second time within 3s

import time
import string
import random
import redis
def send_msg(phone):
    """ Simulate sending verification code to mobile phone"""
    num_list=random.sample(string.digits,4) #Randomly generate four numbers
    nums_code=''.join(num_list) #Combine four randomly generated numbers
    print('%s Verification Code:%s'%(phone,nums_code))
    return nums_code #Return the generated verification code
def phone_code(phone):
    """3s You can send the verification code again"""
    client=redis.Redis() #Instantiate an object
    is_exist = client.exists(phone)
    #You can query from the cache that the verification code has been sent
    if is_exist:#If the value corresponding to the phone queried from the cache is not empty, a verification code has been generated for the phone
        print('Verification code sent frequently please wait')
        return False #Exit function
    else:
        code=send_msg(phone) #The generated verification code returned from the function of generating verification code is saved in the code
        client.set(phone,code,3)#Put the phone and the corresponding verification code into the redis cache. ex=3 means to save 3s in redis cache
        print('send ok')
if __name__ == '__main__':
    print('first send:  ')
    phone_code('110')
    print('secend send:  ')
    phone_code('110')
    time.sleep(4) #4 s after the second transmission
    print('third send:  ')
    phone_code('110')


#Execution result:
first send:  
110 Verification Code: 5103
send ok
secend send:  
//Verification code sent frequently please wait
third send:  
110 Verification Code: 7628
send ok

Redis list common operations

LPUSH names name1,name2,name3 # : insert a list named name1,name2,name3 on the left
LRANGE names 0 3 #View values for list index 0-3 
RPUSH #Right insertion
LPOP #Delete the left element and pop up the deleted element
RPOP  #Delete the element on the right and pop up the deleted element
BLPOP names 10 #Pop up the element on the left. If the current list is empty, you will wait for 10s all the time. If you wait for 10s and there are no elements, you will exit
LPUSH westos #Insert westos in another terminal, and the element westos on the left will pop up in BLPOP names 10 within 10s

Small case: Redis's application of message queuing

class RedisQueue(object):
    def __init__(self,name,**conf):
        import redis
        self.__client=redis.Redis(**conf)
        self.key=name
    def qsize(self):
        return self.__client.llen(self.key) #Return queue length
    def put(self,item):
        """Team entry operation"""
        self.__client.rpush(self.key,item)#redis inserts the element item into the scores list
    def get(self,timeout=5):
        """Get the team head element. If it doesn't arrive, the waiting time is 5 s"""
        item=self.__client.blpop(self.key,timeout=timeout)
        return item
    def get_nowait(self):
        """Get the team head element. If it doesn't arrive, the waiting time is 0 s"""
        item=self.__client.blpop(self.key) #blpop do not write time default wait time is 0
        return item
if __name__ == '__main__':
    q=RedisQueue('scores') #The instantiated object passes in the list name of scores in redis
    for i in range(10):
        q.put(i) #Put element i into the list with name scores
    while True:
        result=q.get(timeout=5) #Get the elements of the list from the left side and wait for 5s to see if any new elements are added within 5s, and then print them out.
        print(result)
        if not result:
            break


#Execution result:
(b'scores', b'0')
(b'scores', b'1')
(b'scores', b'2')
(b'scores', b'3')
(b'scores', b'4')
(b'scores', b'5')
(b'scores', b'6')
(b'scores', b'7')
(b'scores', b'8')
(b'scores', b'9')

Small case: limit the number of IP accesses per minute to no more than 60

def IP_limit(IP):
    """
    //Limit the number of visits per minute to no more than 60
    key:value= IP:count
    :param IP:
    :return:
    """
    import redis
    client=redis.StrictRedis()
    if client.exists(IP):
        count=client.get(IP) #Assign value to count
        if int(count) >= 60:
            print('%s Frequent visits'%(IP))
        else:
            client.incr(IP) #Number of visits plus 1
            print('%s +1'%(IP))
    else:
        client.set(IP,1,60) #IP remains in memory for 60s
        print('%sfirst in: '%(IP))
if __name__ == '__main__':
    IP_limit('127.0.0.1') #Pass an IP to the IP limit function to indicate access once
    for i in range(100):
        IP_limit('172.25.254.197') #172.25.254.197 10 visits

This machine connects to the redis Library of another host

This machine: 172.25.254.18
Another host: 172.25.254.12
The setting steps are as follows:
172.25.254.12 set redis configuration information:
CD / etc / redis / 6379.conf - enter the redis configuration file to modify the information
bind 0.0.0.0 - any IP can be linked to the redis Library of 12 hosts
protected-mode no
Save exit
cd
redis-server /etc/redis/6379.conf
#Open redis in the / etc/redis/6379.conf file
172.25.254.18 test:
Redis-cli-h 172.25.254.12 ා login 12's redis

python and redis

Use of interaction once: pipline
Buffer multiple commands and execute them at one time to improve efficiency

redis_client = redis.StrictRedis(host='127.0.0.1', port=6379)
pipe = redis_client.pipeline()
pipe.set('name', 'westos')
pipe.set('name', 'westos')
pipe.set('name', 'westos')
pipe.set('name', 'westos')
pipe.set('name', 'westos')
pipe.execute()
print(redis_client.get('name'))

#results of enforcement
b'westos'

encapsulation
• the connection to the redis server is consistent
• encapsulate reading and writing of string type

How mail information transfer works
SMTP protocol: Simple Mail Transfer Protocol, which provides reliable and effective e-mail transmission. SMTP is a mail service established on FTP file transfer service. It is mainly used for mail information transmission between systems and provides notification of letters. POP3 protocol: Post O ffi ce Protocol - Version 3, which is mainly used to support remote management of e-mail on the server by the client.
Protocol for users to read mail: POP3

import smtplib
from email.mime.text import MIMEText
from email.utils import formataddr
# Set the suffix of server, user name, password and mailbox
smtp_server = "smtp.163.com"
from_username = 'Tencent Cloud Service Center'
mail_user = "18009211109@163.com"
#Enable the authorization code of smtp
mail_password ='scq123'
#Message subject prefix
mail_prefix = "[Operation and Development Department]-"
def send_mail(to_addrs, subject,msg):
   """
  //Send mail
  :param to_addrs: Mail recipient
  :param subject: Mail title
  :param content: Message body content
  :return: Bool
  """
   # msg = MIMEText(content)
   # Create an instance with attachments
   try:
       msg = MIMEText(msg) #Encapsulate the text message to be sent with MINME
       # Bind message with sender, address, subject
       msg['From'] = formataddr([from_username, mail_user]) #Format the mail user as ROM username, and the qq receiver will not display the original address name
       msg['To'] = to_addrs
       msg['Subject'] = mail_prefix + subject
        #Instantiate smtp object
       server = smtplib.SMTP()
       # Connect to mail server 163 - > QQ
       server.connect(smtp_server)
       # Sign in
       server.login(mail_user, mail_password)
       # Send message content
       server.sendmail(mail_user, to_addrs, msg.as_string())
       # Close connection
       server.quit()
   except Exception as  e:
       print(str(e))
       return False
   else:
       return True

if __name__ == '__main__':
    send_mail('763386638@qq.com', 'Winter vacation operation', 'The real warrior, dare to face the bleak life, dare to face the dripping blood')  
     #mag can't be unreasonable, otherwise it can't be sent out
    print("Send successfully.....")
Oh
Published 43 original articles, won praise 4, visited 918
Private letter follow

Tags: Redis Database snapshot Windows

Posted on Wed, 15 Jan 2020 07:41:01 -0500 by binto