Redis5.x two persistence methods and master-slave replication configuration

Pay attention to the official account number: CoderBuff, reply to "redis" and get the complete version of Redis5.x tutorial PDF.

To configure

In addition to supporting a variety of storage types, redis is also very important. Although it is a memory based storage system, it can also perform data persistence operations. This is quite effective when the cache unfortunately goes down and wants to recover the cached data. Similarly, when we actually use redis, we will configure redis as a cluster master-slave mode for higher performance and availability. This chapter focuses on the configuration of persistence and master-slave replication.

Persistence

There are two ways to persist redis: snapshot persistence (RDB) and AOF persistence.

Snapshot persistence (RDB)

Snapshot persistence is the persistence of all data written to the hard disk at a certain time. Obviously, there is a problem of "when" to write to the hard disk. If the time interval is too long, the data will be lost if it goes down just before persistence. In other words, regardless of the timing of configuration persistence, there is a risk of data loss. As a result, snapshot persistence is suitable for scenarios where loss of some data will not cause problems.

To configure snapshot persistence, you can use commands or profiles directly.

configuration file

Return to the installation location of redis. According to the preparations in Chapter 1, redis is installed in the path of / usr/local/redis-5.0.7, and the redis.conf configuration file is modified. The configuration file contains persistent configuration, template plug-in, lua script and so on. We extract configuration information about snapshot persistence.

################################ SNAPSHOTTING  ################################

# save [seconds] [changes] means that there are [changes] key value changes in [seconds] seconds, then it is persisted. Multiple can be configured at the same time. If one condition is met, it will trigger.
save 900 1              #One key change in 900 seconds is persisted.
save 300 10             #10 key changes in 300 seconds are persisted.
save 60 10000           #10000 key changes in 60 seconds are persisted.

stop-writes-on-bgsave-error yes     #When there is an error in persistence, whether to stop data writing? Data writing is stopped by default. It can be configured as no and can still write to the cache data when there is an error in persistence.

rdbcompression yes      #Whether to compress data, the default is compression. Configurable as no, no compression.

rdbchecksum yes             #Check whether the persistent rdb file is checked by default. It can be configured as no without verification.

dbfilename dump.rdb     #Specifies the name of the file where the rdb is saved.

dir ./                              #Specify the directory saved by rdb, which is the installation directory of redis by default.

In redis, snapshot persistence is used by default. If you want to enable AOF persistence appendonly yes, the configuration of AOF persistence will be described below.

Note that although this is the default configuration file under the redis installation directory, we need to make the path of the configuration file when starting. If you use redis server at startup, you will be prompted as follows:

10768:C 08 Feb 2020 19:52:40.149 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf

So we terminate the redis service and add a save 101 configuration in the configuration file to specify the configuration file under the redis installation directory.

okevindeMacBook-Air:redis-5.0.7 okevin$ redis-server redis.conf 
14206:C 11 Feb 2020 23:58:52.589 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
14206:C 11 Feb 2020 23:58:52.589 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=14206, just started
14206:C 11 Feb 2020 23:58:52.589 # Configuration loaded

As you can see, the configuration file under the installation directory was loaded successfully.

Next, we will verify whether redis is "persistent if there is a key value change in 10 seconds". We enter the redis command line through redis cli to interact and write a piece of data. The following log contents appear in the redis startup window:

14234:M 12 Feb 2020 00:02:13.016 * 1 changes in 10 seconds. Saving...
14234:M 12 Feb 2020 00:02:13.017 * Background saving started by pid 14235
14235:C 12 Feb 2020 00:02:13.020 * DB saved on disk
14234:M 12 Feb 2020 00:02:13.121 * Background saving terminated with success

This means that the configuration is in effect.

To modify the configuration file directly, you need to restart the redis service. You can modify the configuration directly by command

127.0.0.1:6379> config set save "5 1"
OK

At this time, write a piece of data and verify in the redis startup window:

14206:M 12 Feb 2020 00:04:26.133 * 1 changes in 5 seconds. Saving...
14206:M 12 Feb 2020 00:04:26.134 * Background saving started by pid 14237
14232:C 12 Feb 2020 00:04:26.139 * DB saved on disk
14206:M 12 Feb 2020 00:04:26.238 * Background saving terminated with success

As you can see, we have changed the configuration of snapshot persistence dynamically. However, this dynamic configuration method will fail when redis is restarted.

command

In addition to snapshot persistence by configuration file, we can also persist the snapshot "at any time" by command. There are two commands available: bgsave and save. Bgsave, redis will create a child process, write the snapshot to the hard disk through the child process, and the parent process will continue to process the command request. Save is that redis will not respond to other commands before the snapshot is created, that is, blocking, which is not commonly used.

Use the bgsave command on the redis client:

127.0.0.1:6379> bgsave
Background saving started

Log on the redis server:

14234:M 12 Feb 2020 00:15:11.943 * Background saving started by pid 14245
14245:C 12 Feb 2020 00:15:11.948 * DB saved on disk
14234:M 12 Feb 2020 00:15:12.031 * Background saving terminated with success

Use the save command on the redis client:

127.0.0.1:6379> save
OK

Log on the redis server:

14234:M 12 Feb 2020 00:16:12.922 * DB saved on disk

Through the logs of redis server, we can find that the bottom layer of save configuration in the configuration file calls the bgsave command. When will the Save command be used?

That is, when the shutdown command is called, the save command will be called to block other commands and shut down the server after execution.

Call the shutdown command on the redis client:

127.0.0.1:6379> shutdown

Log on the redis server:

14234:M 12 Feb 2020 00:18:52.353 # User requested shutdown...
14234:M 12 Feb 2020 00:18:52.353 * Saving the final RDB snapshot before exiting.
14234:M 12 Feb 2020 00:18:52.356 * DB saved on disk
14234:M 12 Feb 2020 00:18:52.356 * Removing the pid file.
14234:M 12 Feb 2020 00:18:52.357 # Redis is now ready to exit, bye bye...

AOF persistence

Aof persistence is similar to the master-slave synchronization of binlog in MySQL. It records the executed command and writes the executed write command to the end of the AOF file. It also has the problem of persistence timing. Usually we will configure "perform synchronization once per second". Let's still have a visual experience through configuration.

configuration file

############################## APPEND ONLY MODE ###############################

appendonly no           #Whether to enable AOF persistence? It is off by default and yes is on. Before redis 4.0, RDB and AOF were not allowed to be mixed, but after that, mixed mode was allowed to be used, which was configured with the last two parameters.

appendfilename "appendonly.aof"         #The filename of the AOF persisted data.

appendfsync everysec        #The frequency of AOF persistence. By default, synchronization is performed once per second. There is also the "always" option, which means that each redis command needs to write to the hard disk synchronously, but this will seriously reduce the speed of redis. There is also the "no" option, where the timing of persistence will be determined by the operating system. Redis recommends using the "everysec" configuration if you don't know what to do.

no-appendfsync-on-rewrite no        #AOF files are not overridden by default.

#The following two configurations are used for AOF "override" triggers. When the volume of the AOF file is greater than 64m, and the volume of the AOF file is at least twice (100%) of the volume of the last rewritten file, execute the re command.
auto-aof-rewrite-percentage 100     
auto-aof-rewrite-min-size 64mb  

aof-load-truncated yes          #When redis recovers, it ignores the last instruction that may have problems. The default value is yes.

aof-use-rdb-preamble yes        #Whether to turn on the mixed mode of RDB and AOF? yes is on by default.

We modify the appendonly yes in the configuration file to turn on AOF persistence and restart the redis service.

What's slightly different from snapshot persistence is that after we write a piece of data in redis cli, the redis server does not record the persistence log of AOF. But when we start up, it will appear to load data from the AOF file.

15336:M 12 Feb 2020 14:21:42.541 # Server initialized
15336:M 12 Feb 2020 14:21:42.542 * DB loaded from append only file: 0.001 seconds
15336:M 12 Feb 2020 14:21:42.542 * Ready to accept connections

If we shut down the redis server and restart it at this time, we can still get the value just written because we have enabled persistence.

As with snapshot persistence, we can also dynamically modify the configuration file through config set appendonly "yes" on the command line.

In the previous configuration file, we mentioned the term "rewrite". It is actually aimed at the configuration of AOF files that are too large and need to be overwritten. Similar to bgsave, it is operated through a subprocess.

Master slave replication

For a large application using redis, we will configure the redis cluster to ensure the performance of redis. Similar to MySQL, the master of redis sends updates to multiple slave servers. The master server is responsible for writing and the slave server is responsible for reading to improve performance.

In this section, I will find another mac physical machine to implement redis master-slave replication in the same LAN. Of course, you can also experiment in the virtual machine.

Modify the configuration file of the master master server (in order to better demonstrate master-slave replication, make as few configurations as possible):

#bind 127.0.0.1         #By default, only local connection to redis service is allowed, so this line of configuration needs to be commented out.
protected-mode no       #Protection mode is turned on by default. Here we configure no to turn off.
daemonize no                #In practical applications, we usually run redis in the background, that is, we will configure it as yes. By default, no means it does not run in the background. We keep the default configuration here, which is convenient to view the related operation logs.

Modify the slave server's configuration file (only one more line than the master):

#bind 127.0.0.1         #By default, only local connection to redis service is allowed, so this line of configuration needs to be commented out.
protected-mode no       #Protection mode is turned on by default. Here we configure no to turn off.
daemonize no                #In practical applications, we usually run redis in the background, that is, we will configure it as yes. By default, no means it does not run in the background. We keep the default configuration here, which is convenient to view the related operation logs.

slaveof 192.168.31.125 6379

Let's start the master server to display the following information:

okevindeMacBook-Air:redis-5.0.7 okevin$ redis-server redis.conf 
15702:C 12 Feb 2020 22:18:24.777 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
15702:C 12 Feb 2020 22:18:24.778 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=15702, just started
15702:C 12 Feb 2020 22:18:24.778 # Configuration loaded
15702:M 12 Feb 2020 22:18:24.781 * Increased maximum number of open files to 10032 (it was originally set to 2560).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 5.0.7 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 15702
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

15702:M 12 Feb 2020 22:18:24.801 # Server initialized
15702:M 12 Feb 2020 22:18:24.803 * DB loaded from append only file: 0.002 seconds
15702:M 12 Feb 2020 22:18:24.803 * Ready to accept connections

Restart the slave to display the following information from the server:

t4f-mbp-17346:redis-5.0.7 yulinfeng$ redis-server redis.conf 
13801:C 12 Feb 2020 22:20:08.954 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
13801:C 12 Feb 2020 22:20:08.954 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=13801, just started
13801:C 12 Feb 2020 22:20:08.954 # Configuration loaded
13801:S 12 Feb 2020 22:20:08.955 * Increased maximum number of open files to 10032 (it was originally set to 256).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 5.0.7 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 13801
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

13801:S 12 Feb 2020 22:20:08.956 # Server initialized
13801:S 12 Feb 2020 22:20:08.956 * Ready to accept connections
13801:S 12 Feb 2020 22:20:08.957 * Connecting to MASTER 192.168.31.145:6379
13801:S 12 Feb 2020 22:20:08.965 * MASTER <-> REPLICA sync started
13801:S 12 Feb 2020 22:20:09.030 * Non blocking connect for SYNC fired the event.
13801:S 12 Feb 2020 22:20:09.034 * Master replied to PING, replication can continue...
13801:S 12 Feb 2020 22:20:09.039 * Partial resynchronization not possible (no cached master)
13801:S 12 Feb 2020 22:20:09.043 * Full resync from master: b7b0b93ea396930cc622250b959fd9eb8eb71c5f:0
13801:S 12 Feb 2020 22:20:09.138 * MASTER <-> REPLICA sync: receiving 185 bytes from master
13801:S 12 Feb 2020 22:20:09.139 * MASTER <-> REPLICA sync: Flushing old data
13801:S 12 Feb 2020 22:20:09.139 * MASTER <-> REPLICA sync: Loading DB in memory
13801:S 12 Feb 2020 22:20:09.139 * MASTER <-> REPLICA sync: Finished with success

You can see that there are more logs from the server after startup. In the first line, master < - > repeat sync started sends the synchronization command to the master server until the synchronization is successful.

Let's go back to the log of the master service:

15702:M 12 Feb 2020 22:20:09.010 * Replica 192.168.31.215:6379 asks for synchronization
15702:M 12 Feb 2020 22:20:09.010 * Full resync requested by replica 192.168.31.215:6379
15702:M 12 Feb 2020 22:20:09.010 * Starting BGSAVE for SYNC with target: disk
15702:M 12 Feb 2020 22:20:09.011 * Background saving started by pid 15703
15703:C 12 Feb 2020 22:20:09.014 * DB saved on disk
15702:M 12 Feb 2020 22:20:09.106 * Background saving terminated with success
15702:M 12 Feb 2020 22:20:09.106 * Synchronization with replica 192.168.31.215:6379 succeeded

As you can see, the first line indicates that a slave server is requesting synchronization, and you can observe the log to find that the master server has executed the bgsave command for a persistent operation at this time.

At this point, when we write a piece of data in the main service, it can be read from the server. However, if we write data from the server:

127.0.0.1:6379> set slave master
(error) READONLY You cant't write against a read only replica.

Review some details of master-slave replication. In addition to the configuration file, a sync synchronization command will be sent at the bottom when the slave server connects to the master server. When the master server receives the sync synchronization command from the slave server, the bgsave persistence command will be executed at the master server. After execution, a snapshot persistence file will be sent to the slave server To complete the synchronization. And the slave server is read-only and cannot be written.

Pay attention to the official account number: CoderBuff, reply to "redis" and get the complete version of Redis5.x tutorial PDF.

This is a CoderBuff that can give programmers buff official account.

Tags: Java Redis snapshot MySQL

Posted on Sat, 22 Feb 2020 10:21:27 -0500 by illushinz