redis-sentinel

Preface

The persistence function of redis guarantees the security of the data to a certain extent, even if the server is down, it can also guarantee very little data loss. Usually, to avoid a single point of failure of the service, data will be copied to multiple copies on different servers, and these servers with copies of the data can be used to process read requests from the client and extendOverall performance, master-slave replication of redis is described below.

1. Overview of master-slave replication

The replication function of redis is to support data synchronization between multiple servers.The replicated server is called master, which replicates from the server as slave. Master can read and write. When a write operation occurs, the data is synchronized from the server automatically. Generally, the server is read-only and receives master synchronized data. A master can have more than one slave, and a SLAVe can only have one master.
The process of master-slave replication:

1, execute the slaveof command from the node;
2, the slave node only saves the information of the master node in the slavef command and does not initiate replication immediately;
3. Discover the information from the master node from the timed tasks inside the node and start using socket s to connect the master node.
4. After the connection is established successfully, send a ping command in the hope that the pong command will respond, otherwise the connection will be reconnected;
5. If the primary node has set permissions, then permission validation is required; if validation fails, replication terminates;
6. Data synchronization takes the longest time after permission validation. The master node sends all the data to the slave node.
7. When the master node synchronizes the current data to the slave node, the replication process is completed, and the master node will continuously send write commands to the slave master node to ensure the consistency of the master-slave data.

The role of master-slave replication:

  • Data redundancy.Implement hot backup of data;
  • Failure recovery.Avoid unavailability of services due to a single point of failure;
  • Read-write separation, load balancing.The primary node is responsible for reading and writing, and the secondary node is responsible for reading, which improves the concurrency of the server.
  • High Availability Base.It is the basis for the implementation of sentinels and clusters;

2, master-slave deployment

Environment description:

Host address port operating system
Primary redis 172.16.1.100 6379 CentOS 7.3
From redis 172.16.1.110 6379 CentOS 7.3

1, Deploy the primary node
1) Install redis
Official download address: http://download.redis.io/releases/
[root@redis-master ~]# tar zxf redis-4.0.14.tar.gz
[root@redis-master ~]# cd redis-4.0.14
[root@redis-master redis-4.0.14]# make && make install


From the figure above, we can easily see that redis is installed in the / usr/local,/usr/local/bin,/usr/local/share,/usr/local/include,/usr/local/lib,/usr/local/share/man directory.

Then switch to the utisl directory and execute the redis initialization script install_server.sh as follows:

[root@redis-master redis-4.0.14]# cd utils/
[root@redis-master utils]# ./install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf] 
Selected default - /etc/redis/6379.conf
Please select the redis log file name [/var/log/redis_6379.log] 
Selected default - /var/log/redis_6379.log
Please select the data directory for this instance [/var/lib/redis/6379] 
Selected default - /var/lib/redis/6379
Please select the redis executable path [/usr/local/bin/redis-server] 
Selected config:
Port           : 6379
Config file    : /etc/redis/6379.conf
Log file       : /var/log/redis_6379.log
Data dir       : /var/lib/redis/6379
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
#All of the above default carriage returns are fine

From the above installation, we can see that the redis configuration file after redis initialization is / etc/redis/6379.conf, the log file is / var/log/redis_6379.log, the data file dump.rdb is stored in the / var/lib/redis/6379 directory, and the startup script is / etc/init.d/redis_6379.

#After initialization is complete, the redis service is started by default (default listening port 6379):

[root@redis-master ~]# /etc/init.d/redis_6379 status
Redis is running (5693)
[root@redis-master ~]# ss -anput | grep redis
tcp    LISTEN     0      128    127.0.0.1:6379                  *:*                   users:(("redis-server",pid=5693,fd=6))

#Firewall rule settings:

[root@redis-master ~]# firewall-cmd --add-port=6379/tcp --permanent
success
[root@redis-master ~]# firewall-cmd --reload
success

2), Configure redis

[root@redis-master ~]# vim /etc/redis/6379.conf 
#Modify as follows (remove comments and modify):
  70 bind 172.16.1.100   #Modify the monitoring address of redis to the ip of the redis host
 501 requirepass pwd@123   #For security reasons, the requirepass parameter of redis'password verification function needs to be started.
 137 daemonize yes  #Run redis instance as daemon

Restart redis when #modification is complete:

[root@redis-master ~]# /etc/init.d/redis_6379 restart
Stopping ...
Waiting for Redis to shutdown ...
Redis stopped
Starting Redis server...
[root@redis-master ~]# ss -anput | grep redis
tcp    LISTEN     0      128    172.16.1.100:6379                  *:*                   users:(("redis-server",pid=5739,fd=6))

#Remote connection redis:
To execute commands on the redis service requires a redis client, which is in the redis installation package we downloaded earlier.

[root@redis-master ~]# redis-cli --version
redis-cli 4.0.14
[root@redis-master ~]# redis-cli -h 172.16.1.100 -p 6379 -a pwd@123
Warning: Using a password with '-a' option on the command line interface may not be safe.
172.16.1.100:6379> ping   #This command is used to detect whether the redis service is started
PONG

2, Deploy slave nodes
1) The process of installing redis is the same as above, and will not be repeated here.
2) Configure redis

[root@redis-slave ~]# vim /etc/redis/6379.conf 
70 bind 172.16.1.110   #ip modified to redis host
137 daemonize yes  #Background running
501 requirepass pwd@123  #Set the authentication password for redis
282 slaveof 172.16.1.100 6379   #This configuration item is the key to master-slave replication and points to the address and port of the master node
289 masterauth pwd@123 #Configure master's authorization password (if master does not set requirepass option, no configuration is required from the server)

There are actually three ways to configure master-slave replication:

1. Add slaveof [master Host] [master Port] to the configuration file  
(2) redis-server startup with--slaveof [master Host] [master Port]
(3) Log on redis and use the command slaveof [master Host] [master Port]
#Restart redis service:
[root@redis-slave ~]# /etc/init.d/redis_6379 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@redis-slave ~]# ss -anput | grep redis
tcp    LISTEN     0      128    172.16.1.110:6379                  *:*                   users:(("redis-server",pid=4886,fd=6))
tcp    ESTAB      0      0      172.16.1.110:34105              172.16.1.100:6379                users:(("redis-server",pid=4886,fd=7))
#You can see that there is one more master-slave replication process
#Configure Firewall:
[root@redis-slave ~]# firewall-cmd --add-port=6379/tcp --permanent
success
[root@redis-slave ~]# firewall-cmd --reload
success

3, Test data synchronization

main redis: 
[root@redis-master ~]# redis-cli -h 172.16.1.100 -p 6379 -a pwd@123
Warning: Using a password with '-a' option on the command line interface may not be safe.
172.16.1.100:6379> set name abc   #Set a key/value
OK
172.16.1.100:6379> get name
"abc"
from redis: 
[root@redis-slave ~]# redis-cli  -h 172.16.1.110 -p 6379 -a pwd@123
Warning: Using a password with '-a' option on the command line interface may not be safe.
172.16.1.110:6379> get name   #Successful data synchronization
"abc"

4, test read-write separation (redis defaults to read-write separation)

#Testing from redis:
172.16.1.110:6379> set age 20
(error) READONLY You can't write against a read only slave. 

3, master-slave switching

1, stop main redis, simulate failure

[root@redis-master ~]# redis-cli  -h 172.16.1.100 -p 6379 -a pwd@123 shutdown
[root@redis-master ~]# ss -anput | grep redis
[root@redis-master ~]# 

2, set from redis to primary redis (turn off replication)

[root@redis-slave ~]# redis-cli -h 172.16.1.110 -p 6379 -a pwd@123 slaveof no one  
Warning: Using a password with '-a' option on the command line interface may not be safe.
OK

3, Test whether to switch from redis to primary redis
#View the roles of the current host:

[root@redis-slave ~]# redis-cli -h 172.16.1.110 -p 6379 -a pwd@123 info replication
Warning: Using a password with '-a' option on the command line interface may not be safe.
# Replication
role:master            //Role is master
connected_slaves:0
master_replid:51ca62c64f31a7adedfb942a95d01c922f42124b
master_replid2:e5a32a89b7806f0fa7954cb0c422172ea889fff0
master_repl_offset:5583
second_repl_offset:5584
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:4607
repl_backlog_histlen:977

#Test read and write data:

[root@redis-slave ~]# redis-cli -h 172.16.1.110 -p 6379 -a pwd@123 
Warning: Using a password with '-a' option on the command line interface may not be safe.
172.16.1.110:6379> keys *
1) "age"
2) "name"
172.16.1.110:6379> get name
"abc"
172.16.1.110:6379> set name zhangsan
OK
172.16.1.110:6379> get name
"zhangsan"

4, the original master redis are back to normal, switch back
1) Save the data of the current master redis

172.16.1.110:6379> keys *
1) "age"
2) "name"
172.16.1.110:6379> save
OK
172.16.1.110:6379> get name
"zhangsan"

2) Overwrite a copy of the dump.rdb file in the current master redis root directory to the original master redis root directory (make sure the rerun master gets the latest data in redis):
[root@redis-slave ~]# scp /var/lib/redis/6379/dump.rdb root@172.16.1.100:/var/lib/redis/6379/
3) Start the original master redis:

[root@redis-master ~]# /etc/init.d/redis_6379 start
Starting Redis server...
[root@redis-master ~]# ss -anput | grep redis
tcp    LISTEN     0      128    172.16.1.100:6379                  *:*                   users:(("redis-server",pid=19649,fd=6))

4) Switch in the current master redis:

[root@redis-slave ~]# redis-cli -h 172.16.1.110 -p 6379 -a pwd@123 slaveof 172.16.1.100 6379
Warning: Using a password with '-a' option on the command line interface may not be safe.
OK

#View Status:

You can see that the main redis state has now become slave

#View the status of the main redis:

You can see that the state has changed to master and that the data is up-to-date, but this manual approach is certainly slightly inadequate in production environments, so here's the redis sentry mechanism.

4, Redis Sentinel

In the master-slave replication mode of redis, once the master node is unable to provide service due to a failure, it needs to be promoted manually from the master node and notified the application to update the address of the node, which is unacceptable in many scenarios; Fortunately, redis has officially provided Redis since 2.8The Sentinel mechanism solves this problem.
Overview of Sentry Mechanism
The sentinel system for redis is used to manage multiple redis servers and performs three tasks:
1, Monitoring: Sentinels constantly check that your master and slave are working properly.
2, Notification: Sentinels can send notifications to administrators or this other application through the API when a redis being monitored has a problem;
3, Automatic Failure MigrationFailover: When a master is not functioning properly, the Sentry will start automatically migrating one after another. It will upgrade one slave of the invalid master to a new master, and change the other slaves of the invalid master to copy the new master. When a client connection fails, the cluster will also return the address of the new master to the client, enabling the cluster to use the new mastEr replaces the invalid master.

Sentinels are also redis services in nature, but they do not provide the same functionality as ordinary redis services. Sentinels are a distributed architecture because you want to make redis highly available, first of all you need to make them highly available. So if we need to build a sentinel, we need to deploy at least three instances, preferably an odd number, because voting is involved in subsequent failover.
Deploy sentinel to monitor and manage redis master-slave architecture
Our master-slave architecture environment above is one master-slave. According to the Sentinel's voting mechanism, there are at least three instances, so we add a Slave Slave Slave Node (172.16.1.120) to the original environment.
Step omitted to install the configuration from the node as deployed above, ultimately ensuring that the data above the master can be synchronized.

2, configure sentinel (three hosts operate the same)
There are three Sentinels, and each Sentinel has the same configuration.There is a sentinel.conf file in the redis installation directory with a copy to modify:

[root@redis-master ~]# cp redis-4.0.14/sentinel.conf  /etc/redis/
[root@redis-master ~]# ls /etc/redis/
6379.conf  sentinel.conf
[root@redis-master ~]# vim /etc/redis/sentinel.conf 
//The modifications are as follows:
#Bind the ip address of the redis host, note that the other two slave s need to point to their native address from the node
bind 172.16.1.100
#Port number, default is redis instance + 20000, so let's just follow this rule
port 26379  
#Add Daemon Run
daemonize yes
#It is important to add a location where logs are stored to view the failover process
logfile "26379.log"
#The working directory (sentinel-related information files are saved here, including log files), which remains the default (although you can also customize the path)
dir /tmp
#Specify the redis instance sentinel will monitor: Monitor a redis server named mymaster with a customizable name, which is the master ip address.
#The last two represent at least two senders who assume that the primary server is down before failover occurs, or that the primary service is not down, typically set to N/2+1(N is the total number of sentries).
sentinel monitor mymaster 172.16.1.100 6379 2
#Define the password for the service, mymaster is the name of the service, followed by the password for the redis server. If your redis does not have a password set, you need to turn off protected-mode no
sentinel auth-pass mymaster pwd@123
#sentinel judges the response time for a server failure and considers the server to be invalid if no response from the server is received beyond that time
sentinel down-after-milliseconds mymaster 3000
/#Up to how many simultaneous data replication (concurrent) can be initiated from the server after failover is complete.
#The smaller the number, the longer it takes to complete all replication from the service data, the larger the number, and the greater the pressure on the primary server
sentinel parallel-syncs mymaster 1
/#Failover timeout is considered a failure if sentinel fails to complete the failover operation within this configuration value (that is, master/slave auto-switch on failure).
sentinel failover-timeout mymaster 180000

3, Start the Sentinel in turn: (Two methods)

Method 1:
[root@redis-master ~]# redis-sentinel /etc/redis/sentinel.conf 
//Method 2:
[root@redis-master ~]# redis-server  /etc/redis/sentinel.conf --sentinel

Check if the port is working:

The other two slave s start from the node in turn.

Note the startup sequence: If redis and sentinel start at the same time, start the redis service before starting sentinel.

#Configure the firewall: (Sentry listening ports need to be opened on each node)
[root@redis-slave2 ~]# firewall-cmd --add-port=26379/tcp --permanent
success
[root@redis-slave2 ~]# firewall-cmd --reload
fisuccess

#Because the Sentinel is also a redis instance, we view the information currently being monitored by the Sentinel with the following commands:

[root@redis-master ~]# redis-cli  -p 26379 -h 172.16.1.100
172.16.1.100:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.16.1.100:6379,slaves=2,sentinels=3
#You can see that the current state is ok, and the host listening is the current master node, two slave nodes, three sentry

4. Simulate the primary redis server failure, migrate to other secondary servers automatically, and promote from server to primary server automatically

#Close the redis service or kill the process
[root@redis-master ~]# redis-cli -p 6379 -h 172.16.1.100 -a pwd@123 shutdown
Warning: Using a password with '-a' option on the command line interface may not be safe.
[root@redis-master ~]# ps -ef | grep redis
root      19687      1  0 04:35 ?        00:00:00 redis-sentinel 172.16.1.100:26379 [sentinel]
root      19700   2242  0 04:39 pts/0    00:00:00 grep --color=auto redis

#View the sentinel's surveillance information:

[root@redis-master ~]# redis-cli -h 172.16.1.100 -p 26379 info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.16.1.110:6379,slaves=2,sentinels=3
#You can see that the master node that the current Sentry is listening on is not the original, but from the node (172.16.1.110).

#View sentinel's log information:

From the log information, it is known that the original master host has been dropped, and through sentinel sentry mechanism, master has been automatically switched to 172.16.1.110 from the node.

#Verify that the original switch from node succeeded:

You can see that you have switched from the original slave state to master, and 172.16.1.20 is your own slave node.

172.16.1.110:6379> keys *
1) "addr"
2) "age"
3) "name"
172.16.1.110:6379> set linux redis
OK
172.16.1.110:6379> get linux
"redis"
//And can read and write normally

5. Will sentine re-select master when the suspended master master master node returns to normal? Let's verify:

#Restart the redis service:
[root@redis-master ~]# /etc/init.d/redis_6379 start
Starting Redis server...
[root@redis-master ~]# ps -ef | grep redis
root      19687      1  0 04:35 ?        00:00:02 redis-sentinel 172.16.1.100:26379 [sentinel]
root      19713      1  0 04:57 ?        00:00:00 /usr/local/bin/redis-server 172.16.1.100:6379
root      19718   2242  0 04:57 pts/0    00:00:00 grep --color=auto redis

#View your status:

You can know that once a suspended mater is restored, it cannot be a master, but can only be a slave Slave slave from the current master.However, it is important to note that the Sentinel mechanism will not help you to restore the data lost during this period of time after the suspended host is restored, so you need to make a backup of the dump.rdb files of other nodes so that you can import the lost data after the recovery.

Tags: Linux Redis firewall vim CentOS

Posted on Sun, 15 Mar 2020 12:38:29 -0400 by intellivision