The method of master-slave switching technology is: when the master server goes down, it needs to manually switch a slave server to the master-slave server, which requires manual intervention, which is time-consuming and laborious, but also causes service unavailability for a period of time. This is not a recommended way. More time, we give priority to sentinel mode, which is the mainstream mode of current enterprise applications.
Redis Sentinel is a highly available implementation of redis. Sentinel is a tool to manage multiple Redis instances. It can monitor, notify and automatically failover Redis.
1. Schematic diagram of Redis master-slave replication mode and Sentinel high availability architecture
Sentinel's main functions include: primary node survival detection, primary and secondary operation detection, automatic failover, and primary and secondary switching. The minimum sentinel configuration of Redis is one master and one slave;
The Sentinel system of Redis can be used to manage multiple Redis servers. The system can perform the following four tasks:
- Monitoring: Sentinel will constantly check whether the master and slave servers are running normally;
- Notification: when there is a problem with a monitored Redis server, Sentinel sends notifications to administrators or other applications through API scripts;
- Automatic failover: when the primary node fails to work normally, Sentinel will start an automatic failover operation. It will upgrade one of the slave nodes in the master-slave relationship with the failed primary node to a new primary node, and point other slave nodes to the new primary node;
- Configuration provider: in Redis Sentinel mode, when the client application is initialized, it connects the Sentinel node collection to obtain the information of the primary node;
By default, each Sentinel node sends PING commands to the Redis node and other Sentinel nodes at a frequency of once per second, and judges whether the node is online by the reply of the node.
- Subjective logoff: the subjective logoff is applicable to all primary and secondary nodes. If Sentinel does not receive an effective reply from the target node within milliseconds of down after milliseconds, it will be determined that the node is a subjective offline node;
- Objective offline: the objective offline is only applicable to the primary node. If the primary node fails, the Sentinel node will ask other Sentinel nodes for status judgment through the Sentinel is master down by addr command. If more than < quorum > nodes determine that the primary node is not reachable, the Sentinel node will determine that the primary node is an objective offline node.
When the sentinel node connects to a Redis instance, it will create two connections: cmd and pub/sub. Sentinel sends commands to Redis through cmd connection and connects to other sentinel instances on Redis instance through pub/sub.
Commands for Sentinel to interact with Redis master and slave nodes
|PING||Sentinel sends PING command to Redis node to check the status of the node|
|INFO||Sentinel sends INFO command to Redis node to obtain its slave node information|
|PUBLISH||Sentinel publishes its own information and configuration related to the primary node to its monitored Redis node sentinel:hello|
|SUBSCRIBE||Sentinel obtains other sentinel nodes monitoring the same service by subscribing to the channel sentinel:hello of the Redis master node and the slave node|
Commands for Sentinel to interact with Sentinel
|PING||Sentinel sends PING command to other sentinel nodes to check the status of nodes|
|SENTINEL:is-master-down-by-addr||Negotiate with other Sentinel about the state of the master node. If the master node is in SDOWN state, vote to automatically select a new master node|
Each Sentinel node needs to perform the following tasks on a regular basis
Each Sentinel sends a PING command every second to its known master, slave, and other Sentinel instances.
If an instance takes longer than the value specified by down after milliseconds from the last valid reply to PING command, the instance will be marked as subjective offline by Sentinel.
If a primary server is marked as a subjective logoff, all Sentinel nodes of the primary server are being monitored to confirm that the primary server has indeed entered the subjective logoff status once a second.
If a primary server is marked as a subjective logoff and there are enough sentinels (at least the number specified in the configuration file) to agree with this judgment within the specified time range, then the primary server is marked as an objective logoff.
In general, each Sentinel sends INFO commands every 10 seconds to all its known master and slave servers. When a master server is marked as offline by Sentinel, the frequency of Sentinel sending INFO commands to all slave servers of the offline master server will be changed from once every 10 seconds to once every second.
Sentinel and other sentinel negotiate the status of the primary node. If the primary node is in SDOWN status, the vote automatically selects the new primary node. Point the remaining slave nodes to the new master node for data replication.
When there is not enough sentinel to allow the primary server to logoff, the objective logoff status of the primary server will be removed. When the primary server returns a valid reply to Sentinel's PING command, the primary server's subjective offline status will be removed.
Note: a valid PING reply can be + PONG, - LOADING or - MASTERDOWN. If the server returns other replies than the above three replies, or does not reply to the PING command within the specified time, Sentinel considers the reply returned by the server to be invalid (non valid).
- A robust Redis Sentinel cluster should use at least three Sentinel instances, and ensure that these instances are placed on different machines or even different physical areas.
- Sentinel cannot guarantee strong consistency.
- Sentinel is supported in common client application libraries.
- Sentinel needs constant testing and observation to ensure high availability.
|Service type||Master slave replication||IP||Port|
|Redis||slave - 1||192.168.182.11||26379|
|Redis||slave - 2||192.168.182.11||36379|
- Create the sentinel-1, sentinel-2 and sentinel-3 folders in the directory / usr/local/bin/sentinel respectively
[root@localhost sentinel]# mkdir sentinel-1 sentinel-2 sentinel-3 [root@localhost sentinel]# ls sentinel-1 sentinel-2 sentinel-3
- Copy the redis.conf configuration file to the sentinel-1, sentinel-2 and sentinel-3 folders respectively
[root@localhost bin]# cp redis.conf sentinel/sentinel-1 [root@localhost bin]# cp redis.conf sentinel/sentinel-2 [root@localhost bin]# cp redis.conf sentinel/sentinel-3
PS: the configuration file must be modified in order, otherwise an error will be reported and the master cannot be recognized
- Primary node: redis-16379
daemonize yes pidfile /var/run/redis-16379.pid logfile /var/log/redis/redis-16379.log port 16379 bind 0.0.0.0 timeout 300 databases 16 dbfilename dump-16379.db dir ./redis-workdir masterauth 123456 requirepass 123456
- Slave node 1: redis-26379
daemonize yes pidfile /var/run/redis-26379.pid logfile /var/log/redis/redis-26379.log port 26379 bind 0.0.0.0 timeout 300 databases 16 dbfilename dump-26379.db dir ./redis-workdir masterauth 123456 requirepass 123456 slaveof 127.0.0.1 16379
- Slave node 2: redis-36379
daemonize yes pidfile /var/run/redis-36379.pid logfile /var/log/redis/redis-36379.log port 36379 bind 0.0.0.0 timeout 300 databases 16 dbfilename dump-36379.db dir ./redis-workdir masterauth 123456 requirepass 123456 slaveof 127.0.0.1 16379
PS: if you want to do automatic failover, it is recommended to set masterauth for all redis.conf. Because automatic failure will only override the master-slave relationship, that is, slavof, and will not automatically write to masterauth. If Redis did not set the password, it can be ignored.
- Start redis-1, redis-2 and redis-3 nodes in sequence
[root@localhost bin]# redis-server sentinel/sentinel-1/redis.conf [root@localhost bin]# redis-server sentinel/sentinel-2/redis.conf [root@localhost bin]# redis-server sentinel/sentinel-3/redis.conf
- View startup process
root 80625 1 0 15:24 ? 00:00:17 redis-server 0.0.0.0:36379 root 80631 1 0 15:24 ? 00:00:16 redis-server 0.0.0.0:26379 root 82108 1 0 17:22 ? 00:00:01 redis-server 0.0.0.0:16379
- Write data to the primary node, redis-16379, and check whether the slave node 1: redis-26379 and the slave node 2: redis-36379 synchronize the data
127.0.0.1:16379> set mykeys redis OK
127.0.0.1:26379> keys * 1) "mykey"
127.0.0.1:36379> keys * 1) "mykey"
The master-slave node data has been synchronized, and the master-slave replication has been set up and deployed
- Copy the sentinel.conf configuration file to the sentinel-1, sentinel-2 and sentinel-3 folders respectively
[root@localhost bin]# cp sentinel.conf sentinel/sentinel-1/ [root@localhost bin]# cp sentinel.conf sentinel/sentinel-2/ [root@localhost bin]# cp sentinel.conf sentinel/sentinel-3/
PS: the configuration file must be modified in order, otherwise an error will be reported and the master cannot be recognized
- Node 1: sentinel-16380
protected-mode no bind 0.0.0.0 port 16380 daemonize yes sentinel monitor master 127.0.0.1 16379 2 sentinel down-after-milliseconds master 5000 sentinel failover-timeout master 180000 sentinel parallel-syncs master 1 sentinel auth-pass master 123456 logfile /var/log/redis/sentinel-16380.log
- Node 2: sentinel-26380
protected-mode no bind 0.0.0.0 port 26380 daemonize yes sentinel monitor master 127.0.0.1 16379 2 sentinel down-after-milliseconds master 5000 sentinel failover-timeout master 180000 sentinel parallel-syncs master 1 sentinel auth-pass master 123456 logfile /var/log/redis/sentinel-26380.log
- Node 3: sentinel-36380
protected-mode no bind 0.0.0.0 port 36380 daemonize yes sentinel monitor master 127.0.0.1 16379 2 sentinel down-after-milliseconds master 5000 sentinel failover-timeout master 180000 sentinel parallel-syncs master 1 sentinel auth-pass master 123456 logfile /var/log/redis/sentinel-36380.log
- Start 1638026380 and 36380 Sentinel nodes in sequence. The start command and start log are as follows:
[root@localhost bin]# redis-server sentinel/sentinel-1/sentinel.conf --sentinel [root@localhost bin]# redis-server sentinel/sentinel-2/sentinel.conf --sentinel [root@localhost bin]# redis-server sentinel/sentinel-3/sentinel.conf --sentinel
- To view Sentinel's startup process:
[root@localhost bin]# ps -ef | grep redis root 80625 1 0 15:24 ? 00:00:18 redis-server 0.0.0.0:36379 root 80631 1 0 15:24 ? 00:00:17 redis-server 0.0.0.0:26379 root 82108 1 0 17:22 ? 00:00:04 redis-server 0.0.0.0:16379 root 82573 1 0 17:51 ? 00:00:00 redis-server 0.0.0.0:16380 [sentinel] root 82595 1 0 17:52 ? 00:00:00 redis-server 0.0.0.0:26380 [sentinel] root 82617 1 0 17:53 ? 00:00:00 redis-server 0.0.0.0:36380 [sentinel]
- Node sentinel-16380
[root@localhost sentinel-1]# cat sentinel.log 8166:X 12 Jan 2020 18:42:45.941 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 8166:X 12 Jan 2020 18:42:45.941 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=8166, just started 8166:X 12 Jan 2020 18:42:45.941 # Configuration loaded 8167:X 12 Jan 2020 18:42:45.956 * Increased maximum number of open files to 10032 (it was originally set to 1024). 8167:X 12 Jan 2020 18:42:45.957 * Running mode=sentinel, port=16380. 8167:X 12 Jan 2020 18:42:45.957 # Sentinel ID is ddf49c992bceba4fe13eb213bd72aa7db9ff101b 8167:X 12 Jan 2020 18:42:45.957 # +monitor master master 127.0.0.1 16379 quorum 2 8167:X 12 Jan 2020 18:42:48.970 # +sdown master master 127.0.0.1 16379
The Sentinel ID of sentinel-16380 node is ddf49c992bceba4fe13eb213bd72aa7db9ff101b, and it adds itself to the sentinel cluster through Sentinel ID.
- Node sentinel-26380
[root@localhost sentinel-2]# cat sentinel.log 8180:X 12 Jan 2020 18:42:58.483 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 8180:X 12 Jan 2020 18:42:58.483 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=8180, just started 8180:X 12 Jan 2020 18:42:58.483 # Configuration loaded 8181:X 12 Jan 2020 18:42:58.496 * Increased maximum number of open files to 10032 (it was originally set to 1024). 8181:X 12 Jan 2020 18:42:58.496 * Running mode=sentinel, port=26380. 8181:X 12 Jan 2020 18:42:58.496 # Sentinel ID is ddf49c992bceba4fe13eb213bd72aa7db9ff101b 8181:X 12 Jan 2020 18:42:58.496 # +monitor master master 127.0.0.1 16379 quorum 2 8181:X 12 Jan 2020 18:42:58.499 * +slave slave 127.0.0.1:26379 127.0.0.1 26379 @ master 127.0.0.1 16379 8181:X 12 Jan 2020 18:42:58.500 * +slave slave 127.0.0.1:36379 127.0.0.1 36379 @ master 127.0.0.1 16379
It can be noted that sentinel-26380.conf refreshes and writes all the slave nodes redis-26379 and redis-36379 associated with the Redis master node, as well as the IP addresses, port numbers and Sentinel ID s of the remaining two Sentinel nodes sentinel-36380 and sentinel-16380.
- kill the process of the primary node: redis-16379
[root@localhost sentinel-1]# kill 8757
- View the log (at this time, the primary node has been switched to redis-36379)
...... 8801:X 12 Jan 2020 19:17:47.631 * +slave slave 127.0.0.1:26379 127.0.0.1 26379 @ master 127.0.0.1 36379 8801:X 12 Jan 2020 19:17:47.631 * +slave slave 127.0.0.1:16379 127.0.0.1 16379 @ master 127.0.0.1 36379 8801:X 12 Jan 2020 19:17:50.668 # +sdown slave 127.0.0.1:16379 127.0.0.1 16379 @ master 127.0.0.1 36379 ......
127.0.0.1:26380> sentinel master master 1) "name" 2) "master" 3) "ip" 4) "127.0.0.1" 5) "port" 6) "36379" 7) "runid" 8) "15ea078acd8471bf2072a68c1c3f3fc570cb5cca" 9) "flags" 10) "master"
127.0.0.1:16380> info sentinel # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=master,status=oup,address=127.0.0.1:36379,slaves=2,sentinels=3