The way of operation and maintenance

The way of operation and maintenance

Preface

The method of master-slave switching technology is: when the master server goes down, it needs to manually switch a slave server to the master-slave server, which requires manual intervention, which is time-consuming and laborious, but also causes service unavailability for a period of time. This is not a recommended way. More time, we give priority to sentinel mode, which is the mainstream mode of current enterprise applications.

Redis Sentinel is a highly available implementation of redis. Sentinel is a tool to manage multiple Redis instances. It can monitor, notify and automatically failover Redis.

1, Basic concepts of Redis Sentinel

1. Schematic diagram of Redis master-slave replication mode and Sentinel high availability architecture

2. Redis Sentinel architecture

3. Main functions of Redis Sentinel

Sentinel's main functions include: primary node survival detection, primary and secondary operation detection, automatic failover, and primary and secondary switching. The minimum sentinel configuration of Redis is one master and one slave;

The Sentinel system of Redis can be used to manage multiple Redis servers. The system can perform the following four tasks:

  • Monitoring: Sentinel will constantly check whether the master and slave servers are running normally;
  • Notification: when there is a problem with a monitored Redis server, Sentinel sends notifications to administrators or other applications through API scripts;
  • Automatic failover: when the primary node fails to work normally, Sentinel will start an automatic failover operation. It will upgrade one of the slave nodes in the master-slave relationship with the failed primary node to a new primary node, and point other slave nodes to the new primary node;
  • Configuration provider: in Redis Sentinel mode, when the client application is initialized, it connects the Sentinel node collection to obtain the information of the primary node;

4. Subjective offline and objective offline

By default, each Sentinel node sends PING commands to the Redis node and other Sentinel nodes at a frequency of once per second, and judges whether the node is online by the reply of the node.

  • Subjective logoff: the subjective logoff is applicable to all primary and secondary nodes. If Sentinel does not receive an effective reply from the target node within milliseconds of down after milliseconds, it will be determined that the node is a subjective offline node;
  • Objective offline: the objective offline is only applicable to the primary node. If the primary node fails, the Sentinel node will ask other Sentinel nodes for status judgment through the Sentinel is master down by addr command. If more than < quorum > nodes determine that the primary node is not reachable, the Sentinel node will determine that the primary node is an objective offline node.

5. Sentinel's communication command

When the sentinel node connects to a Redis instance, it will create two connections: cmd and pub/sub. Sentinel sends commands to Redis through cmd connection and connects to other sentinel instances on Redis instance through pub/sub.

Commands for Sentinel to interact with Redis master and slave nodes

command Effect
PING Sentinel sends PING command to Redis node to check the status of the node
INFO Sentinel sends INFO command to Redis node to obtain its slave node information
PUBLISH Sentinel publishes its own information and configuration related to the primary node to its monitored Redis node sentinel:hello
SUBSCRIBE Sentinel obtains other sentinel nodes monitoring the same service by subscribing to the channel sentinel:hello of the Redis master node and the slave node

Commands for Sentinel to interact with Sentinel

command Effect
PING Sentinel sends PING command to other sentinel nodes to check the status of nodes
SENTINEL:is-master-down-by-addr Negotiate with other Sentinel about the state of the master node. If the master node is in SDOWN state, vote to automatically select a new master node

6. How Redis Sentinel works

Each Sentinel node needs to perform the following tasks on a regular basis
Each Sentinel sends a PING command every second to its known master, slave, and other Sentinel instances.

If an instance takes longer than the value specified by down after milliseconds from the last valid reply to PING command, the instance will be marked as subjective offline by Sentinel.

If a primary server is marked as a subjective logoff, all Sentinel nodes of the primary server are being monitored to confirm that the primary server has indeed entered the subjective logoff status once a second.

If a primary server is marked as a subjective logoff and there are enough sentinels (at least the number specified in the configuration file) to agree with this judgment within the specified time range, then the primary server is marked as an objective logoff.

In general, each Sentinel sends INFO commands every 10 seconds to all its known master and slave servers. When a master server is marked as offline by Sentinel, the frequency of Sentinel sending INFO commands to all slave servers of the offline master server will be changed from once every 10 seconds to once every second.

Sentinel and other sentinel negotiate the status of the primary node. If the primary node is in SDOWN status, the vote automatically selects the new primary node. Point the remaining slave nodes to the new master node for data replication.

When there is not enough sentinel to allow the primary server to logoff, the objective logoff status of the primary server will be removed. When the primary server returns a valid reply to Sentinel's PING command, the primary server's subjective offline status will be removed.

Note: a valid PING reply can be + PONG, - LOADING or - MASTERDOWN. If the server returns other replies than the above three replies, or does not reply to the PING command within the specified time, Sentinel considers the reply returned by the server to be invalid (non valid).

2, Redis Sentinel building

1. Redis Sentinel deployment notice

  • A robust Redis Sentinel cluster should use at least three Sentinel instances, and ensure that these instances are placed on different machines or even different physical areas.
  • Sentinel cannot guarantee strong consistency.
  • Sentinel is supported in common client application libraries.
  • Sentinel needs constant testing and observation to ensure high availability.

2. Redis Sentinel node planning

Service type Master slave replication IP Port
Redis Master 192.168.182.11 16379
Redis slave - 1 192.168.182.11 26379
Redis slave - 2 192.168.182.11 36379
Sentinel - 192.168.182.11 26380
Sentinel - 192.168.182.11 26380
Sentinel - 192.168.182.11 26380

3. Redis server configuration deployment

  • Create the sentinel-1, sentinel-2 and sentinel-3 folders in the directory / usr/local/bin/sentinel respectively
[root@localhost sentinel]# mkdir sentinel-1 sentinel-2 sentinel-3
[root@localhost sentinel]# ls
sentinel-1  sentinel-2  sentinel-3
  • Copy the redis.conf configuration file to the sentinel-1, sentinel-2 and sentinel-3 folders respectively
[root@localhost bin]# cp redis.conf sentinel/sentinel-1
[root@localhost bin]# cp redis.conf sentinel/sentinel-2
[root@localhost bin]# cp redis.conf sentinel/sentinel-3

Modify three configuration files as follows:

PS: the configuration file must be modified in order, otherwise an error will be reported and the master cannot be recognized

  • Primary node: redis-16379
daemonize yes
pidfile /var/run/redis-16379.pid
logfile /var/log/redis/redis-16379.log
port 16379
bind 0.0.0.0
timeout 300
databases 16
dbfilename dump-16379.db
dir ./redis-workdir
masterauth 123456
requirepass 123456
  • Slave node 1: redis-26379
daemonize yes
pidfile /var/run/redis-26379.pid
logfile /var/log/redis/redis-26379.log
port 26379
bind 0.0.0.0
timeout 300
databases 16
dbfilename dump-26379.db
dir ./redis-workdir
masterauth 123456
requirepass 123456
slaveof 127.0.0.1 16379
  • Slave node 2: redis-36379
daemonize yes
pidfile /var/run/redis-36379.pid
logfile /var/log/redis/redis-36379.log
port 36379
bind 0.0.0.0
timeout 300
databases 16
dbfilename dump-36379.db
dir ./redis-workdir
masterauth 123456
requirepass 123456
slaveof 127.0.0.1 16379

PS: if you want to do automatic failover, it is recommended to set masterauth for all redis.conf. Because automatic failure will only override the master-slave relationship, that is, slavof, and will not automatically write to masterauth. If Redis did not set the password, it can be ignored.

Redis server start verification

  • Start redis-1, redis-2 and redis-3 nodes in sequence
[root@localhost bin]# redis-server sentinel/sentinel-1/redis.conf 
[root@localhost bin]# redis-server sentinel/sentinel-2/redis.conf 
[root@localhost bin]# redis-server sentinel/sentinel-3/redis.conf 
  • View startup process
root      80625      1  0 15:24 ?        00:00:17 redis-server 0.0.0.0:36379
root      80631      1  0 15:24 ?        00:00:16 redis-server 0.0.0.0:26379
root      82108      1  0 17:22 ?        00:00:01 redis-server 0.0.0.0:16379
  • Write data to the primary node, redis-16379, and check whether the slave node 1: redis-26379 and the slave node 2: redis-36379 synchronize the data
127.0.0.1:16379> set mykeys redis
OK
127.0.0.1:26379> keys *
1) "mykey"
127.0.0.1:36379> keys *
1) "mykey"

The master-slave node data has been synchronized, and the master-slave replication has been set up and deployed

4, Sentinel configuration deployment

  • Copy the sentinel.conf configuration file to the sentinel-1, sentinel-2 and sentinel-3 folders respectively
[root@localhost bin]# cp sentinel.conf sentinel/sentinel-1/
[root@localhost bin]# cp sentinel.conf sentinel/sentinel-2/
[root@localhost bin]# cp sentinel.conf sentinel/sentinel-3/

Modify three configuration files as follows:

PS: the configuration file must be modified in order, otherwise an error will be reported and the master cannot be recognized

  • Node 1: sentinel-16380
protected-mode no
bind 0.0.0.0
port 16380
daemonize yes
sentinel monitor master 127.0.0.1 16379 2
sentinel down-after-milliseconds master 5000
sentinel failover-timeout master 180000
sentinel parallel-syncs master 1
sentinel auth-pass master 123456
logfile /var/log/redis/sentinel-16380.log
  • Node 2: sentinel-26380
protected-mode no
bind 0.0.0.0
port 26380
daemonize yes
sentinel monitor master 127.0.0.1 16379 2
sentinel down-after-milliseconds master 5000
sentinel failover-timeout master 180000
sentinel parallel-syncs master 1
sentinel auth-pass master 123456
logfile /var/log/redis/sentinel-26380.log
  • Node 3: sentinel-36380
protected-mode no
bind 0.0.0.0
port 36380
daemonize yes
sentinel monitor master 127.0.0.1 16379 2
sentinel down-after-milliseconds master 5000
sentinel failover-timeout master 180000
sentinel parallel-syncs master 1
sentinel auth-pass master 123456
logfile /var/log/redis/sentinel-36380.log

Sentinel start verification

  • Start 1638026380 and 36380 Sentinel nodes in sequence. The start command and start log are as follows:
[root@localhost bin]# redis-server sentinel/sentinel-1/sentinel.conf --sentinel
[root@localhost bin]# redis-server sentinel/sentinel-2/sentinel.conf --sentinel
[root@localhost bin]# redis-server sentinel/sentinel-3/sentinel.conf --sentinel
  • To view Sentinel's startup process:
[root@localhost bin]# ps -ef | grep redis
root      80625      1  0 15:24 ?        00:00:18 redis-server 0.0.0.0:36379
root      80631      1  0 15:24 ?        00:00:17 redis-server 0.0.0.0:26379
root      82108      1  0 17:22 ?        00:00:04 redis-server 0.0.0.0:16379
root      82573      1  0 17:51 ?        00:00:00 redis-server 0.0.0.0:16380 [sentinel]
root      82595      1  0 17:52 ?        00:00:00 redis-server 0.0.0.0:26380 [sentinel]
root      82617      1  0 17:53 ?        00:00:00 redis-server 0.0.0.0:36380 [sentinel]

To view Sentinel's startup log:

  • Node sentinel-16380
[root@localhost sentinel-1]# cat sentinel.log 
8166:X 12 Jan 2020 18:42:45.941 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
8166:X 12 Jan 2020 18:42:45.941 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=8166, just started
8166:X 12 Jan 2020 18:42:45.941 # Configuration loaded
8167:X 12 Jan 2020 18:42:45.956 * Increased maximum number of open files to 10032 (it was originally set to 1024).
8167:X 12 Jan 2020 18:42:45.957 * Running mode=sentinel, port=16380.
8167:X 12 Jan 2020 18:42:45.957 # Sentinel ID is ddf49c992bceba4fe13eb213bd72aa7db9ff101b
8167:X 12 Jan 2020 18:42:45.957 # +monitor master master 127.0.0.1 16379 quorum 2
8167:X 12 Jan 2020 18:42:48.970 # +sdown master master 127.0.0.1 16379

The Sentinel ID of sentinel-16380 node is ddf49c992bceba4fe13eb213bd72aa7db9ff101b, and it adds itself to the sentinel cluster through Sentinel ID.

  • Node sentinel-26380
[root@localhost sentinel-2]# cat sentinel.log 
8180:X 12 Jan 2020 18:42:58.483 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
8180:X 12 Jan 2020 18:42:58.483 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=8180, just started
8180:X 12 Jan 2020 18:42:58.483 # Configuration loaded
8181:X 12 Jan 2020 18:42:58.496 * Increased maximum number of open files to 10032 (it was originally set to 1024).
8181:X 12 Jan 2020 18:42:58.496 * Running mode=sentinel, port=26380.
8181:X 12 Jan 2020 18:42:58.496 # Sentinel ID is ddf49c992bceba4fe13eb213bd72aa7db9ff101b
8181:X 12 Jan 2020 18:42:58.496 # +monitor master master 127.0.0.1 16379 quorum 2
8181:X 12 Jan 2020 18:42:58.499 * +slave slave 127.0.0.1:26379 127.0.0.1 26379 @ master 127.0.0.1 16379
8181:X 12 Jan 2020 18:42:58.500 * +slave slave 127.0.0.1:36379 127.0.0.1 36379 @ master 127.0.0.1 16379

It can be noted that sentinel-26380.conf refreshes and writes all the slave nodes redis-26379 and redis-36379 associated with the Redis master node, as well as the IP addresses, port numbers and Sentinel ID s of the remaining two Sentinel nodes sentinel-36380 and sentinel-16380.

Redis Sentinel failover and recovery

  • kill the process of the primary node: redis-16379
[root@localhost sentinel-1]# kill 8757
  • View the log (at this time, the primary node has been switched to redis-36379)
......
8801:X 12 Jan 2020 19:17:47.631 * +slave slave 127.0.0.1:26379 127.0.0.1 26379 @ master 127.0.0.1 36379
8801:X 12 Jan 2020 19:17:47.631 * +slave slave 127.0.0.1:16379 127.0.0.1 16379 @ master 127.0.0.1 36379
8801:X 12 Jan 2020 19:17:50.668 # +sdown slave 127.0.0.1:16379 127.0.0.1 16379 @ master 127.0.0.1 36379
......
127.0.0.1:26380> sentinel master master

 1) "name"
 2) "master"
 3) "ip"
 4) "127.0.0.1"
 5) "port"
 6) "36379"
 7) "runid"
 8) "15ea078acd8471bf2072a68c1c3f3fc570cb5cca"
 9) "flags"
10) "master"
127.0.0.1:16380> info sentinel

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=master,status=oup,address=127.0.0.1:36379,slaves=2,sentinels=3

Learn to: Redis sentinel mode and high availability cluster

97 original articles published, 10 praised, 3297 visited
Private letter follow

Tags: Redis

Posted on Sun, 12 Jan 2020 06:50:45 -0500 by ridgedale