Centos7 redis sentinel mode setup

preface

   the actual MES project needs Redis cache, and single Redis is not suitable for formal environment deployment. Therefore, consider Redis server cluster to realize master-slave configuration and disaster recovery deployment, so that disaster recovery can be switched automatically in case of host failure. Here is a detailed explanation on how to use Redis to realize it.

Article focus

1, Introduction to Redis

  • Redis (Remote Dictionary Server remote Dictionary Service) is an open source high-performance key value database. It is usually called a data structure server because values can be string, hash, list, sets and sorted sets. The official website is: https://redis.io/ .
  • Redis supports data persistence. It can save the data in memory on disk and can be loaded again for use when restarting.
  • Redis not only supports simple key value data, but also provides storage of list, set, zset, hash and other data structures.
  • Redis supports data backup, that is, data backup in master slave mode.

Why Redis?
  one word, come on! Traditional relational databases such as Mysql can no longer be applied to all scenarios. For example, in the case of high concurrency and peak access traffic, the database is easy to collapse. Redis runs in memory and can play a buffer role. Because the reading and writing speed of memory is much faster than that of hard disk, redis has obvious advantages over other databases based on hard disk storage in performance.
  besides being fast, it can also be applied to automatic disaster recovery switching of clusters and data read-write separation to reduce the pressure of high concurrency.

2, Redis installation and deployment

1. Download the installation package

Official website address: https://redis.io/download

2. Upload and unzip the virtual machine

  bloggers upload redis-6.2.5.tar.gz through FileZilla

Unzip the package

tar -zxvf redis-6.2.5.tar.gz

Enter the unzipped file directory and see the Makefile file, indicating that the installation needs to be compiled

Use the make command to compile the extracted redis file, as follows:

make

Error message: cc command does not exist

gpcc needs to be installed

yum install gcc -y

Continue to execute make or report an error:

After adding MALLOC=libc parameter, execute:

make MALLOC=libc

3. Deployment

   because there are many files under src, we can copy several common commands and conf configuration files for unified management, as follows:
(1) Create new bin and etc folders

cd  /root/redis-6.2.5
mkdir etc
mkdir bin

(2) Go back to the installation directory and copy redis.conf to the ect folder

cp redis.conf /root/redis-6.2.5/etc

(3) Enter the src folder and copy the mkreleasehdr.sh, redis benchmark, redis check AOF, redis check RDB, redis cli, redis server and redis sentinel files to the bin folder

cd src/
cp mkreleasehdr.sh redis-benchmark redis-check-aof redis-check-rdb redis-cli redis-server redis-sentinel /root/redis-6.2.5/bin/

View the copied file as follows:

4. Start redis service

   switch to the bin directory and use. / redis server to start the redis service. However, this method does not specify the configuration file, and redis will adopt the default configuration, so we need to start redis according to our configuration file, as shown below. During startup, specify the redis.conf that we copied to the etc folder just now.

#Redis can be set_ Home, so you don't have to start in the specified directory
vim /etc/profile
export REDIS_HOME=/root/redis-6.2.5
export PATH=$REDIS_HOME/bin:$PATH

source /etc/profile
redis-server /root/redis-6.2.5/etc/redis.conf


  because the default configuration file is used instead of silent startup, we open another connection session and check the redis process. As shown below, we can see that port 6379 is the redis service we run (the default port in the configuration file is 6379).

Use the redis cli command to connect to the client, as follows:

/root/redis-6.2.5/bin/redis-cli  -p 6379

Note: ctrl+c exits the command side of redis cli

5. Turn off redis service

If the redis service needs to be shut down, use the following command to shut it down:

/root/redis-6.2.5/bin/redis-cli -p 6379 shutdown

3, Overall Redis cluster architecture

   the overall cluster architecture we adopt here is master-slave structure + sentinel to realize automatic switching of disaster recovery, as shown in the following figure:

A master node can have multiple slave nodes. The slave node can replicate the master node to ensure data synchronization. The sentinel monitors each node, mainly including master node survival detection, master-slave operation detection, etc. once the master node goes down, the sentinel can automatically perform failover and master-slave switching.
Next, we will start to build such a cluster, first the master-slave structure, then the sentinel mode, and then look down.

4, Redis master-slave configuration and data synchronization

In the above Redis installation and deployment, we have started the Redis service, but the configuration file has not been modified. Because the master-slave configuration is mainly realized by modifying the configuration file, the modification of Redis configuration file is explained here.
Here, I created three virtual machines to demonstrate, and installed Redis according to the above installation method. The three virtual machines are configured as follows:

IP addressPort numberrole
192.168.45.476379host
192.168.45.486379Slave
192.168.45.496379Slave

First, close the firewall of the three virtual machines (otherwise, set the access port):

systemctl stop firewalld  
systemctl disable firewalld 
systemctl status firewalld 

redis is installed on 48 and 49 machines respectively, and files can be copied through scp:

scp root@192.168.45.47:/root/redis-6.2.5.tar.gz /root/redis-6.2.5.tar.gz

Now enter the etc folder and use the vim redis.conf command to open and edit the redis.conf configuration file, as follows:

cat redis.conf|grep -v "#"|grep -v "^$"

First, take a look at the parameters in the redis.conf configuration file. The details are as follows:

# Whether the redis process runs as a daemon. Yes is yes and no is no (if it does not run as a daemon, it will occupy a terminal).
daemonize no
# Specify the PID file storage location of the redis process
pidfile /var/run/redis.pid
# The port number of the redis process
port 6379
#Whether to enable the protection mode. It is enabled by default. If the bind and password are not specified in the configuration. After this parameter is enabled, redis will only access locally and refuse external access. If the password and bind are enabled, they can be enabled. Otherwise, it is better to turn off and set it to no.
protected-mode yes
# Bound host address
bind 127.0.0.1
# After the client is idle for a long time, close the connection. The default parameter is 0, that is, close this function
timeout 300
# redis log level. The available levels are debug.verbose.notice.warning
loglevel verbose
# Log file output location. If the process runs as a daemon and the output file is set to stdout here, the log information will be output to / dev/null
logfile stdout
# Set the number of databases. The default value is 0. You can use the select < dbid > command to specify the database id on the connection
databases 16
# Specifies how many times the data will be synchronized to the data file in how long
save <seconds> <changes>
# Specifies whether to compress the file when storing to the local database. The default value is yes, which enables storage
rdbcompression yes
# Specify the local database file name
dbfilename dump.db
# Specify local data storage location
dir ./
# Specify to set the IP address and port of the master service when the local machine is a slave service. When redis is started, it will automatically synchronize data with the master
replicaof <masterip> <masterport>
# When password protection is set for the master, the slave service uses the password to connect to the master
masterauth <master-password>
# Set the redis connection password. If the connection password is configured, the client needs to provide the password through the auth < password > command when connecting to redis. It is closed by default
requirepass footbared
# Set the maximum number of customer connections at the same time. It is unlimited by default. The number of clients that redis can connect to at the same time is the maximum file descriptor that redis can open. If maxclients 0 is set, it means there is no limit. When the number of client connections reaches the limit, redis will close the new connection and return the max number of clients reached error message to the client
maxclients 128
# Specify the maximum memory limit of redis. When redis starts, it will load data into memory. After reaching the maximum memory, redis will first try to clear expired or about to expire keys. After this method is processed, the maximum memory setting is still reached. No more write operations can be performed, but read operations can still be performed. Redis's new vm mechanism will store the Key in memory and the Value in the swap area
maxmemory<bytes>
# Specify whether to log after each update operation. Redis writes data to disk asynchronously by default. If it is not enabled, it may cause data loss for a period of time during power failure. Because redis synchronizes data files according to the above save conditions, some data will only exist in memory for a period of time. The default is no.
appendonly no
# Specify the new log file name. The default is appendonly.aof
appendfilename appendonly.aof
# Specify the conditions for updating the log. There are three optional parameters - no: it means that the operating system will synchronize the data cache to the disk (fast), always: it means that fsync() is called manually to write the data to the disk after each update operation (slow, safe), and everysec: it means to synchronize once per second (compromise, default value);
appendfsync everysec

1. Host configuration

   the configuration of the host (192.168.45.47) to be changed is as follows. After modification, press esc, and then: wq command to save and exit.
The following points need to be modified:

#Each computer binds its own IP address
bind 192.168.45.47
protected-mode no
#The logs directory needs to be created
logfile /root/redis-6.2.5/logs/redis.log
daemonize yes

2. Slave configuration

   the configuration of the slave is similar to that of the host. I won't explain in detail the same places. The difference is that replica of needs to be used to specify the IP address and port of the master. It should be noted that the old version uses slavof, and the current version 6.2.5 uses replica of, as follows:

#You can use scp to copy the configuration of the host first
scp root@192.168.45.47:/root/redis-6.2.5/etc/redis.conf /root/redis-6.2.5/etc/redis.conf
#Change the bind of the slave
bind 192.168.45.49
#Set replication host
replicaof 192.168.45.47 6379

replicaof 192.168.45.47 6379 specifies that when the local machine is a slave service, set the IP address and port of the master service, and automatically synchronize data with the master when redis is started, so both slaves can be configured in this way.
Then start the redis service. Note that you can observe the startup sequence. The following is that the slave will be started first and will continue to connect to the host. If the host is not started, an error will be reported until the host is successfully online

redis-server /root/redis-6.2.5/etc/redis.conf


So far, the master-slave structure has been built. Unexpectedly, the master-slave can synchronize data:

redis-cli -h 192.168.45.47

3. Master-slave verification

Next, we add several pieces of data to the master to see if the slave can get it. If it can, it means that the data has been synchronized to the slave. The master adds data as follows:

Get data from:

Slave write failed, indicating that the slave is read-only

5, Redis sentinel mode setup

1. Detailed explanation of sentry mode

  • Redis Sentinel is redis's high availability solution, which consists of one or more Sentinel instances. It can monitor any number of master servers and all slave servers under these master servers, and automatically upgrade a slave server under the offline master server to a new master server when the monitored master server enters the offline state. Its main functions are as follows:
  • Monitoring: Sentinel will constantly check whether your master server and slave server are working normally.
  • Notification: when a monitored Redis server has a problem, Sentinel can send a notification to the administrator or other applications through the API.
    Failover: when the primary server fails to work normally, Sentinel will automatically perform failover, that is, master-slave switching.
  • Unified configuration management: the connector asks sentinel for the master-slave address.
Sentinel principle

   the core algorithm used by Sentinel is the Raft algorithm, which is mainly used for distributed system, system fault tolerance and Leader election. Each Sentinel needs to perform the following tasks regularly:

  • Each Sentinel will automatically discover other Sentinel and slave servers. It sends a PING command to the master server, slave server and other Sentinel instances it knows once a second.
  • If the time of an instance from the last valid reply to the PING command exceeds the value specified by the down after milliseconds option, the instance will be marked as offline by Sentinel. Valid replies can be: + PONG, - LOADING or - MASTERDOWN.
  • If a master server is marked as subjective offline, all sentinels monitoring the master server should confirm that the master server has indeed entered the subjective offline state once per second.
  • If a master server is marked as subjective offline and a sufficient number of sentinels (at least up to the number specified in the configuration file) agree with this judgment within the specified time range, the master server is marked as objective offline.
  • In general, each Sentinel will send INFO commands to all master and slave servers it knows once every 10 seconds. When a master server is marked as objectively offline by Sentinel, the frequency of Sentinel sending INFO commands to all slave servers of the offline master server will be changed from once in 10 seconds to once per second.
  • When a sufficient number of sentinel agree that the primary server has been offline, the objective offline status of the primary server will be removed. When the master server returns a valid reply to Sentinel's PING command again, the supervisor offline status of the master server will be removed.
Detailed description of configuration file

Sentinel configuration is mainly to modify the parameters in sentinel.conf configuration file, which can be seen in Redis installation directory. The details of each parameter are as follows:

# The port on which the sentinel sentinel instance runs. The default is 26379  
port 26379
# Whether to set as background startup.
daemonize no
#pid file address
pidfile /var/run/redis-sentinel.pid
#Log file address
logfile ""
# Specify sentinel working directory
dir /tmp
# redis master node monitored by sentinel 
## ip: host ip address
## Port: sentinel port number
## Master Name: the name of the master node that can be named by itself (it can only be composed of letters A-z, numbers 0-9 and the three characters ". -.)
## quorum: when sentinel considers that the master master node is lost, it is objectively considered that the master node is lost  
# sentinel monitor <master-name> <ip> <redis-port> <quorum>  
sentinel monitor mymaster 127.0.0.1 6379 2
# Specify the maximum time interval for the master node to respond to the sentinel. Beyond this time, the sentinel subjectively thinks that the master node is offline. The default is 30 seconds  
# sentinel auth-pass <master-name> <password>  
sentinel down-after-milliseconds mymaster 30000
acllog-max-len 128
# Specifies the maximum number of slaves that can synchronize the new master at the same time when a failover active / standby switch occurs. The smaller the number, the longer it takes to complete the failover; On the contrary, if the number is larger, it means that more slaves are unavailable due to replication. You can set this value to 1 to ensure that there is only one slave at a time and that it cannot process command requests.
# sentinel parallel-syncs <master-name> <numslaves>
sentinel parallel-syncs mymaster 1
# The failover timeout, which is three minutes by default, can be used in the following aspects:
## 1. The interval between two failover of the same sentinel to the same master.  
## 2. When a slave synchronizes data from a wrong master, it starts until the slave is corrected to synchronize data from the correct master.  
## 3. The time required to cancel an ongoing failover.
## 4. During failover, configure the maximum time required for all slaves to point to the new master. However, even after this timeout, the slaves will still be correctly configured to point to the master, but the data will not be synchronized according to the rules configured by parallel syncs
# sentinel failover-timeout <master-name> <milliseconds>  
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no

2. Sentry building

Here we deploy three sentinels, one for each server. The configuration method is the same, as follows:

cd /root/redis-6.2.5
cp sentinel.conf  /root/redis-6.2.5/etc/sentinel.conf

Edit sentinel.conf and make the following configuration changes (all three are changed):

daemonize yes
logfile /root/redis-6.2.5/logs/sentinel.log
sentinel monitor mymaster 192.168.45.47 6379 2

3. Activate three sentinels:

redis-sentinel /root/redis-6.2.5/etc/sentinel.conf

After all three sentinels are activated, you can use the following command to view sentinel information:

redis-cli -h 192.168.45.47 -p 26379
info sentinel


4. Disaster recovery switching
Now let's simulate the host downtime and shut down the host redis service (and shut down sentinel), as follows:

Now let's look at the three servers and find that the host (192.168.45.48 6379) has become the host as follows:

However, sentinel was also hung here, so it was not immediately migrated to 48. It took nearly 10 minutes to wait. This is based on the redis.conf configuration of another slave:

 cat redis.conf |grep -v "#"|grep -v "^$"|grep "rep"

This has been changed to a new host (48), so the migration of the host will also change the configuration file

Therefore, before starting the 47 server that was previously hung up, you need to change the configuration file to configure it as a 48 Slave:

vim /root/redis-6.2.5/etc/redis.conf
replicaof 192.168.45.48 6379
#Start after saving 47
redis-server /root/redis-6.2.5/etc/redis.conf

At this time, redis cli -h 192.168.45.48 is used to see the connection:

47 has been connected normally, and sentinel needs to be started at the same time:

redis-sentinel /root/redis-6.2.5/etc/sentinel.conf
redis-cli -h 192.168.45.47 -p 26379
info sentinel

As you can see here, sentinel has been connected to the new host

After viewing the configuration file, it is found that the following information is added at the end of the configuration file:

So far, the redis sentinel mode has been set up

Tags: CentOS Redis

Posted on Sun, 26 Sep 2021 19:41:15 -0400 by rahuul