Redis High Availability Cluster - Build (Master, Sentry, Cluster)

1. Build master-slave replication

  • Environmental Science
Node name	IP address

1. Install redis

  • All three servers need to be installed
#Turn off security policy
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
  • Copy the following script

systemctl stop firewalld
systemctl disable firewalld
setenforce 0

yum -y install gcc gcc-c++ make

cd /opt
if [ ! -f " redis-5.0.7.tar.gz" ]; then

tar zxvf redis-5.0.7.tar.gz -C /opt/
cd redis-5.0.7/
make PREFIX=/usr/local/redis install

yum -y install tcl expect
cd /opt/redis-5.0.7/utils/
/usr/bin/expect <<EOF
spawn ./
expect "instance" {send "\r"} 
expect "config" {send "\r"}
expect "log" {send "\r"}
expect "data" {send "\r"}
expect "executable" {send "/usr/local/redis/bin/redis-server\r"}
expect "abort" {send "\r"}
expect eof

ln -s /usr/local/redis/bin/* /usr/local/bin/

sed -i '/bind bind' /etc/redis/6379.conf
sed -i 's/appendonly no/appendonly yes/' /etc/redis/6379.conf

/etc/init.d/redis_6379 restart
/etc/init.d/redis_6379 status

netstat -natp | grep "redis"

pgrep "redis" &> /dev/null
if [ $? -eq 0 ];then
        echo -e "\033[46;37m Redis Service is running normally \033[0m"
        echo -e "\033[46;37m Redis Service is running abnormally, please check \033[0m"
sleep 2
echo ' '
echo -e "\033[46;37m Redis No password set, execute redis-cli Command Logon \033[0m"

2. Modify Redis Profile

  • master node
vim /etc/redis/6379.conf

#70 lines, modified listening address

daemonize yes                       
#Line 137, start daemon

logfile /var/1og/redis_6379.1og    
#172 lines, specify log file directory

dir /var/lib/redis/6379             
#264 lines, specify working directory

appendonly yes                      
#700 lines, turn on AOF persistence

/etc/init.d/redis_6379 restart
#Restart service for configuration to take effect
  • Slave1/2 Node
vim /etc/redis/6379.conf

#70 lines, modified listening address

daemonize yes                       
#Line 137, start daemon

logfile /var/log/redis_6379.log   
#172 lines, specify log file directory

dir /var/lib/redis/6379             
#264 lines, specify working directory

replicaof 6379       
#288 lines, specifying the Master node IP and port to synchronize

appendonly yes                      
#700 lines, turn on AOF persistence

/etc/init.d/redis_6379 restart
#Restart service for configuration to take effect

3. Verify the master-slave effect

  • Primary Node View Log
tail -f /var/log/redis_6379.log

  • Validation at Primary Node
[root@c7-1 ~]# redis-cli info replication
# Replication
master_replid:a1e500591376020abc8f501177a9d1bf8a506350	//A 40-bit hexadecimal random string generated at master startup to identify the master node
master_replid2:0000000000000000000000000000000000000000	//Master node identity changes when switching master slave

4. Error investigation

① WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128
 The current maximum listening queue length per port does not meet this high load environment and needs to be adjusted

echo 2048 > /proc/sys/net/core/somaxconn

② WARNING overcommit_memory is set to 0! Background save may fail under low memory condition
 Memory Excess Warning, current memory set to 0 will cause background save to fail

echo "vm.overcommit_memory=1" > /etc/sysctl.conf
#Refresh profile to make it valid
sysctl vm.overcommit_memory=1

③ WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis
 Large transparent pages are enabled in the kernel ( THP)Support will result in Redis Delay and memory usage issues with

echo never > /sys/kernel/mm/transparent_hugepage/enabled

④ Error condition on socket for SYNC: Connection reset by peer
 Connection refused because the primary server may be bound to itself IP address

 Primary Node Profile

2. Set up Sentinel Mode

  • Acts as the primary node automatic failover based on master-slave replication

1. Functions and Principles

1.1 Role of Sentry Mode

  • Monitor
    Sentinels constantly check whether the primary and secondary nodes are functioning properly
  • Automatic Failover
    When the primary node is not functioning properly, the Sentry will start an automatic failover operation, which will upgrade one of the failed primary nodes to a new primary node and change the other secondary nodes to replicate the new primary node
  • notice
    Sentinels can send failover results to clients

1.2 Principles of Sentry Mode

  • It is a distributed system for monitoring each server in the master-slave architecture, selecting a new Master through a voting mechanism and connecting all slaves to a new Master when a failure occurs

  • The number of entire Sentinel-running clusters must not be less than three nodes

1.3 Structure

  • The sentinel structure consists of two parts, the sentinel node and the data node:

  • Sentinel Nodes
    A sentinel system consists of one or more sentinel nodes, which are special redis nodes and do not store data

  • Data Node
    Both primary and secondary nodes are data nodes

1.4 Work Process

  • Sentinel startup depends on master-slave mode, so you have to install master-slave mode before you go to sentry mode, so you need to deploy sentry mode on your node, which monitors whether all Redis working nodes are working properly

  • When a Master has a problem, because the other nodes are out of contact with the primary node, they vote, voting more than half to think that the Master is indeed a problem, and then notify the sentry that a Sentry will be selected to fail over (which slave will be designated by the sentry to make the new master). Then choose Slaves as the new Master filter by sending messages to each other and voting, with the majority of votes selected

  • It is important to note that an objective offline is a concept only available to the primary node, that is, if a failure occurs from a node and a sentinel node and the sentinel is subjectively offline, there will be no subsequent objective offline and failover operations (and the sentinel mode will only be responsible for aspects of Master, regardless of Slaves)

  • When a sentry finds that the master server has been dropped, it changes the master in SentinelRedistance s in master to SRI_S_DOWN (Subjective Offline) and notify other sentinels that they find the master has dropped the message sent by the other sentinels and attempt to connect to the master. If more than half (set in the configuration file) confirm that the master has been dropped, the master in Sentinel Redistance in the master will be changed to SRI_O_DOWN (objective offline)

2. Environment

Node name	IP address

3. Modify Sentry Profile

vim /opt/redis-5.0.7/sentinel.conf

#17 lines, turn off protection mode
protected-mode no		

#Line 21, Redis Sentry default listening port
port 26379			

#26 rows to start daemon
daemonize yes		

#36 lines, specify log storage path
logfile "/var/log/sentinel.log"		

#65 rows, specify the database store path
dir "/var/lib/redis/6379"		

#84 lines, designate sentinel nodes
#2 indicates that at least two Sentinel nodes are required to agree to fail over the primary node
sentinel monitor mymaster 6379 2 	

#113 lines, determines the time period for the server to down load, defaulting to 30,000 milliseconds (30 seconds)
sentinel down-after-milliseconds mymaster 3000 		

#146 lines with a maximum timeout of 180,000 (180 seconds) for the failing node
sentinel failover-timeout mymaster 180000	

4. Start Sentry Mode

  • Start Primary Node First Start Slave Node
cd /opt/redis-5.0.7/
redis-sentinel sentinel.conf &
  • View Sentry Information
[root@master redis-5.0.7]# redis-cli -p 26379 info sentinel
# Sentinel

  • Analog failure
Simulated failure ( rm -rf /var/run/

#View the process number of redis-server
[root@c7-1 redis-5.0.7]# netstat -anpt|grep redis
tcp        0      0 *               LISTEN      9818/redis-sentinel 
tcp        0      0  *               LISTEN      7616/redis-server 0 

kill -9 7616
#Process number to kill redis-server on Master node

  • View sentry information again
[root@c7-1 redis-5.0.7]# redis-cli -p 26379 info sentinel
# Sentinel

o That is objectively ,objective

#View master sentry log
 Verification results
tail -f /var/log/sentinel.log

redis-cli -p 26379 info Sentinel

3. Setting up Cluster Cluster

  • The primary node is responsible for the maintenance of read and write requests and cluster information, and only replicates master node data and status information from the secondary node

1. Role

1.1 Data Partition

  • Data partitioning (or data fragmentation) is the core function of a cluster
  • Clusters distribute data among multiple nodes. On the one hand, they break the limit of Redis single-machine memory size and greatly increase the storage capacity. On the other hand, each primary node can provide read and write services to the outside world, which greatly improves the responsiveness of the cluster.
  • Redis single-machine memory size limitation, mentioned in both persistence and master-slave replication
  • For example, if single-machine memory is too large, fork operations of bgsave and bgrewriteaof may cause the master process to block, switch between master and slave environments may cause the slave node to be unable to provide service for a long time, and the replication buffer of the master node may overflow during the full replication phase

1.2 High Availability

  • Clusters support master-slave replication and automatic failover of primary nodes (similar to sentinels), which can still serve out when any node sends a failure

1.3 Data fragmentation

  • Redis Cluster introduced the concept of hash slots, with 16384 hash slots (numbered 0~16383)
  • Each node of the cluster is responsible for a portion of the hash slot, and each Key, after the CRC16 check, redeems 16384 to determine which hash slot to place. With this value, the corresponding node of the slot is found, and then jumps directly to the corresponding node for access.
  • Take a three-node cluster as an example:
    Node A contains hash slots between 0 and 5469
    Node B contains hash slots 5461~10922
    Node C contains hash slots from 10923 to 16383

2. Environment

  • A redis cluster typically requires six nodes, three masters and three slaves. For convenience, all nodes here are simulated on the same server:
  • Differentiate by port number: three primary node port numbers: 6001/6002/6003, corresponding slave node port number: 6004/6005/6006

3. Create Port Working Directory

  • Create working directory for redis 6 ports
cd /etc/redis/
mkdir -p redis-cluster/redis600{1..6}

#One-click Copy of 6001~6006 Scripts
vim /opt/
for i in {1..6}
cp /opt/redis-5.0.7/redis.conf /etc/redis/redis-cluster/redis600$i
cp /opt/redis-5.0.7/src/redis-cli /opt/redis-5.0.7/src/redis-server /etc/redis/redis-cluster/redis600$i

sh -x /opt/

4. Modify the configuration file

cd /etc/redis/redis-cluster/redis6001
vim redis.conf

#69 lines, comment out the bind item or do not modify, default listens on all network cards

protected-mode no
#88 lines, change, turn off protection mode

port 6001
#92 lines, modification, redis listening port,

daemonize yes
#Line 136, start daemon, start as independent process

cluster-enabled yes
#832 lines, uncomment, cluster on

cluster-config-file nodes-6001.conf
#840 lines, uncomment, cluster name file settings

cluster-node-timeout 15000
#846 lines, uncomment cluster timeout setting

appendonly yes
#700 lines, modify, turn on AOF persistence

The other five profiles have the same changes except for port number and cluster profile name
cp redis.conf ../redis6002/
cp redis.conf ../redis6003/
cp redis.conf ../redis6004/
cp redis.conf ../redis6005/
cp redis.conf ../redis6006/
  • Write startup scripts
#Start redis based on the corresponding profile
vim /opt/
for d in {1..6}
cd /etc/redis/redis-cluster/redis600$d
redis-server redis.conf
ps -ef | grep redis

sh -x /opt/

5. Join the cluster

redis-cli --cluster create --cluster-replicas 1

#The six instances are divided into three groups, one master and one slave for each group, the former being the master node, and the latter being the slave node.
#You need to enter yes to create the following interaction
#-replicas 1 indicates that each primary node has a slave node

6. Test Cluster

redis-cli -p 6001 -c         #With the -c parameter, nodes can jump to each other> cluster slots    #View the hash slot number range of nodes
1) 1) (integer) 5461
   2) (integer) 10922        #Hash slot number range
   3) 1) ""
      2) (integer) 6003        #Primary node IP and port number
      3) " fdca661922216dd69a63a7c9d3c4540cd6baef44"
   4) 1) ""
      2) (integer) 6004        #From Node IP and Port Number
     3) " a2c0c32aff0f38980accd2b63d6d952812e44740"
2) 1) (integer) 0
   2) (integer) 5460
   3) 1) ""
      2) (integer) 6001
      3) "0e5873747a2e26bdc935bc76c2bafb19d0a54b11"
      4) 1) ""
         2) (integer) 6006
         3) "8842ef5584a85005e135fd0ee59e5a0d67b0cf8e"
3) 1) (integer) 10923
   2) (integer) 16383
   3) 1) ""
      2) (integer) 6002
      3) "81 6ddaa3d1469540b2ffbcaaf9aa867646846b30"
   4) 1) ""
      2) (integer) 6005
      3) " f847077bfe6722466e96178ae8cbb09dc8b4d5eb"

  • Create data in 6001 and view his slots> set name zhangsan	
-> Redirected to slot [5798] located at	//Slice to 6002 for storage
OK> cluster keyslot name	//View the slot number of the name key
(integer) 5798			//Slot 5798 belongs to 6002 nodes

Tags: Database Redis Cache

Posted on Fri, 12 Nov 2021 17:34:26 -0500 by adnan856