Redis cluster deployment

1, Redis cluster related concepts
Redis supports cluster since version 3.0. It adopts the hash slot method, which can integrate multiple redis instances to form a cluster, that is, to store data on multiple nodes in the cluster.

Redis cluster is a decentralized structure. In a cluster, the identity of each master is equal. Each node saves data and the status of the whole cluster, and knows the slot that other nodes are responsible for. It also sends heartbeat information regularly, which can sense abnormal nodes in the cluster in a timely manner, and decide whether the node is unavailable by voting. If the number of votes is If more than half of the nodes in the cluster are considered unavailable, it is precisely because of this feature that redis cluster should be deployed with at least three or more nodes.

There are master and slave roles in the cluster. Slots (slots) are allocated between master roles. Slot point numbers are 0-16383 (16384 in total).

By default, each cluster node has two TCP ports listening, one is 6379 (used to listen for access connections of clients), and the other is 16379 (used for node communication between clusters). Note that the firewall needs to release traffic from both ports.

All data of Redis is saved in memory, and then saved to disk asynchronously from time to time (this is called "semi persistent mode"); every data change can also be written into an append only file(aof) (this is called "fully persistent mode").

Redis provides two methods for persistence: one is RDB persistence (the principle is to periodically record the dump of redis's database in memory to RDB persistence on disk); the other is AOF (append only file) persistence (the principle is to write redis's operation logs to files in an additional way)

Advantages and disadvantages of RDB

Advantages of RDB semi persistence:

  • It only contains one file, which is good for file backup;
  • Disaster recovery is faster than aof;
  • Maximize performance. For Redis's service process, the only thing it needs to do when it starts persistence is to fork out subprocesses, and then the subprocesses complete the persistence work, which can greatly avoid the service process from executing IO operations.
  • Compared with AOF mechanism, if the data set is too large, the start-up efficiency of RDB will be higher.
  • Disadvantages of RDB semi persistence:
  • Because RDB assists in data persistence through fork subprocesses, if the data set is large, it may cause the whole service to stop for several hundred milliseconds, or even one second

Advantages and disadvantages of AOF

Advantages of AOF full persistence:

  • High availability of data can be guaranteed;
  • In the process of writing, there will be downtime in time, and the existing content in the log file will not be damaged. If there is downtime in the process of writing, it can be solved by Redis check AOF tool after restarting Redis;
  • If the log is too large, Redis can automatically enable the rewrite mechanism to generate new files to store aof logs;
  • This mechanism can bring higher data security and data persistence. Redis provides three synchronization strategies, i.e. synchronization per second, synchronization per modification and non synchronization.
  • Disadvantages of AOF full persistence:
  • For the same number of data sets, AOF files are usually larger than RDB files. RDB can recover large data sets faster than AOF;
  • According to different synchronization policies, AOF is often slower than RDM in running efficiency. In a word, the efficiency of synchronization policy per second is relatively high, and the efficiency of synchronization disable policy is as efficient as RDB.

If RDB and AOF exist at the same time, AOF mode is preferred

2, Deploy Redis cluster
1. The environment is as follows:

There are six centos servers in total, and three master servers correspond to one slave respectively
In order to avoid cluster crash due to physical server downtime, you can also configure multiple Redis instances on one server to realize cross master-slave replication. The so-called cross is that the master is on node01, but the corresponding slave is on node02, node02, and the master's slave is on node03, while the node03's slave is on node01
The above is only for multiple servers to deploy Redis cluster.
But I mainly deploy multiple nodes on one server, so there will be multiple instances on node6
Before configuration, you can download the required software package, or you can download and prepare by yourself. Link:
Extraction code: 72hi
2. Configure Redis instance
Here, take the configuration of node01 as an example. The configuration of other nodes is basically the same as that of node1

[root@node1 /]# mkdir redis          # Personal habits
[root@node1 /]# cd redis/
[root@node1 redis]# rz            # Upload required packages
[root@node1 redis]# ls             # node nodes only need redis 4.0 package, and the other two packages are required to install ruby environment
redis-3.3.0.gem  redis-4.0.14.tar.gz  ruby-2.3.1.tar.gz
[root@node1 redis]# tar zxf redis-4.0.14.tar.gz          # Unpacking
[root@node1 redis]# mv redis-4.0.14 /usr/local/redis          # Move and rename
[root@node1 redis]# cd /usr/local/redis/          # Entry directory
[root@node1 redis]# make && make install          # Compile and install
[root@node1 redis]# ./utils/        # Initialize redis
# All initialization options remain the default. Just press enter all the way to confirm the listening port, configuration file, log file and pid store Is this ok? Then press ENTER to go on or Ctrl-C to abort
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
//Path and other information. When the information appears, it indicates that the initialization is successful
[root@node1 redis]# echo "512" > /proc/sys/net/core/somaxconn            # Next, do some optimization
[root@node1 redis]# echo  "vm.overcommit_memory = 1" >> /etc/sysctl.conf 
[root@node1 redis]# sysctl -p
vm.overcommit_memory = 1
[root@node1 redis]# echo "never" > /sys/kernel/mm/transparent_hugepage/enabled 
[root@node1 redis]# vim /etc/redis/6379.conf           # Edit profile
bind          #Find the uncommented line and change it to
daemonize yes    #If there is an annotation symbol, you need to delete it for it to take effect
cluster-enabled yes            # If there is a comment, the comment symbol needs to be removed
cluster-node-timeout 5000         # modify
appendonly yes         # Change to yes
[root@node1 redis]# /etc/init.d/redis_6379 restart        # Restart service for configuration to take effect
[root@node1 redis]# netstat -anput | grep 6379        # Make sure to listen to 6379 and 16379
tcp        0      0  *               LISTEN      44263/redis-server  
tcp        0      0 *               LISTEN      44263/redis-server  

The above configuration can be performed successively on other node servers, except node6

3. Configure multiple Redis instances of node06 host
On node06, I will configure it to run multiple Redis database instances, so it is not exactly the same as the configuration of the previous five nodes. Be careful

[root@node6 /]# mkdir redis
[root@node6 /]# cd redis/
[root@node6 redis]# rz
[root@node6 redis]# tar zxf redis-4.0.14.tar.gz 
[root@node6 redis]# mv redis-4.0.14 /usr/local/redis
[root@node6 redis]# cd /usr/local/redis/
[root@node6 redis]# make && make install            # Compile and install without initialization after installation
[root@node6 redis]# redis-server         # There are three warnings when executing this command, which are the three options we need to optimize
[root@node6 redis]# echo "512" > /proc/sys/net/core/somaxconn
[root@node6 redis]# echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf 
[root@node6 redis]# sysctl -p
vm.overcommit_memory = 1
[root@node6 redis]# echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
[root@node6 redis]# vim redis.conf          # Edit profile
port 7000          # The default port of redis is 6379. Here I change it to 7000 for easy identification
daemonize yes        #Turn on the background daemons to run in the background
cluster-enabled yes        #Open cluster
cluster-node-timeout 5000     #Timeout between cluster nodes, in milliseconds
appendonly yes       #Whether to enable synchronization to disk
appendfilename "appendonly-7000.aof"     #The name of the aof log. Because there are multiple instances on this host, each port is different. Therefore, the port is used to name the log name
[root@node6 redis]# mkdir -p /usr/local/redis-cluster/{7000..7004}
#The above is to run several Redis instances, just create a few directories. Here I name the directory with the port number of the instance (I plan to run four Redis instances for the time being)
#The following is a copy of the modified configuration file to the specified directory
[root@node6 redis]# cp redis.conf /usr/local/redis-cluster/7000/
[root@node6 redis]# cp redis.conf /usr/local/redis-cluster/7001/
[root@node6 redis]# cp redis.conf /usr/local/redis-cluster/7002/
[root@node6 redis]# cp redis.conf /usr/local/redis-cluster/7003/
[root@node6 redis]# cp redis.conf /usr/local/redis-cluster/7004/
#Next, change the port number corresponding to the Redis instance in the copied configuration files
[root@node6 redis]# cd /usr/local/redis-cluster/
[root@node6 redis-cluster]# ls
7000  7001  7002  7003  7004
[root@node6 redis-cluster]# sed -i s/7000/7001/g 7001/redis.conf 
[root@node6 redis-cluster]# sed -i s/7000/7002/g 7002/redis.conf 
[root@node6 redis-cluster]# sed -i s/7000/7003/g 7003/redis.conf 
[root@node6 redis-cluster]# sed -i s/7000/7004/g 7004/redis.conf 
[root@node6 redis-cluster]# cd 7000/         # Enter 7000 instance directory
[root@node6 7000]# redis-server redis.conf         # Start the instance
8142:C 13 Feb 10:44:17.501 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
8142:C 13 Feb 10:44:17.501 # Redis version=4.0.14, bits=64, commit=00000000, modified=0, pid=8142, just started
8142:C 13 Feb 10:44:17.501 # Configuration loaded
# To start an instance, enter the instance directory one by one, and I won't show you

4. The host node01 installs and configures the running environment of ruby to facilitate the management of Redis cluster

[root@node1 /]# yum -y install rpm-build openssl openssl-devel       # Install required dependency package
[root@node1 /]# cd /redis/
[root@node1 redis]# ls
redis-3.3.0.gem  redis-4.0.14.tar.gz  ruby-2.3.1.tar.gz
[root@node1 redis]# tar zxf ruby-2.3.1.tar.gz         # Unpacking
[root@node1 redis]# cd ruby-2.3.1/
[root@node1 ruby-2.3.1]# ./configure --prefix=/usr/local/ruby && make && make install          # This environment takes a long time to compile
[root@node1 ruby-2.3.1]# ln -s /usr/local/ruby/bin/* /usr/local/bin/         # Making soft connections for commands
[root@node1 ruby-2.3.1]# ln -s /usr/local/redis/src/redis-trib.rb /usr/local/bin/
[root@node1 ruby-2.3.1]# ln -s /usr/local/ruby/bin/* /usr/bin/
[root@node1 ruby-2.3.1]# cd ..
[root@node1 redis]# gem install redis-3.3.0.gem       # Go back to the directory with the gem file and install it
Successfully installed redis-3.3.0
Parsing documentation for redis-3.3.0
Installing ri documentation for redis-3.3.0
Done installing documentation for redis after 0 seconds
1 gem installed
# The above information indicates that the installation is successful

5. Configuring nodes in a cluster

[root@node1 /]# redis-cli -p 6379          # Log in to ben'di local instance> CLUSTER MEET 6379       # Add nodes to the cluster
OK> CLUSTER MEET 7000     # This is because node6 has changed the port number
OK> set test aaa             # Data cannot be added because there is no hash slot assigned
(error) CLUSTERDOWN Hash slot not served> CLUSTER INFO           # View the status of the cluster
cluster_state:fail                   # You'll find that this is a fail ure
cluster_slots_assigned:0> exit

# Come on down, we need to allocate slot points carefully
#Although I could change "add" to "del" in the command, I failed
#Be sure to allocate 0 to 16383 completely, preferably equally
#Just assign the node as the master node. Here, node01 to node03 are the master nodes
[root@node1 /]# redis-cli -h -p 6379 cluster addslots {0..5461}
[root@node1 /]# redis-cli -h -p 6379 cluster addslots {5462..10922}
[root@node1 /]# redis-cli -h -p 6379 cluster addslots {10923..16383}
[root@node1 /]# redis-cli -p 6379 -c          # To enter the cluster, you need to add the - c option> CLUSTER NODES            # View cluster node information
#Next, bind each slave node to the master
#node04 is the slave node of node01, node05 is the slave node of node02, and the 6379 instance of node06 is the slave node of node03
#Which slave node needs to be configured, which instance needs to be logged in to> CLUSTER NODES     #You can execute this command first to view the ID of the corresponding node so that you can specify
[root@node1 /]# redis-cli -h -p 6379        # Log in to node4> CLUSTER REPLICATE 4397bb3e5e43fdf1700551d0a2587f502e6c02a8
#The node id of node1 is added above> exit       # Exit the node
[root@node1 /]# redis-cli -h -p 6379> CLUSTER REPLICATE 1eca51f415ba5194a95d5400176daa5713d64990
#The node id of node2 is added above> exit
[root@node1 /]# redis-cli -h -p 7000> CLUSTER REPLICATE ee71a2fd995f35992ce69328096ccd50fc68f89e
#The node id of node3 is added above

At this point, the cluster can read and write data normally, as follows:

[root@node1 /]# redis-cli -h -p 6379 -c> set test aaa
-> Redirected to slot [6918] located at
OK> get test

6. Manage Redis cluster with the command of ruby installation

[root@node1 /]# redis-trib.rb check

7. Add node 7001 instance of node06 to Redis cluster
Add node

[root@node1 /]# redis-trib.rb add-node
#When adding nodes, no other configuration will be added. After joining the cluster by default, the role is master

Corresponding slot points need to be allocated after adding nodes
In order for a cluster to operate normally, all slot points must be allocated, so when new nodes are added, they need to be allocated slot points again, as follows:

[root@node01 /]# redis-trib.rb check    #When executing this command to confirm the newly added node, master
[root@node01 /]# redis-trib.rb reshard   #Specify the cluster address and port
How many slots do you want to move (from 1 to 16384)? 4096
#If there are four master s, the average value is 4096, so enter 4096 here
What is the receiving node ID? 
#Specifies the ID of the receiving node, that is, the instance 7001 on the newly added node06 host   
Source node #1:all   #Specify the slot point assignment from which node, enter "all" here to select all nodes
Do you want to proceed with the proposed reshard plan (yes/no)? yes    #Enter "yes" to confirm

At this point, the new node is added and the corresponding slot points are allocated, but there is no slave node, so next, assign a slave node to the newly added master
8. Assign slave nodes to newly joined master
There are two ways to assign a slave node: one is to automatically bind to the master without the slave node, and the other is to directly specify the master to which the slave node is bound. Write down both ways
Method 1:

[root@node1 /]# redis-trib.rb add-node --slave
#Join the 7002 instance on node06 as a slave to the cluster
#Note that the returned information must not be in red, which means there is an error
[root@node1 /]# redis-trib.rb check

Method two:
[root@node1 /]# redis-trib.rb add-node --slave --master-id eca1db0392f69207c4fc3ad4ff290d5a94ea5f99
#The above id is the id of 7001master
[root @ node1 /] ා redis-trib.rb check 6379 ා if you check the cluster status, you will find that 7001 has two slave nodes

9. Delete master operation
The operation of deleting the master node is actually to reverse the operation of adding the master node. You need to assign the slot points on the master node to be deleted to other masters before you can perform the operation. After deleting the master node, the slave corresponding to the master node will also be transferred to the new master along with the slots.

Here is an example of removing the 7001 instance on node06

[root@node1 /]# redis-trib.rb reshard

How many slots do you want to move (from 1 to 16384)? 4096  # Specify how many slot points to delete
What is the receiving node ID? 1eca51f415ba5194a95d5400176daa5713d64990
#The id added above is to whom the remaining slot points are assigned after deleting the primary node. Here I add the id of node2
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:eca1db0392f69207c4fc3ad4ff290d5a94ea5f99
#The node to be deleted is specified above. Here is 7001 on node6
Source node #2:done    # End
Do you want to proceed with the proposed reshard plan (yes/no)? yes
#After removing the slot point, next remove the 7001 instance from the cluster as follows:
[root@node1 /]# redis-trib.rb del-node eca1db0392f69207c4fc3ad4ff290d5a94ea5f99
>>> Removing node eca1db0392f69207c4fc3ad4ff290d5a94ea5f99 from cluster               # The specified id is 7001
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

At this point, the 7001 instance has been completely removed from the cluster, and the slave corresponding to it has also become the slave of node02 along with the transfer of slot points to node02. Now look at the cluster information. The master of node02 should have three slaves

Tags: Redis Ruby Database vim

Posted on Thu, 13 Feb 2020 07:28:09 -0500 by deezzer