Quick Understanding LVS Load Balancing Clusters


In a variety of Internet applications, as the site requires more and more hardware performance, response speed, service stability, data reliability, etc., it will be difficult for a single server to undertake all access. In addition to expensive mainframes and dedicated load-splitting devices, companies have another option to solve the problem by building cluster servers that provide the same services at the same address by integrating multiple relatively inexpensive common servers.

1. Overview of Clustering Technology

Clusters (clusters) represent a large collection of servers in the server realm to distinguish them from a single server.

1. Type

Regardless of the cluster, you need to include at least two node servers, which are represented as a whole and provide only one access entry (domain name or IP), equivalent to a large computer.

There are three types of clusters based on the target differences they target:

1.1 Load Balancing Cluster

  • The goal is to improve the responsiveness of the application system, handle as many access requests as possible, and reduce latency so as to achieve high concurrency and high load (LB) overall performance
  • For example, "DNS Polling", "Application Layer Switching", "Reverse Proxy" and so on can all act as load balancing clusters
  • LB's load allocation relies on the shunting algorithm of the primary node, which allocates access requests from clients to multiple server nodes, thereby alleviating load pressure on the entire system

1.2 Highly Available Clusters

  • To improve the reliability of the application system and minimize interruption time, ensure the continuity of service and achieve high availability (HA) fault tolerance
  • For example, "failover", "dual-machine warm-up", "multiple-machine hot-up" and so on are highly available cluster technologies
  • HA works in two modes, duplex and master-slave, where all nodes are online simultaneously. The master-slave node is only online, but it can automatically switch to the master node when a failure occurs

1.3 High Performance Operations Cluster

  • To improve the CPU speed of the application system, expand hardware resources and analysis capabilities, and obtain high performance computing (HPC) capabilities equivalent to large supercomputers
  • For example, Cloud Computing and Grid Computing can also be considered a high-performance operation
  • High performance relies on "distributed computing", "parallel computing". By integrating CPU, memory and other resources from multiple servers with dedicated hardware and software, only large, supercomputers can achieve computing power.

2. Layered structure of load balancing

In a typical load balancing cluster, there are three levels of components, as shown in the following figure

  • At least one load balancing dispatcher on the front end is responsible for responding to and distributing access requests from clients
  • The backend consists of a pool of real servers that provide the actual application services, and the scalability of the entire cluster is accomplished by adding and removing server nodes, which are transparent to clients
  • To maintain service consistency, all nodes use shared storage devices

2.1 Load Scheduler

  • A load dispatcher is the only access point to the entire cluster system, using external VIP (Virtual IP, Virtual IP) addresses common to all servers, also known as cluster IP addresses
  • Hot backups are usually configured with primary and standby dispatchers, which are smoothly replaced with standby dispatchers when they fail to ensure high availability

2.2 Server Pool

  • The application services provided by the cluster (such as HTTP, FTP) are assumed by the server pool, where each node has a separate RIP (Real IP, Real IP) address and handles only client requests distributed by the scheduler.
  • When a node temporarily fails, the load dispatcher's fault-tolerant mechanism isolates it and waits for the error to be eliminated before it is reintegrated into the server pool

2.3 Shared Storage

  • Shared storage provides a stable and consistent file access service to all nodes in the server pool, ensuring the uniformity of the entire cluster
  • In Linux/UNIX environments, you can use NAS devices or dedicated servers that provide NFS shared services

3. Work mode of load balancing

  • Cluster load balancing scheduling technology, which can be distributed based on IP, ports, content, etc., among which IP-based load scheduling is the most efficient
  • In IP-based load balancing mode, there are three common working modes: address translation, IP tunneling and direct routing.

3.1 Address Translation

  • NAT mode, a private network structure similar to a firewall
  • The load dispatcher acts as the gateway to the server node, acting as both the access entry point for the client and the access exit for each node to respond to the client
  • Server nodes use private IP addresses and are on the same physical network as the load dispatcher, with better security than the other two methods

3.2 IP Tunnel

  • TUN mode for short, with an open network structure

  • A load dispatcher acts as an access gateway for clients only, and each node responds directly to the clients through its own Internet connection instead of passing through the load dispatcher

  • Server nodes are scattered across different locations in the Internet, have separate public network IP addresses, and communicate with each other through dedicated IP tunnels and load dispatchers

3.3 Direct Routing

  • DR mode, a semi-open network structure, is similar to TUN mode in structure, but the nodes are not scattered around, but are in the same physical network as the scheduler
  • Load dispatcher connects to each node server over a local network without establishing a dedicated IP tunnel

Among the three working modes mentioned above, NAT only needs one public IP address, which makes it the easiest to use load balancing mode with better security. Many hardware load balancing devices use this mode.
Compared with DR and TUNs, DR and TUNs are more powerful and applicable, but the security of nodes is slightly worse.

2. LVS Virtual Server

1. Overview of LVS

  • Linux Virtual Server (LVS) is a load balancing project developed for the Linux kernel, which was created by Dr. Zhang Wensong of China in May 1998.

  • LVS is essentially equivalent to IP address-based virtualization, providing an efficient solution for load balancing based on IP address and content request distribution.

  • Manually Load ip_vs module, and view the version information of the module

[root@ng ~]#modprobe ip_vs                               #Loading modules
[root@ng ~]#cat /proc/net/ip_vs						    #View version information
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn

2. Load Scheduling Algorithm for LVS

For different network services and configuration needs, the LVS scheduler provides a variety of different load scheduling algorithms, of which the four most commonly used are polling, weighted polling, least connection and weighted least connection.

  • Polling: Allocate incoming access requests in turn to nodes in the cluster (real servers) and treat each server equally, regardless of the actual number of connections and system load on the server
  • Weighted Polling: Allocates incoming access requests in turn based on the processing power of the real server, allowing the dispatcher to automatically query each node's load and dynamically adjust its weights to ensure that the more capable server can handle more traffic
  • Minimum Connections: Allocate access requests received based on the number of connections established by the real server to the node with the least number of connections. This way you can better balance load if all server nodes have similar performance
  • Weighted Minimum Connections: When there is a large performance difference between server nodes, the weights can be automatically adjusted for the real server, and nodes with higher weights will bear a larger proportion of the active connection load

3. ipvsadm management tools

ipvsadm is a LVS cluster management tool used on a load dispatcher by calling ip_vs module to add, remove server nodes, and view the running state of the cluster

  • ipvsadm package needs to be installed manually
[root@ng ~]#yum install -y ipvsadm
already installed:
  ipvsadm.x86_64 0:1.27-8.el7 

[root@ng ~]#ipvsadm -v
ipvsadm v1.27 2008/5/15 (compiled with popt and IPVS v1.2.1)

3.1 ipvsadm Common Options

The management of LVS clusters mainly includes creating virtual servers, adding server nodes, viewing the status of cluster nodes, deleting server nodes, and saving load distribution policies.

-AAdd Virtual Server
-DDelete entire virtual server
-sSpecify the load scheduling algorithm (polling: rr, weighted polling: wrr, least connections: lc, weighted least connections: wlc)
-aRepresents adding a real server (node server)
-dDelete a node
-tSpecify VIP address and TCP port
-rSpecify RIP address and TCP port
-mIndicates the use of NAT cluster mode.
-gRepresents using DR mode
-iRepresents using TUN mode
-wSet weights (0 means pause node)
-p 60Represents a 60-second long connection
-lList View LVS Virtual Server (default is to view all)
-nDisplays information such as addresses, ports, etc. in digital form, often in combination with the'-l'option

3.2 Example

  • Create Virtual Server

The cluster's VIP address is, which provides load shunting services for TCP80 ports, using a scheduling algorithm for polling, as follows

ipvsadm -A -t -s rr
#-s is used to specify the algorithm, rr means polling
#For a load balancing scheduler, the VIP must be an IP address that is actually enabled locally
  • Add Server Node

Add four server nodes for virtual server, with IP addresses to, as follows

ipvsadm -a -t -r -m -w 1
ipvsadm -a -t -r -m -w 1
ipvsadm -a -t -r -m -w 1
ipvsadm -a -t -r -m -w 1
  • View Cluster Node Status

The combination option -l allows you to view the lvs virtual server, specifies that you only view one VIP address (default is to view all), and the combination-n displays address, port, and other information as numbers

ipvsadm -ln
  • Delete Server Node
ipvsadm -d -r -t

#If you want to delete the entire virtual server, use option-D and specify a virtual IP address instead of specifying a node
ipvsadm -D -t
  • Maintain Load Distribution Policy
ipvsadm-save > /etc/sysconfig/ipvsadm             #Maintenance Policy
cat /etc/sysconfig/ipvsadm						  #View Saved Results
service ipvsadm stop							  #Cleanup Policy
service ipvsadm start						      #Re-create Rules

3. Building LVS-NAT Load Balancing Cluster

1. Topology

In a NAT-mode cluster, the LVS load dispatcher is a gateway server where all nodes access the Internet and its external network address is also a VIP address for the entire cluster.
The LVS scheduler has two network cards that connect the internal and external networks, as shown in the topology diagram

For LVS load dispatchers, configure SNAT forwarding rules for outbound responses using iptables so that node servers can access the Internet.
The default gateway for all node servers is set to the intranet address of the LVS load dispatcher (

2. Case environment

Client win10:; Gateway:
LVS:ens33: NAT mode); ens37: (VM1 host mode only)
web1:; Gateway:
web2:; Gateway:

  • Local VM1 Network Settings

  • Client Network Settings

3. Configure Load Scheduler LVS

ntpdate aliyun.com                                            #Synchronize Ali Cloud Time
systemctl stop firewalld.service 						      #Close Firewall
systemctl disable firewalld.service 
setenforce 0
ifconfig ens37 						      #Temporarily add virtual network card (add network card manually first)
#Configure SNAT forwarding rules
[root@lvs ~]#vim /etc/sysctl.conf						      #Adjust/proc parameters
sysctl -p			 									      #Refresh effect
echo '1' > /proc/sys/net/ipv4/ip_forward

[root@lvs ~]#systemctl  start firewalld					      #Open Firewall
[root@lvs ~]#iptables -F		                              #Empty All Rules
[root@lvs ~]#iptables -t nat -F		                          #Empty the rules in the nat table
[root@lvs ~]# iptables -t nat -nL						      #See
# The traffic of http server maps segments to through LVS through ens37 interface
[root@lvs ~]#iptables -t nat -A POSTROUTING -s -o ens37 -j SNAT --to-source
#Loading LVS Kernel Modules
[root@lvs ~]#ll /proc/nat		 						      #Query has no ip_vs module
[root@lvs ~]#modprobe ip_vs	    						      #Manually Load ip_vs module
[root@lvs ~]#cat /proc/net/ip_vs                              #View ip_vs version information
#Install ipvsadm management tools
[root@lvs ~]#yum install -y ipvsadm

#Load distribution policies must be saved before starting the service, otherwise errors will occur
ipvsadm-save > /etc/sysconfig/ipvsadm		
ipvsadm --save > /etc/sysconfig/ipvsadm
touch /etc/sysconfig/ipvsadm

[root@lvs ~]#systemctl start ipvsadm			 		    	#Open management tools
#Configure Load Allocation Policy
ipvsadm -C     													#Clear Old Policy
ipvsadm -A -t -s rr		 					#Specifies that the shunt algorithm is rr polling
ipvsadm -a -t -r -m -w 1     #-m mode-w weight
ipvsadm -a -t -r -m -w 1
ipvsadm     													#Startup Policy
ipvsadm -ln    													#View node status, Masq represents NAT mode
ipvsadm-save > /etc/sysconfig/ipvsadm   						#Save Policy

#-A 	  Add Virtual Server
#-s 	  Specify the load scheduling algorithm (polling: rr, weighted polling: wrr, least connections: lc, weighted least connections: wlc)
#-a 	  Represents adding a real server (backend node server)
#-t 	  Specify VIP address and TCP port
#-m 	  Represents using NAT cluster mode
#-w 	  Set weights (0 means pause node)

4. Web Node Server Configuration

All node servers use the same configuration, including httpd service ports, content of site documents, and so on. Different pages are used in this case to test the effect of load balancing.

  • Web server 1 Gateway changed to
systemctl stop firewalld.service 
systemctl disable firewalld.service 
setenforce 0

yum install -y httpd			 							
systemctl start httpd.service 		 						
systemctl enable httpd.service	

[root@wb1]#cd /var/www/html/
[root@wb1]#vim index.html
this is test 1 web
[root@wb1]#systemctl restart httpd.service
    • Web server 2 gateway also changed to

systemctl stop firewalld.service 
systemctl disable firewalld.service 
setenforce 0

yum install -y httpd			 							
systemctl start httpd.service 		 						
systemctl enable httpd.service	

[root@wb2]#cd /var/www/html/
[root@wb2]#vim index.html
this is test 2 web
[root@wb2]#systemctl restart httpd.service

5. Testing LVS Clusters

Open win10 and use the browser to access, you will see the web page content provided by the real server. If the web page of each node is different, the web page that different clients see may be different (you can refresh the view effect multiple times)


  • Common cluster types include load balancing clusters, highly available clusters, and high-performance operational clusters
  • Different types of clusters can be merged when necessary, such as highly available load balanced clusters
  • The working modes of load balanced clusters: address translation (NAT) mode, IP tunneling (TUN) mode, and direct routing (DR) mode
  • Common scheduling algorithms for LVS load balancing clusters: polling (rr), weighted polling (wrr), least connections (lc), and weighted least connections (wlc)
  • NAT mode clusters use a single entry and exit, a public IP address; DR-mode clusters use a single entry and multiple exits, requiring multiple public network IP addresses

Tags: Operation & Maintenance Database Load Balance http memcached

Posted on Tue, 19 Oct 2021 12:21:32 -0400 by coowboy