Introduction
In a variety of Internet applications, as the site requires more and more hardware performance, response speed, service stability, data reliability, etc., it will be difficult for a single server to undertake all access. In addition to expensive mainframes and dedicated load-splitting devices, companies have another option to solve the problem by building cluster servers that provide the same services at the same address by integrating multiple relatively inexpensive common servers.
1. Overview of Clustering Technology
Clusters (clusters) represent a large collection of servers in the server realm to distinguish them from a single server.
1. Type
Regardless of the cluster, you need to include at least two node servers, which are represented as a whole and provide only one access entry (domain name or IP), equivalent to a large computer.
There are three types of clusters based on the target differences they target:
1.1 Load Balancing Cluster
- The goal is to improve the responsiveness of the application system, handle as many access requests as possible, and reduce latency so as to achieve high concurrency and high load (LB) overall performance
- For example, "DNS Polling", "Application Layer Switching", "Reverse Proxy" and so on can all act as load balancing clusters
- LB's load allocation relies on the shunting algorithm of the primary node, which allocates access requests from clients to multiple server nodes, thereby alleviating load pressure on the entire system
1.2 Highly Available Clusters
- To improve the reliability of the application system and minimize interruption time, ensure the continuity of service and achieve high availability (HA) fault tolerance
- For example, "failover", "dual-machine warm-up", "multiple-machine hot-up" and so on are highly available cluster technologies
- HA works in two modes, duplex and master-slave, where all nodes are online simultaneously. The master-slave node is only online, but it can automatically switch to the master node when a failure occurs
1.3 High Performance Operations Cluster
- To improve the CPU speed of the application system, expand hardware resources and analysis capabilities, and obtain high performance computing (HPC) capabilities equivalent to large supercomputers
- For example, Cloud Computing and Grid Computing can also be considered a high-performance operation
- High performance relies on "distributed computing", "parallel computing". By integrating CPU, memory and other resources from multiple servers with dedicated hardware and software, only large, supercomputers can achieve computing power.
2. Layered structure of load balancing
In a typical load balancing cluster, there are three levels of components, as shown in the following figure
- At least one load balancing dispatcher on the front end is responsible for responding to and distributing access requests from clients
- The backend consists of a pool of real servers that provide the actual application services, and the scalability of the entire cluster is accomplished by adding and removing server nodes, which are transparent to clients
- To maintain service consistency, all nodes use shared storage devices
2.1 Load Scheduler
- A load dispatcher is the only access point to the entire cluster system, using external VIP (Virtual IP, Virtual IP) addresses common to all servers, also known as cluster IP addresses
- Hot backups are usually configured with primary and standby dispatchers, which are smoothly replaced with standby dispatchers when they fail to ensure high availability
2.2 Server Pool
- The application services provided by the cluster (such as HTTP, FTP) are assumed by the server pool, where each node has a separate RIP (Real IP, Real IP) address and handles only client requests distributed by the scheduler.
- When a node temporarily fails, the load dispatcher's fault-tolerant mechanism isolates it and waits for the error to be eliminated before it is reintegrated into the server pool
2.3 Shared Storage
- Shared storage provides a stable and consistent file access service to all nodes in the server pool, ensuring the uniformity of the entire cluster
- In Linux/UNIX environments, you can use NAS devices or dedicated servers that provide NFS shared services
3. Work mode of load balancing
- Cluster load balancing scheduling technology, which can be distributed based on IP, ports, content, etc., among which IP-based load scheduling is the most efficient
- In IP-based load balancing mode, there are three common working modes: address translation, IP tunneling and direct routing.
3.1 Address Translation
- NAT mode, a private network structure similar to a firewall
- The load dispatcher acts as the gateway to the server node, acting as both the access entry point for the client and the access exit for each node to respond to the client
- Server nodes use private IP addresses and are on the same physical network as the load dispatcher, with better security than the other two methods
3.2 IP Tunnel
-
TUN mode for short, with an open network structure
-
A load dispatcher acts as an access gateway for clients only, and each node responds directly to the clients through its own Internet connection instead of passing through the load dispatcher
-
Server nodes are scattered across different locations in the Internet, have separate public network IP addresses, and communicate with each other through dedicated IP tunnels and load dispatchers
3.3 Direct Routing
- DR mode, a semi-open network structure, is similar to TUN mode in structure, but the nodes are not scattered around, but are in the same physical network as the scheduler
- Load dispatcher connects to each node server over a local network without establishing a dedicated IP tunnel
Summary:
Among the three working modes mentioned above, NAT only needs one public IP address, which makes it the easiest to use load balancing mode with better security. Many hardware load balancing devices use this mode.
Compared with DR and TUNs, DR and TUNs are more powerful and applicable, but the security of nodes is slightly worse.
2. LVS Virtual Server
1. Overview of LVS
-
Linux Virtual Server (LVS) is a load balancing project developed for the Linux kernel, which was created by Dr. Zhang Wensong of China in May 1998.
-
LVS is essentially equivalent to IP address-based virtualization, providing an efficient solution for load balancing based on IP address and content request distribution.
-
Manually Load ip_vs module, and view the version information of the module
[root@ng ~]#modprobe ip_vs #Loading modules [root@ng ~]#cat /proc/net/ip_vs #View version information IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn
2. Load Scheduling Algorithm for LVS
For different network services and configuration needs, the LVS scheduler provides a variety of different load scheduling algorithms, of which the four most commonly used are polling, weighted polling, least connection and weighted least connection.
- Polling: Allocate incoming access requests in turn to nodes in the cluster (real servers) and treat each server equally, regardless of the actual number of connections and system load on the server
- Weighted Polling: Allocates incoming access requests in turn based on the processing power of the real server, allowing the dispatcher to automatically query each node's load and dynamically adjust its weights to ensure that the more capable server can handle more traffic
- Minimum Connections: Allocate access requests received based on the number of connections established by the real server to the node with the least number of connections. This way you can better balance load if all server nodes have similar performance
- Weighted Minimum Connections: When there is a large performance difference between server nodes, the weights can be automatically adjusted for the real server, and nodes with higher weights will bear a larger proportion of the active connection load
3. ipvsadm management tools
ipvsadm is a LVS cluster management tool used on a load dispatcher by calling ip_vs module to add, remove server nodes, and view the running state of the cluster
- ipvsadm package needs to be installed manually
[root@ng ~]#yum install -y ipvsadm ...... already installed: ipvsadm.x86_64 0:1.27-8.el7 [root@ng ~]#ipvsadm -v ipvsadm v1.27 2008/5/15 (compiled with popt and IPVS v1.2.1)
3.1 ipvsadm Common Options
The management of LVS clusters mainly includes creating virtual servers, adding server nodes, viewing the status of cluster nodes, deleting server nodes, and saving load distribution policies.
| option | Explain |
|---|---|
| -A | Add Virtual Server |
| -D | Delete entire virtual server |
| -s | Specify the load scheduling algorithm (polling: rr, weighted polling: wrr, least connections: lc, weighted least connections: wlc) |
| -a | Represents adding a real server (node server) |
| -d | Delete a node |
| -t | Specify VIP address and TCP port |
| -r | Specify RIP address and TCP port |
| -m | Indicates the use of NAT cluster mode. |
| -g | Represents using DR mode |
| -i | Represents using TUN mode |
| -w | Set weights (0 means pause node) |
| -p 60 | Represents a 60-second long connection |
| -l | List View LVS Virtual Server (default is to view all) |
| -n | Displays information such as addresses, ports, etc. in digital form, often in combination with the'-l'option |
3.2 Example
- Create Virtual Server
The cluster's VIP address is 172.16.16.172, which provides load shunting services for TCP80 ports, using a scheduling algorithm for polling, as follows
ipvsadm -A -t 172.16.16.172:80 -s rr #-s is used to specify the algorithm, rr means polling #For a load balancing scheduler, the VIP must be an IP address that is actually enabled locally
- Add Server Node
Add four server nodes for virtual server 172.16.16.172, with IP addresses 192.168.8.135 to 192.168.8.138, as follows
ipvsadm -a -t 172.16.16.172:80 -r 192.168.8.135 -m -w 1 ipvsadm -a -t 172.16.16.172:80 -r 192.168.8.136 -m -w 1 ipvsadm -a -t 172.16.16.172:80 -r 192.168.8.137 -m -w 1 ipvsadm -a -t 172.16.16.172:80 -r 192.168.8.138 -m -w 1
- View Cluster Node Status
The combination option -l allows you to view the lvs virtual server, specifies that you only view one VIP address (default is to view all), and the combination-n displays address, port, and other information as numbers
ipvsadm -ln
- Delete Server Node
ipvsadm -d -r 192.168.8.135:80 -t 172.16.16.123:80 #If you want to delete the entire virtual server, use option-D and specify a virtual IP address instead of specifying a node ipvsadm -D -t 172.16.16.123:80
- Maintain Load Distribution Policy
ipvsadm-save > /etc/sysconfig/ipvsadm #Maintenance Policy cat /etc/sysconfig/ipvsadm #View Saved Results service ipvsadm stop #Cleanup Policy service ipvsadm start #Re-create Rules
3. Building LVS-NAT Load Balancing Cluster
1. Topology
In a NAT-mode cluster, the LVS load dispatcher is a gateway server where all nodes access the Internet and its external network address 192.168.100.100 is also a VIP address for the entire cluster.
The LVS scheduler has two network cards that connect the internal and external networks, as shown in the topology diagram
For LVS load dispatchers, configure SNAT forwarding rules for outbound responses using iptables so that node servers can access the Internet.
The default gateway for all node servers is set to the intranet address of the LVS load dispatcher (192.168.8.10)
2. Case environment
Client win10:192.168.100.10; Gateway: 192.168.100.100
LVS:ens33:192.168.8.10(VM8 NAT mode); ens37:192.168.100.100 (VM1 host mode only)
web1:192.168.8.11; Gateway: 192.168.8.10
web2:192.168.8.13; Gateway: 192.168.8.10
-
Local VM1 Network Settings
-
Client Network Settings
3. Configure Load Scheduler LVS
ntpdate aliyun.com #Synchronize Ali Cloud Time systemctl stop firewalld.service #Close Firewall systemctl disable firewalld.service setenforce 0 ifconfig ens37 192.168.100.100/24 #Temporarily add virtual network card (add network card manually first) ------------------------------------------------------------------------------------------------ #Configure SNAT forwarding rules [root@lvs ~]#vim /etc/sysctl.conf #Adjust/proc parameters net.ipv4.ip_forward=1 sysctl -p #Refresh effect or echo '1' > /proc/sys/net/ipv4/ip_forward [root@lvs ~]#systemctl start firewalld #Open Firewall [root@lvs ~]#iptables -F #Empty All Rules [root@lvs ~]#iptables -t nat -F #Empty the rules in the nat table [root@lvs ~]# iptables -t nat -nL #See # The traffic of http server maps 192.168.8.0 segments to 192.168.100.100 through LVS through ens37 interface [root@lvs ~]#iptables -t nat -A POSTROUTING -s 192.168.8.0/24 -o ens37 -j SNAT --to-source 192.168.100.100 ------------------------------------------------------------------------------------------------- #Loading LVS Kernel Modules [root@lvs ~]#ll /proc/nat #Query has no ip_vs module [root@lvs ~]#modprobe ip_vs #Manually Load ip_vs module [root@lvs ~]#cat /proc/net/ip_vs #View ip_vs version information ------------------------------------------------------------------------------------------------- #Install ipvsadm management tools [root@lvs ~]#yum install -y ipvsadm #Load distribution policies must be saved before starting the service, otherwise errors will occur ipvsadm-save > /etc/sysconfig/ipvsadm or ipvsadm --save > /etc/sysconfig/ipvsadm or touch /etc/sysconfig/ipvsadm [root@lvs ~]#systemctl start ipvsadm #Open management tools ------------------------------------------------------------------------------------------------- #Configure Load Allocation Policy ipvsadm -C #Clear Old Policy ipvsadm -A -t 192.168.100.100:80 -s rr #Specifies that the shunt algorithm is rr polling ipvsadm -a -t 192.168.100.100:80 -r 192.168.8.11:80 -m -w 1 #-m mode-w weight ipvsadm -a -t 192.168.100.100:80 -r 192.168.8.13:80 -m -w 1 ipvsadm #Startup Policy ipvsadm -ln #View node status, Masq represents NAT mode ipvsadm-save > /etc/sysconfig/ipvsadm #Save Policy #-A Add Virtual Server #-s Specify the load scheduling algorithm (polling: rr, weighted polling: wrr, least connections: lc, weighted least connections: wlc) #-a Represents adding a real server (backend node server) #-t Specify VIP address and TCP port #-m Represents using NAT cluster mode #-w Set weights (0 means pause node)
4. Web Node Server Configuration
All node servers use the same configuration, including httpd service ports, content of site documents, and so on. Different pages are used in this case to test the effect of load balancing.
- Web server 1 Gateway changed to 192.168.8.10
systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 yum install -y httpd systemctl start httpd.service systemctl enable httpd.service [root@wb1]#cd /var/www/html/ [root@wb1]#vim index.html this is test 1 web [root@wb1]#systemctl restart httpd.service
-
- Web server 2 gateway also changed to 192.168.8.10
systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 yum install -y httpd systemctl start httpd.service systemctl enable httpd.service [root@wb2]#cd /var/www/html/ [root@wb2]#vim index.html this is test 2 web [root@wb2]#systemctl restart httpd.service
5. Testing LVS Clusters
Open win10 and use the browser to access 192.168.100.100, you will see the web page content provided by the real server. If the web page of each node is different, the web page that different clients see may be different (you can refresh the view effect multiple times)
summary
- Common cluster types include load balancing clusters, highly available clusters, and high-performance operational clusters
- Different types of clusters can be merged when necessary, such as highly available load balanced clusters
- The working modes of load balanced clusters: address translation (NAT) mode, IP tunneling (TUN) mode, and direct routing (DR) mode
- Common scheduling algorithms for LVS load balancing clusters: polling (rr), weighted polling (wrr), least connections (lc), and weighted least connections (wlc)
- NAT mode clusters use a single entry and exit, a public IP address; DR-mode clusters use a single entry and multiple exits, requiring multiple public network IP addresses