Cluster Load Balancing Setup--LVS-DR Cluster Deployment

Article Directory

Preface
I: How LVS-DR works

1.1: Overview of DR mode
1.2: How to analyze LVS-DR packet flow direction?
3.3: ARP Problems in LVS-DR
3.4: Solutions to the above two ARP problems

II: LVS-DR deployment

2.1: Experimental Environment
2.2: Experimental Topology
2.3: Experimental purposes
2.4: Experimental Processes

2.4.1: Scheduler configuration
2.4.2: Configure two node IP address binding VIP
2.4.3: Node server configuration suppresses ARP response
2.4.4: Two Node Servers Configuration Test Web Address

2.5: Experimental Validation

Preface I: How LVS-DR works

1.1: Overview of DR mode

Load Balancing Cluster Working Mode - Direct Routing
- DR mode, a semi-open network structure, is similar to TUN mode in structure, but the nodes are not scattered around, but are in the same physical network as the scheduler
- Load dispatcher connects to each node server over a local network without establishing a dedicated IP tunnel

1.2: How to analyze LVS-DR packet flow direction?

For the convenience of principle analysis, the Client is placed in the same network as the cluster machine, and the data packets flow through a path of 1-2-3-4
- 1.Client makes a request to the target VIP and Director (Load Balancer) receives it.In this case, the IP header and data frame header information are:
- 2. The Director chooses Realserver_1 according to the load balancing algorithm, does not modify or encapsulate the IP message, but changes the MAC address of the data frame to that of RealServer_1, and sends it on the local area network.The IP header and data frame header information are as follows:
- - 3.RealServer_1 receives this frame, unpacks it and finds that the destination IP matches the local machine (RealServer has a VIP pre-bound), and processes the message.The message is then re-encapsulated and sent to the local area network.At this time, the IP header and data frame header information are:.
  - 4.Client will receive a response.Client thinks it's serviced properly and doesn't know which server is handling it
    
    Note: If across network segments, the message is returned to the user via terne via the router
3.3: ARP Problems in LVS-DR
- In LVS-DR load balancing clusters, both load balancers and node servers are configured with the same VIP address
- Having the same IP address in the LAN is bound to cause the disturbance of ARP communication among servers.
  - When an ARP broadcast is sent to the LVS-DR cluster, both load balancers and node servers receive ARP broadcasts because they are connected to the same network
  - Only the front-end load balancer responds at this time, other node servers should not respond to ARP broadcasts
- Processing node servers so that they do not respond to ARP requests for VIP s
  - Host VI P address with virtual interface lo:0
  - Setting the kernel parameter arp_ignore=1: The system only responds to ARP requests whose destination IP is local IP
- RealServe return message (source IP is VIP) is forwarded by the router and the MAC address of the router needs to be obtained before the message is re-encapsulated
- In LVS-DR load balancing clusters, both load balancers and node servers are configured with the same VIP address
- Having the same IP address in the LAN is bound to cause the disturbance of ARP communication among servers.
  - When an ARP broadcast is sent to the LVS-DR cluster, both load balancers and node servers receive ARP broadcasts because they are connected to the same network
  - Only the front-end load balancer responds at this time, other node servers should not respond to ARP broadcasts
- Processing node servers so that they do not respond to ARP requests for VIP s
  - Host VI P address with virtual interface lo:0
  - Setting the kernel parameter arp_ignore=1: The system only responds to ARP requests whose destination IP is local IP
- RealServe return message (source IP is VIP) is forwarded by the router and the MAC address of the router needs to be obtained before the message is re-encapsulated
- When sending ARP requests, Linux defaults to using the source IP address (VIP) of the IP package as the source IP address in the ARP request package instead of the IP address of the sending interface (such as ens33)
- When the router receives an ARP request, it updates the ARP table entries
- The original VIP corresponding Director's MAC address will be updated to VI P corresponding RealServer's MAC address
- At this point, the new request message, which the router forwards to RealServer according to the ARP table entries, causes the Director's VIP to fail
  - Solution
    
    Processing the node server, setting the kernel parameter arp_announce=2: The system does not use the source address of the IP package to set the source address of the ARP request, but chooses the IP address of the sending interface
3.4: Solutions to the above two ARP problems
- Modify/etc/sysctl.conf file
  - Processing node servers so that they do not respond to ARP requests for VIP s
  - net.ipv4.conf.lo.arp_ignore = 1
  - net.ipv4.conf.lo.arp_announce = 2
  - Instead of using the source address of the IP package to set the source address of the ARP request, the system chooses the IP address of the sending interface
  - net.ipv4.conf.all.arp_ignore = 1
  - net.ipv4.conf.all.arp_announce = 2

II: LVS-DR deployment

2.1: Experimental Environment

In order to further improve the load capacity of the company's website, the company decided to expand the existing website platform and build a load balancing cluster based on LVS.For cluster access efficiency, administrators are prepared to use the DR mode of the LVS cluster to store shared storage devices on their internal proprietary network
LVS1: 192.168.100.100
LVS2: 192.168.100.110
Web1: 192.168.100.200
Web2: 192.168.100.222
VIP=192.168.100.10
Win 10 : 192.168.100.50

2.2: Experimental Topology

2.3: Experimental purposes

win10 clients can successfully access web node server content by accessing lvs scheduler drift IP address

2.4: Experimental Processes

2.4.1: Scheduler configuration

Toolkits are installed on both LVS servers

[root@lvs1 ~]# yum install keepalived ipvsadm -y '//keepalived: dual hot standby, ipvsadm: dispatch management to use' [root@lvs2 ~]# yum install keepalived ipvsadm -y
Routing forwarding is turned on and redirection is turned off on both LVS servers

[root@lvs1 ~]# vim /etc/sysctl.conf '//End row insert next segment configuration' net.ipv4.ip_forward = 1 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.ens33.send_redirects = 0 [root@lvs1 ~]# sysctl -p '//Overload Configuration for Effectiveness' net.ipv4.ip_forward = 1 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.ens33.send_redirects = 0
Both LVS servers create scripts to facilitate service management

[root@lvs1 network-scripts]# vim /etc/init.d/dr.sh '//Edit the following script' #!/bin/bash GW=192.168.100.1 VIP=192.168.100.10 ##virtual ip RIP1=192.168.100.200 ##Real web server ip RIP2=192.168.100.222 case "$1" in start) /sbin/ipvsadm --save > /etc/sysconfig/ipvsadm ##Save Configuration systemctl start ipvsadm ##Start Services /sbin/ifconfig ens33:0 $VIP broadcast $VIP netmask 255.255.255.255 broadcast $VIP up ##Set up ens33:0 Address, Broadcast Address, Subnet Mask, and Open /sbin/route add -host $VIP dev ens33:0 ##Add Routing Segment Information /sbin/ipvsadm -A -t $VIP:80 -s rr ##Specify virtual service access entry, specify polling algorithm /sbin/ipvsadm -a -t $VIP:80 -r $RIP1:80 -g ##Specify real server, dr mode /sbin/ipvsadm -a -t $VIP:80 -r $RIP2:80 -g echo "ipvsadm starting --------------------[ok]" ;; stop) /sbin/ipvsadm -C ##wipe cache systemctl stop ipvsadm ##Shut down services ifconfig ens33:0 down ##Close Interface route del $VIP ##Delete Routing Information echo "ipvsamd stoped----------------------[ok]" ;; status) if [ ! -e /var/lock/subsys/ipvsadm ];then ##Determining whether a file exists or not determines its status echo "ipvsadm stoped---------------" exit 1 else echo "ipvsamd Runing ---------[ok]" fi ;; *) echo "Usage: $0 {start|stop|status}" exit 1 esac exit 0 [root@lvs1 network-scripts]# chmod +x /etc/init.d/dr.sh
Both LVS servers modified the network card to host-only mode
Both LVS servers configure IP address for ens33 and VI P address for ens33:0

[root@lvs1 ~]# cd /etc/sysconfig/network-scripts/ [root@lvs1 network-scripts]# cp -p ifcfg-ens33 ifcfg-ens33:0 [root@lvs1 network-scripts]# vim ifcfg-ens33:0 '//Configure virtual ip address' '//Delete the original content and add the following configuration' DEVICE=ens33:0 ONBOOT=yes IPADDR=192.168.100.10 NETMASK=255.255.255.0 [root@lvs1 network-scripts]# vim ifcfg-ens33 '//...omit and change to static' BOOTPROTO=static '//...omit content, add content at the end of the line' IPADDR=192.168.100.100 NETMASK=255.255.255.0 GATEWAY=192.168.100.1 [root@lvs2 network-scripts]# vim ifcfg-ens33 '//Note that the IP address of lvs2 is 192.168.100.110' IPADDR=192.168.100.110
Both LVS servers restart various configurations

[root@lvs1 network-scripts]# systemctl restart network [root@lvs1 network-scripts]# ifup ens33:0 [root@lvs1 network-scripts]# service dr.sh start ipvsadm starting --------------------[ok] [root@lvs1 network-scripts]# systemctl stop firewalld.service [root@lvs1 network-scripts]# setenforce 0

2.4.2: Configure two node IP address binding VIP

In LVS-DR, two node servers are configured to bind VIP addresses
VIP addresses are used only as source addresses for Web response packets and do not listen for client access requests
All modified to host-only mode
Two node servers configure IP address for ens33 and VIP address for lo:0

[root@web1 ~]# cd /etc/sysconfig/network-scripts/ [root@web1 network-scripts]# cp -p ifcfg-lo ifcfg-lo:0 [root@web1 network-scripts]# vim ifcfg-lo:0 DEVICE=lo:0 IPADDR=192.168.100.10 NETMASK=255.255.255.0 ONBOOT=yes [root@web1 network-scripts]# vim ifcfg-ens33 '//...omit and change to static' BOOTPROTO=static '//...omit content, add content at the end of the line' IPADDR=192.168.100.200 NETMASK=255.255.255.0 GATEWAY=192.168.100.1 '//The web2 server was modified to 192.168.100.222' IPADDR=192.168.100.222

2.4.3: Node server configuration suppresses ARP response

Configuration Suppression ARP Script [root@web1 network-scripts]# vim /etc/init.d/web.sh #!/bin/bash VIP=192.168.100.10 case "$1" in start) ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP /sbin/route add -host $VIP dev lo:0 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore '//arp ignore' echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce sysctl -p >/dev/null 2>&1 echo "RealServer Start OK " ;; stop) ifconfig lo:0 down route del $VIP /dev/null 2>&1 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore '//arp on' echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce echo "RealServer Stopd" ;; *) echo "Usage: $0 {start|stop}" exit 1 esac exit 0 [root@web1 network-scripts]# chmod +x /etc/init.d/web.sh

2.4.4: Two Node Servers Configuration Test Web Address

Install the configuration and enable the httpd server on both node servers to create test pages with different content to distinguish the different node servers

[root@web1 ~]# yum install httpd -y [root@web2 ~]# yum install httpd -y [root@web1 ~]# vim /var/www/html/index.html <h1>this is web1 web</h1> [root@web2 ~]# vim /var/www/html/index.html <h1>this is web2 web</h1>
Restart services

[root@web1 network-scripts]# systemctl restart network [root@web1 network-scripts]# ifup lo:0 [root@web1 network-scripts]# service web.sh start RealServer Start OK [root@web1 network-scripts]# systemctl start httpd [root@web1 network-scripts]# systemctl stop firewalld.service [root@web1 network-scripts]# setenforce 0