Cluster Load Balancing Setup--LVS-DR Cluster Deployment

Article Directory

Preface

I: How LVS-DR works

1.1: Overview of DR mode

  • Load Balancing Cluster Working Mode - Direct Routing
    • DR mode, a semi-open network structure, is similar to TUN mode in structure, but the nodes are not scattered around, but are in the same physical network as the scheduler
    • Load dispatcher connects to each node server over a local network without establishing a dedicated IP tunnel

1.2: How to analyze LVS-DR packet flow direction?

  • For the convenience of principle analysis, the Client is placed in the same network as the cluster machine, and the data packets flow through a path of 1-2-3-4

    • 1.Client makes a request to the target VIP and Director (Load Balancer) receives it.In this case, the IP header and data frame header information are:

    • 2. The Director chooses Realserver_1 according to the load balancing algorithm, does not modify or encapsulate the IP message, but changes the MAC address of the data frame to that of RealServer_1, and sends it on the local area network.The IP header and data frame header information are as follows:

      • 3.RealServer_1 receives this frame, unpacks it and finds that the destination IP matches the local machine (RealServer has a VIP pre-bound), and processes the message.The message is then re-encapsulated and sent to the local area network.At this time, the IP header and data frame header information are:.

      • 4.Client will receive a response.Client thinks it's serviced properly and doesn't know which server is handling it

        Note: If across network segments, the message is returned to the user via terne via the router

    3.3: ARP Problems in LVS-DR

    • In LVS-DR load balancing clusters, both load balancers and node servers are configured with the same VIP address

    • Having the same IP address in the LAN is bound to cause the disturbance of ARP communication among servers.

      • When an ARP broadcast is sent to the LVS-DR cluster, both load balancers and node servers receive ARP broadcasts because they are connected to the same network
      • Only the front-end load balancer responds at this time, other node servers should not respond to ARP broadcasts
    • Processing node servers so that they do not respond to ARP requests for VIP s

      • Host VI P address with virtual interface lo:0
      • Setting the kernel parameter arp_ignore=1: The system only responds to ARP requests whose destination IP is local IP
    • RealServe return message (source IP is VIP) is forwarded by the router and the MAC address of the router needs to be obtained before the message is re-encapsulated

    • In LVS-DR load balancing clusters, both load balancers and node servers are configured with the same VIP address

    • Having the same IP address in the LAN is bound to cause the disturbance of ARP communication among servers.

      • When an ARP broadcast is sent to the LVS-DR cluster, both load balancers and node servers receive ARP broadcasts because they are connected to the same network
      • Only the front-end load balancer responds at this time, other node servers should not respond to ARP broadcasts
    • Processing node servers so that they do not respond to ARP requests for VIP s

      • Host VI P address with virtual interface lo:0
      • Setting the kernel parameter arp_ignore=1: The system only responds to ARP requests whose destination IP is local IP
    • RealServe return message (source IP is VIP) is forwarded by the router and the MAC address of the router needs to be obtained before the message is re-encapsulated

    • When sending ARP requests, Linux defaults to using the source IP address (VIP) of the IP package as the source IP address in the ARP request package instead of the IP address of the sending interface (such as ens33)

    • When the router receives an ARP request, it updates the ARP table entries

    • The original VIP corresponding Director's MAC address will be updated to VI P corresponding RealServer's MAC address

    • At this point, the new request message, which the router forwards to RealServer according to the ARP table entries, causes the Director's VIP to fail

      • Solution

        Processing the node server, setting the kernel parameter arp_announce=2: The system does not use the source address of the IP package to set the source address of the ARP request, but chooses the IP address of the sending interface

    3.4: Solutions to the above two ARP problems

    • Modify/etc/sysctl.conf file
      • Processing node servers so that they do not respond to ARP requests for VIP s
      • net.ipv4.conf.lo.arp_ignore = 1
      • net.ipv4.conf.lo.arp_announce = 2
      • Instead of using the source address of the IP package to set the source address of the ARP request, the system chooses the IP address of the sending interface
      • net.ipv4.conf.all.arp_ignore = 1
      • net.ipv4.conf.all.arp_announce = 2

II: LVS-DR deployment

2.1: Experimental Environment

  • In order to further improve the load capacity of the company's website, the company decided to expand the existing website platform and build a load balancing cluster based on LVS.For cluster access efficiency, administrators are prepared to use the DR mode of the LVS cluster to store shared storage devices on their internal proprietary network
  • LVS1: 192.168.100.100
  • LVS2: 192.168.100.110
  • Web1: 192.168.100.200
  • Web2: 192.168.100.222
  • VIP=192.168.100.10
  • Win 10 : 192.168.100.50

2.2: Experimental Topology

2.3: Experimental purposes

  • win10 clients can successfully access web node server content by accessing lvs scheduler drift IP address

2.4: Experimental Processes

2.4.1: Scheduler configuration
  • Toolkits are installed on both LVS servers

    [root@lvs1 ~]# yum install keepalived ipvsadm -y	'//keepalived: dual hot standby, ipvsadm: dispatch management to use'
    [root@lvs2 ~]# yum install keepalived ipvsadm -y
    
  • Routing forwarding is turned on and redirection is turned off on both LVS servers

    [root@lvs1 ~]# vim /etc/sysctl.conf 
    '//End row insert next segment configuration'
    net.ipv4.ip_forward = 1
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.default.send_redirects = 0
    net.ipv4.conf.ens33.send_redirects = 0
    [root@lvs1 ~]# sysctl -p	'//Overload Configuration for Effectiveness'
    net.ipv4.ip_forward = 1
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.default.send_redirects = 0
    net.ipv4.conf.ens33.send_redirects = 0
    
    
  • Both LVS servers create scripts to facilitate service management

    [root@lvs1 network-scripts]# vim /etc/init.d/dr.sh
    '//Edit the following script'
    #!/bin/bash
    GW=192.168.100.1
    VIP=192.168.100.10   ##virtual ip
    RIP1=192.168.100.200 ##Real web server ip
    RIP2=192.168.100.222
    case "$1" in
    start)
                    /sbin/ipvsadm --save > /etc/sysconfig/ipvsadm  ##Save Configuration
                    systemctl start ipvsadm  ##Start Services
                    /sbin/ifconfig ens33:0 $VIP broadcast $VIP netmask 255.255.255.255 broadcast $VIP up  
    ##Set up ens33:0 Address, Broadcast Address, Subnet Mask, and Open
                    /sbin/route add -host $VIP dev ens33:0  ##Add Routing Segment Information
                    /sbin/ipvsadm -A -t $VIP:80 -s rr ##Specify virtual service access entry, specify polling algorithm
                    /sbin/ipvsadm -a -t $VIP:80 -r $RIP1:80 -g ##Specify real server, dr mode
                    /sbin/ipvsadm -a -t $VIP:80 -r $RIP2:80 -g
                    echo "ipvsadm starting --------------------[ok]"
                    ;;
    stop)
                    /sbin/ipvsadm -C   ##wipe cache
                    systemctl stop ipvsadm  ##Shut down services
                    ifconfig ens33:0 down  ##Close Interface
                    route del $VIP   ##Delete Routing Information
                    echo "ipvsamd stoped----------------------[ok]"
                     ;;
    status)
                    if [ ! -e /var/lock/subsys/ipvsadm ];then  ##Determining whether a file exists or not determines its status
                    echo "ipvsadm stoped---------------"
                    exit 1
                                    else
                                    echo "ipvsamd Runing ---------[ok]"
                    fi
                    ;;
    *)
                    echo "Usage: $0 {start|stop|status}"
                    exit 1
    esac
    exit 0
    [root@lvs1 network-scripts]# chmod +x /etc/init.d/dr.sh 
    
    
  • Both LVS servers modified the network card to host-only mode

  • Both LVS servers configure IP address for ens33 and VI P address for ens33:0

    [root@lvs1 ~]# cd /etc/sysconfig/network-scripts/
    [root@lvs1 network-scripts]# cp -p ifcfg-ens33 ifcfg-ens33:0	
    [root@lvs1 network-scripts]# vim ifcfg-ens33:0	'//Configure virtual ip address'
    '//Delete the original content and add the following configuration'
    DEVICE=ens33:0
    ONBOOT=yes
    IPADDR=192.168.100.10
    NETMASK=255.255.255.0
    [root@lvs1 network-scripts]# vim ifcfg-ens33
    '//...omit and change to static'
    BOOTPROTO=static
    '//...omit content, add content at the end of the line'
    IPADDR=192.168.100.100
    NETMASK=255.255.255.0
    GATEWAY=192.168.100.1
    [root@lvs2 network-scripts]# vim ifcfg-ens33
        '//Note that the IP address of lvs2 is 192.168.100.110'
    IPADDR=192.168.100.110
    
  • Both LVS servers restart various configurations

    [root@lvs1 network-scripts]# systemctl restart network
    [root@lvs1 network-scripts]# ifup ens33:0
    [root@lvs1 network-scripts]# service dr.sh start
    ipvsadm starting --------------------[ok]
    [root@lvs1 network-scripts]# systemctl stop firewalld.service 
    [root@lvs1 network-scripts]# setenforce 0
    
2.4.2: Configure two node IP address binding VIP
  • In LVS-DR, two node servers are configured to bind VIP addresses

  • VIP addresses are used only as source addresses for Web response packets and do not listen for client access requests

  • All modified to host-only mode

  • Two node servers configure IP address for ens33 and VIP address for lo:0

    [root@web1 ~]# cd /etc/sysconfig/network-scripts/
    [root@web1 network-scripts]# cp -p ifcfg-lo ifcfg-lo:0
    [root@web1 network-scripts]# vim ifcfg-lo:0
    DEVICE=lo:0
    IPADDR=192.168.100.10
    NETMASK=255.255.255.0
    ONBOOT=yes
    [root@web1 network-scripts]# vim ifcfg-ens33
    '//...omit and change to static'
    BOOTPROTO=static
    '//...omit content, add content at the end of the line'
    IPADDR=192.168.100.200
    NETMASK=255.255.255.0
    GATEWAY=192.168.100.1
    '//The web2 server was modified to 192.168.100.222'
    IPADDR=192.168.100.222
    
2.4.3: Node server configuration suppresses ARP response
  • Configuration Suppression ARP Script
    [root@web1 network-scripts]# vim /etc/init.d/web.sh
    #!/bin/bash
    VIP=192.168.100.10
                    case "$1" in
                    start)
                                    ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP
                                    /sbin/route add -host $VIP dev lo:0
                                    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore   '//arp ignore'
                                    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
                                    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
                                    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
                                    sysctl -p >/dev/null 2>&1
                                    echo "RealServer Start OK "
                                    ;;
                    stop)
                                    ifconfig lo:0 down
                                    route del $VIP /dev/null 2>&1
                                    echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore '//arp on'
                                    echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
                                    echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
                                    echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
                                    echo "RealServer Stopd"
                                    ;;
                    *)
                                    echo "Usage: $0 {start|stop}"
                                    exit 1
                    esac
                    exit 0
    [root@web1 network-scripts]# chmod +x /etc/init.d/web.sh 
    
2.4.4: Two Node Servers Configuration Test Web Address
  • Install the configuration and enable the httpd server on both node servers to create test pages with different content to distinguish the different node servers

    [root@web1 ~]# yum install httpd -y
    [root@web2 ~]# yum install httpd -y
    [root@web1 ~]# vim /var/www/html/index.html
    <h1>this is web1 web</h1>
    [root@web2 ~]# vim /var/www/html/index.html
    <h1>this is web2 web</h1>
    
  • Restart services

    [root@web1 network-scripts]# systemctl restart network
    [root@web1 network-scripts]# ifup lo:0
    [root@web1 network-scripts]# service web.sh start
    RealServer Start OK 
    [root@web1 network-scripts]# systemctl start httpd
    [root@web1 network-scripts]# systemctl stop firewalld.service 
    [root@web1 network-scripts]# setenforce 0
       
    

2.5: Experimental Validation

  • Modify win10 host IP address

  • Access 192.168.100.10 authentication using win10

  • Access Successful

Published 100 original articles, received 60 reviews, and received 10,000 visits+
Private letter follow

Tags: network vim Mac yum

Posted on Wed, 15 Jan 2020 22:41:04 -0500 by garydt