Three modes of LVS

Device address naming of LVS Cluster

CIP: Client IP, Client IP address
VIP: Virtual IP, LVS IP address for user requests
DIP: Director IP, the IP address where LVS users communicate with back-end servers
RIP: Real server IP, the IP address used by the back-end server to communicate with LVS

NAT network address translation

(Network address translation)

LVS NAT is essentially a multi-objective DNAT (iptables)

LVS NAT mode forwards the request message by modifying the target IP address (and possibly the target port) to the selected RIP address of an RS

LVS-NAT model is similar to DNAT, and its working mechanism is the same as DNAT. When the client requests cluster services, LVS modifies the target address of the request message to RIP, forwards it to the backend RealServer, and modifies the source address of the backend response message to VIP to respond to the client.

Features of LVS-NAT:

RS and DIP should use private network address, and RS gateway should point to DIP
Both request and response messages must be forwarded through the director, so the forwarding function of the director needs to be turned on during configuration. In the scenario of extremely high load, the director may become a system performance bottleneck
Support port mapping
RS can use any OS
RIP of RS and DIP of Director must be on the same IP network

Implementing httpd load balancing in NAT mode

name	network address
DS	DIP:192.168.101.200;VIP:192.168.101.100
RS(1)	DIP:192.168.101.110;VIP:192.168.101.2
RS(2)	DIP:192.168.101.210;VIP:192.168.101.2

Turn off firewall

[root@DR ~]# systemctl disable --now firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@DR ~]# setenforce 0

[root@RS1 ~]# systemctl disable --now firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@RS1 ~]# setenforce 0

[root@RS2 ~]# systemctl disable --now firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@RS2 ~]# setenforce 0

Configure temporary IP for DR

[root@DR ~]#  ifconfig ens33:0 192.168.101.100/24 up
[root@DR ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:27:fa:31 brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.200/24 brd 192.168.101.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.101.100/24 brd 192.168.101.255 scope global secondary ens33:0
       valid_lft forever preferred_lft forever
    inet6 fe80::c4bc:49bf:b759:3272/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@DR ~]# tail /etc/sysconfig/network-scripts/ifcfg-ens33 
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=c5c9eb7d-47c3-4b19-aa9c-e7c78db2c28a
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.101.200
NETMASK=255.255.255.0
GATEWAY=192.168.101.2     # gateway
DNS1=114.114.114.114

Download httpd and modify the display

[root@RS1 ~]# yum -y install httpd
[root@RS1 ~]# systemctl enable --now httpd
[root@RS1 ~]# echo "hello 192.168.101.210" > /usr/share/httpd/noindex/index.html
[root@RS1 ~]# systemctl restart httpd
[root@RS2 ~]# yum -y install httpd
[root@RS2 ~]# echo "hello 192.168.101.110" > /usr/share/httpd/noindex/index.html
[root@RS2 ~]# systemctl restart httpd

Configure RS1 gateway as DR gateway

[root@RS1 ~]# tail /etc/sysconfig/network-scripts/ifcfg-ens33
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=ed299cc0-ef82-48a9-9f37-2a1639dae9f8
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.101.210
NETMASK=255.255.255.0
GATEWAY=192.168.101.2
DNS1=114.114.114.114

Configure RS2 gateway as DR gateway

[root@RS2 ~]# tail /etc/sysconfig/network-scripts/ifcfg-ens33
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=1d3ba2cd-8e05-4e5f-a390-3b21e137e1d9
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.101.110
NETMASK=255.255.255.0
GATEWAY=192.168.101.2
DNS1=114.114.114.114

Configure forwarding rules on DR

[root@DR ~]# yum -y install ipvsadm

[root@DR ~]# vim /etc/sysctl.conf 
[root@DR ~]# sysctl -p
net.ipv4.ip_forward = 1

[root@DR ~]# vim /etc/sysctl.conf 
[root@DR ~]# sysctl -p
net.ipv4.ip_forward = 1
[root@DR ~]# ipvsadm -A -t 192.168.101.100:80 -s rr
[root@DR ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.101.100:80 rr
[root@DR ~]# ipvsadm -a -t 192.168.101.100:80 -r 192.168.101.210:80 -m
[root@DR ~]# ipvsadm -a -t 192.168.101.100:80 -r 192.168.101.110:80 -m
[root@DR ~]# ipvsadm -Sn > /etc/sysconfig/ipvsadm
[root@DR ~]# cat /etc/sysconfig/ipvsadm
-A -t 192.168.101.100:80 -s rr
-a -t 192.168.101.100:80 -r 192.168.101.110:80 -m -w 1
-a -t 192.168.101.100:80 -r 192.168.101.210:80 -m -w 1

test

[root@DR ~]# curl http://192.168.101.100
hello 192.168.101.110
[root@DR ~]# curl http://192.168.101.100
hello 192.168.101.210

DR direct routing mode

(Direct routing) the default mode of lvs (using the same VIP)
LVS Dr mode forwards by modifying the target MAC address of the request message

Director: the scheduler needs to configure VIP and DIP

RSs: all real servers should be configured with RIP and VIP

Features of LVS-DR:

Ensure that the front-end router sends the request message with the target IP of VIP to the director. There are three solutions:

Static binding
arptables
Modify 2 parameters of RS host kernel:

arp_ Announcement: whether to receive and record other people's announcements and whether to announce your mac address to others.

arp_ Announcement has three optional values, 0, 1 and 2, which have the following meanings:

0 (default, the default value is 0): announce all the addresses you own

1: try not to announce the addresses that are not in the same network segment as yourself. For example, do not announce the addresses of 2.0 network segment to 1.0 network segment, and do not announce the addresses of 3.0 network segment to 1.0 network segment. However, try not to announce them, and they may still be announced

Try not to make arp response at the local address of the subnet of the network interface

This is useful when the source IP address of the ARP request is set to reach this network interface via routing

At this time, it will check whether the visiting IP is one of the IP in the subnet segment on all interfaces. If the visiting IP does not belong to the subnet segment on each network interface, it will be processed in Level 2

2: always do not announce the addresses that are not in the same network segment as yourself. For example, do not announce the addresses of 2.0 network segment to 1.0 network segment, and do not announce the addresses of 3.0 network segment to 1.0 network segment. Never announce them

Use the most appropriate local address for the query target. In this mode, the source address of this IP packet will be ignored and an attempt will be made to select a local address that can communicate with this address

The first is to select the local address containing the target IP address in the outgoing access subnet of all network interfaces

If no suitable address is found, the current sending network interface or other network interface that may receive the ARP response will be selected for transmission

arp_ignore: whether to respond to ARP requests

arp_ignore has 9 optional values, 0, 1, 2, 3, 4, 5, 6, 7 and 8, which have the following meanings:

0 (default, the default value is 0): respond to arp query requests for any local IP address on any network interface

1: From which interface does the request message enter and the requested target address is the address configured for this interface, it will respond. Otherwise, it will not respond, and only answer the ARP query request that the target IP address is the local address of the visiting network interface

2: Only answer the ARP query request that the target IP address is the local address of the visiting network interface, and the visiting IP must be within the subnet segment of the network interface

3: It does not respond to the arp request of the network interface, but only responds to the set unique and connection address

4-7: reserved unused

8: Do not respond to all arp queries (local addresses)

- The RIP of RS can use either private address or public address

-- RS and Director must be in the same physical network, and there must be no router between them

-- the request message is dispatched through the Director, but the response message must not be dispatched through the Director

-- port mapping is not supported

-- RS can be most OS

-- RS gateway cannot point to DIP

DR mode to achieve httpd load balancing

Experimental configuration information

name	network address
DS	DIP:192.168.101.200;VIP:192.168.101.100
RS(1)	DIP:192.168.101.110;VIP:192.168.101.100
RS(2)	DIP:192.168.101.210;VIP:192.168.101.100

Turn off firewall

[root@DR ~]# systemctl disable --now firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@DR ~]# setenforce 0

[root@RS1 ~]# systemctl disable --now firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@RS1 ~]# setenforce 0

[root@RS2 ~]# systemctl disable --now firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@RS2 ~]# setenforce 0

RS1 and RS2 download and install httpd modifications to provide access

[root@RS1 ~]# yum -y install httpd
[root@RS1 ~]# systemctl enable --now httpd
[root@RS1 ~]# echo "hello 192.168.101.210" > /usr/share/httpd/noindex/index.html
[root@RS1 ~]# systemctl restart httpd
[root@RS2 ~]# yum -y install httpd
[root@RS2 ~]# echo "hello 192.168.101.110" > /usr/share/httpd/noindex/index.html
[root@RS2 ~]# systemctl restart httpd

DR configuration VIP

[root@DR ~]#  ifconfig ens33:0 192.168.101.100/24 up
[root@DR ~]#  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:27:fa:31 brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.200/24 brd 192.168.101.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.101.100/24 brd 192.168.101.255 scope global secondary ens33:0
       valid_lft forever preferred_lft forever
    inet6 fe80::c4bc:49bf:b759:3272/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

RS configuration apr kernel parameters

[root@RS1 ~]# cat /etc/sysctl.conf   # Add two lines
net.ipv4.conf.all.arp_ignore = 1  # Set the corresponding network card to respond only to ARP requests with the target IP as its own interface address
net.ipv4.conf.all.arp_announce = 2 # Set the source IP of the ARP request to the IP on ens33, that is, RIP

[root@RS1 ~]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

[root@RS2 ~]# cat /etc/sysctl.conf   # Add two lines
net.ipv4.conf.all.arp_ignore = 1  # Set the corresponding network card to respond only to ARP requests with the target IP as its own interface address
net.ipv4.conf.all.arp_announce = 2 # Set the source IP of the ARP request to the IP on ens33, that is, RIP

[root@RS2 ~]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

Configure VIP on RS

[root@RS1 ~]# ifconfig ens33:0 192.168.101.100/24 up
[root@RS1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:39:72:e7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.210/24 brd 192.168.101.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.101.100/24 brd 192.168.101.255 scope global secondary ens33:0
       valid_lft forever preferred_lft forever
    inet6 fe80::9384:81ed:87b3:c531/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@RS2 ~]# ifconfig ens33:0 192.168.101.100/24 up
[root@RS2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:e0:e7:fa brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.110/24 brd 192.168.101.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.101.100/24 brd 192.168.101.255 scope global secondary ens33:0
       valid_lft forever preferred_lft forever
    inet6 fe80::a481:4471:9402:58ac/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Configure forwarding rules on DR

[root@DR ~]# yum -y install ipvsadm
ipvsadm -A -t 192.168.101.100:80 -s rr
[root@DR ~]# ipvsadm -a -t 192.168.101.100:80 -r 192.168.101.210:80 -g
[root@DR ~]# ipvsadm -a -t 192.168.101.100:80 -r 192.168.101.110:80 -g
[root@DR ~]#  ipvsadm -Sn
-A -t 192.168.101.100:80 -s rr
-a -t 192.168.101.100:80 -r 192.168.101.110:80 -g -w 1
-a -t 192.168.101.100:80 -r 192.168.101.210:80 -g -w 1

LVS load balancing based on TUN

In the cluster environment of LVS (NAT) mode, the response of all packet requests needs to be processed by the scheduler, but in TUN mode, the problems in NAT mode are solved. Because the request packet of the data packet is often much smaller than the size of the response packet. Because the response packet contains specific data required by the customer, the principle of TUN is to separate the request from the response data., Let the scheduler only process the data request, and let the real server respond to the data packet and return it directly to the customer.

IP (Tunneling) is a packet subcontracting technology. It can subcontract the original packet and add a new packet header (including new source address and port, target address and port), so as to subcontract a packet whose target is the vip address of the scheduler and forward it to the real back-end server through the tunnel, By repacking the original data packets sent by the client to the scheduler and adding a new data packet header (modifying the target address to the IP address and corresponding port of the real service selected by the scheduler), the LVS (Tun) mode requires that the real server can connect with the external network, and the real server directly returns response data to the client after receiving the requested data packet

RIP, DIP and VIP must be public network addresses
RS gateway cannot point to DIP
The request message must be dispatched through the director, but the response message must not be dispatched through the director
Port mapping is not supported
The OS of RS must support tunnel function

lvs-fullnat: keepalived

The director forwards the request message by modifying the target address and source address at the same time
LVS fullnat features:
VIP is the public network address, RIP and DIP are private network addresses, and RIP and DIP do not need to be in the same network
The source address of the request message received by RS is DIP, so it should respond to DIP
Both request message and response message must be sent through Director
Support port mapping mechanism
RS can use any OS

Tags: Linux

Posted on Thu, 14 Oct 2021 15:08:25 -0400 by socio

Programmer Help

Three modes of LVS

Three modes of LVS

NAT network address translation

Features of LVS-NAT:

Implementing httpd load balancing in NAT mode

DR direct routing mode

Features of LVS-DR:

DR mode to achieve httpd load balancing

LVS load balancing based on TUN

Hot Tags