Three modes of LVS
Device address naming of LVS Cluster
-
CIP: Client IP, Client IP address
-
VIP: Virtual IP, LVS IP address for user requests
-
DIP: Director IP, the IP address where LVS users communicate with back-end servers
-
RIP: Real server IP, the IP address used by the back-end server to communicate with LVS
NAT network address translation
(Network address translation)
LVS NAT is essentially a multi-objective DNAT (iptables)
LVS NAT mode forwards the request message by modifying the target IP address (and possibly the target port) to the selected RIP address of an RS
LVS-NAT model is similar to DNAT, and its working mechanism is the same as DNAT. When the client requests cluster services, LVS modifies the target address of the request message to RIP, forwards it to the backend RealServer, and modifies the source address of the backend response message to VIP to respond to the client.
Features of LVS-NAT:
-
RS and DIP should use private network address, and RS gateway should point to DIP
-
Both request and response messages must be forwarded through the director, so the forwarding function of the director needs to be turned on during configuration. In the scenario of extremely high load, the director may become a system performance bottleneck
-
Support port mapping
-
RS can use any OS
-
RIP of RS and DIP of Director must be on the same IP network
Implementing httpd load balancing in NAT mode
name | network address |
---|---|
DS | DIP:192.168.101.200;VIP:192.168.101.100 |
RS(1) | DIP:192.168.101.110;VIP:192.168.101.2 |
RS(2) | DIP:192.168.101.210;VIP:192.168.101.2 |
Turn off firewall
[root@DR ~]# systemctl disable --now firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@DR ~]# setenforce 0 [root@RS1 ~]# systemctl disable --now firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@RS1 ~]# setenforce 0 [root@RS2 ~]# systemctl disable --now firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@RS2 ~]# setenforce 0
Configure temporary IP for DR
[root@DR ~]# ifconfig ens33:0 192.168.101.100/24 up [root@DR ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:27:fa:31 brd ff:ff:ff:ff:ff:ff inet 192.168.101.200/24 brd 192.168.101.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.101.100/24 brd 192.168.101.255 scope global secondary ens33:0 valid_lft forever preferred_lft forever inet6 fe80::c4bc:49bf:b759:3272/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@DR ~]# tail /etc/sysconfig/network-scripts/ifcfg-ens33 IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 UUID=c5c9eb7d-47c3-4b19-aa9c-e7c78db2c28a DEVICE=ens33 ONBOOT=yes IPADDR=192.168.101.200 NETMASK=255.255.255.0 GATEWAY=192.168.101.2 # gateway DNS1=114.114.114.114
Download httpd and modify the display
[root@RS1 ~]# yum -y install httpd [root@RS1 ~]# systemctl enable --now httpd [root@RS1 ~]# echo "hello 192.168.101.210" > /usr/share/httpd/noindex/index.html [root@RS1 ~]# systemctl restart httpd [root@RS2 ~]# yum -y install httpd [root@RS2 ~]# echo "hello 192.168.101.110" > /usr/share/httpd/noindex/index.html [root@RS2 ~]# systemctl restart httpd
Configure RS1 gateway as DR gateway
[root@RS1 ~]# tail /etc/sysconfig/network-scripts/ifcfg-ens33 IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 UUID=ed299cc0-ef82-48a9-9f37-2a1639dae9f8 DEVICE=ens33 ONBOOT=yes IPADDR=192.168.101.210 NETMASK=255.255.255.0 GATEWAY=192.168.101.2 DNS1=114.114.114.114
Configure RS2 gateway as DR gateway
[root@RS2 ~]# tail /etc/sysconfig/network-scripts/ifcfg-ens33 IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 UUID=1d3ba2cd-8e05-4e5f-a390-3b21e137e1d9 DEVICE=ens33 ONBOOT=yes IPADDR=192.168.101.110 NETMASK=255.255.255.0 GATEWAY=192.168.101.2 DNS1=114.114.114.114
Configure forwarding rules on DR
[root@DR ~]# yum -y install ipvsadm [root@DR ~]# vim /etc/sysctl.conf [root@DR ~]# sysctl -p net.ipv4.ip_forward = 1 [root@DR ~]# vim /etc/sysctl.conf [root@DR ~]# sysctl -p net.ipv4.ip_forward = 1 [root@DR ~]# ipvsadm -A -t 192.168.101.100:80 -s rr [root@DR ~]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.101.100:80 rr [root@DR ~]# ipvsadm -a -t 192.168.101.100:80 -r 192.168.101.210:80 -m [root@DR ~]# ipvsadm -a -t 192.168.101.100:80 -r 192.168.101.110:80 -m [root@DR ~]# ipvsadm -Sn > /etc/sysconfig/ipvsadm [root@DR ~]# cat /etc/sysconfig/ipvsadm -A -t 192.168.101.100:80 -s rr -a -t 192.168.101.100:80 -r 192.168.101.110:80 -m -w 1 -a -t 192.168.101.100:80 -r 192.168.101.210:80 -m -w 1
test
[root@DR ~]# curl http://192.168.101.100 hello 192.168.101.110 [root@DR ~]# curl http://192.168.101.100 hello 192.168.101.210
DR direct routing mode
-
(Direct routing) the default mode of lvs (using the same VIP)
-
LVS Dr mode forwards by modifying the target MAC address of the request message
Director: the scheduler needs to configure VIP and DIP
RSs: all real servers should be configured with RIP and VIP
Features of LVS-DR:
Ensure that the front-end router sends the request message with the target IP of VIP to the director. There are three solutions:
-
Static binding
-
arptables
-
Modify 2 parameters of RS host kernel:
arp_ Announcement: whether to receive and record other people's announcements and whether to announce your mac address to others.
arp_ Announcement has three optional values, 0, 1 and 2, which have the following meanings:
0 (default, the default value is 0): announce all the addresses you own
1: try not to announce the addresses that are not in the same network segment as yourself. For example, do not announce the addresses of 2.0 network segment to 1.0 network segment, and do not announce the addresses of 3.0 network segment to 1.0 network segment. However, try not to announce them, and they may still be announced
Try not to make arp response at the local address of the subnet of the network interface
This is useful when the source IP address of the ARP request is set to reach this network interface via routing
At this time, it will check whether the visiting IP is one of the IP in the subnet segment on all interfaces. If the visiting IP does not belong to the subnet segment on each network interface, it will be processed in Level 2
2: always do not announce the addresses that are not in the same network segment as yourself. For example, do not announce the addresses of 2.0 network segment to 1.0 network segment, and do not announce the addresses of 3.0 network segment to 1.0 network segment. Never announce them
Use the most appropriate local address for the query target. In this mode, the source address of this IP packet will be ignored and an attempt will be made to select a local address that can communicate with this address
The first is to select the local address containing the target IP address in the outgoing access subnet of all network interfaces
If no suitable address is found, the current sending network interface or other network interface that may receive the ARP response will be selected for transmission
arp_ignore: whether to respond to ARP requests
arp_ignore has 9 optional values, 0, 1, 2, 3, 4, 5, 6, 7 and 8, which have the following meanings:
0 (default, the default value is 0): respond to arp query requests for any local IP address on any network interface
1: From which interface does the request message enter and the requested target address is the address configured for this interface, it will respond. Otherwise, it will not respond, and only answer the ARP query request that the target IP address is the local address of the visiting network interface
2: Only answer the ARP query request that the target IP address is the local address of the visiting network interface, and the visiting IP must be within the subnet segment of the network interface
3: It does not respond to the arp request of the network interface, but only responds to the set unique and connection address
4-7: reserved unused
8: Do not respond to all arp queries (local addresses)
-
- The RIP of RS can use either private address or public address
-- RS and Director must be in the same physical network, and there must be no router between them
-- the request message is dispatched through the Director, but the response message must not be dispatched through the Director
-- port mapping is not supported
-- RS can be most OS
-- RS gateway cannot point to DIP
DR mode to achieve httpd load balancing
Experimental configuration information
name | network address |
---|---|
DS | DIP:192.168.101.200;VIP:192.168.101.100 |
RS(1) | DIP:192.168.101.110;VIP:192.168.101.100 |
RS(2) | DIP:192.168.101.210;VIP:192.168.101.100 |
Turn off firewall
[root@DR ~]# systemctl disable --now firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@DR ~]# setenforce 0 [root@RS1 ~]# systemctl disable --now firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@RS1 ~]# setenforce 0 [root@RS2 ~]# systemctl disable --now firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@RS2 ~]# setenforce 0
RS1 and RS2 download and install httpd modifications to provide access
[root@RS1 ~]# yum -y install httpd [root@RS1 ~]# systemctl enable --now httpd [root@RS1 ~]# echo "hello 192.168.101.210" > /usr/share/httpd/noindex/index.html [root@RS1 ~]# systemctl restart httpd [root@RS2 ~]# yum -y install httpd [root@RS2 ~]# echo "hello 192.168.101.110" > /usr/share/httpd/noindex/index.html [root@RS2 ~]# systemctl restart httpd
DR configuration VIP
[root@DR ~]# ifconfig ens33:0 192.168.101.100/24 up [root@DR ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:27:fa:31 brd ff:ff:ff:ff:ff:ff inet 192.168.101.200/24 brd 192.168.101.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.101.100/24 brd 192.168.101.255 scope global secondary ens33:0 valid_lft forever preferred_lft forever inet6 fe80::c4bc:49bf:b759:3272/64 scope link noprefixroute valid_lft forever preferred_lft forever
RS configuration apr kernel parameters
[root@RS1 ~]# cat /etc/sysctl.conf # Add two lines net.ipv4.conf.all.arp_ignore = 1 # Set the corresponding network card to respond only to ARP requests with the target IP as its own interface address net.ipv4.conf.all.arp_announce = 2 # Set the source IP of the ARP request to the IP on ens33, that is, RIP [root@RS1 ~]# sysctl -p net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 [root@RS2 ~]# cat /etc/sysctl.conf # Add two lines net.ipv4.conf.all.arp_ignore = 1 # Set the corresponding network card to respond only to ARP requests with the target IP as its own interface address net.ipv4.conf.all.arp_announce = 2 # Set the source IP of the ARP request to the IP on ens33, that is, RIP [root@RS2 ~]# sysctl -p net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2
Configure VIP on RS
[root@RS1 ~]# ifconfig ens33:0 192.168.101.100/24 up [root@RS1 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:39:72:e7 brd ff:ff:ff:ff:ff:ff inet 192.168.101.210/24 brd 192.168.101.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.101.100/24 brd 192.168.101.255 scope global secondary ens33:0 valid_lft forever preferred_lft forever inet6 fe80::9384:81ed:87b3:c531/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@RS2 ~]# ifconfig ens33:0 192.168.101.100/24 up [root@RS2 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:e0:e7:fa brd ff:ff:ff:ff:ff:ff inet 192.168.101.110/24 brd 192.168.101.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.101.100/24 brd 192.168.101.255 scope global secondary ens33:0 valid_lft forever preferred_lft forever inet6 fe80::a481:4471:9402:58ac/64 scope link noprefixroute valid_lft forever preferred_lft forever
Configure forwarding rules on DR
[root@DR ~]# yum -y install ipvsadm ipvsadm -A -t 192.168.101.100:80 -s rr [root@DR ~]# ipvsadm -a -t 192.168.101.100:80 -r 192.168.101.210:80 -g [root@DR ~]# ipvsadm -a -t 192.168.101.100:80 -r 192.168.101.110:80 -g [root@DR ~]# ipvsadm -Sn -A -t 192.168.101.100:80 -s rr -a -t 192.168.101.100:80 -r 192.168.101.110:80 -g -w 1 -a -t 192.168.101.100:80 -r 192.168.101.210:80 -g -w 1
LVS load balancing based on TUN
In the cluster environment of LVS (NAT) mode, the response of all packet requests needs to be processed by the scheduler, but in TUN mode, the problems in NAT mode are solved. Because the request packet of the data packet is often much smaller than the size of the response packet. Because the response packet contains specific data required by the customer, the principle of TUN is to separate the request from the response data., Let the scheduler only process the data request, and let the real server respond to the data packet and return it directly to the customer.
IP (Tunneling) is a packet subcontracting technology. It can subcontract the original packet and add a new packet header (including new source address and port, target address and port), so as to subcontract a packet whose target is the vip address of the scheduler and forward it to the real back-end server through the tunnel, By repacking the original data packets sent by the client to the scheduler and adding a new data packet header (modifying the target address to the IP address and corresponding port of the real service selected by the scheduler), the LVS (Tun) mode requires that the real server can connect with the external network, and the real server directly returns response data to the client after receiving the requested data packet
-
RIP, DIP and VIP must be public network addresses
-
RS gateway cannot point to DIP
-
The request message must be dispatched through the director, but the response message must not be dispatched through the director
-
Port mapping is not supported
-
The OS of RS must support tunnel function
lvs-fullnat: keepalived
-
The director forwards the request message by modifying the target address and source address at the same time
LVS fullnat features: -
VIP is the public network address, RIP and DIP are private network addresses, and RIP and DIP do not need to be in the same network
-
The source address of the request message received by RS is DIP, so it should respond to DIP
-
Both request message and response message must be sent through Director
-
Support port mapping mechanism
-
RS can use any OS