Virtual Server via IP Tunneling
This page contains information about the working principle of Virtual Server via IP Tunneling and how to use IP Tunneling to greatly increase the scalability of server clusters.
IP tunneling
IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagrams, which allows datagrams destined for one IP address to be wrapped and redirected to another IP address. IP encapsulation is now commonly used in Extranet, Mobile-IP, IP-Multicast, tunneled host or network. Please see the NET-3-HOWTO document for details.
How to use IP tunneling on virtual server
First, let's look at the figure of virtual server via IP tunneling. The most different thing of virtual server via IP tunneling to that of virtual server via NAT is that the load balancer sends requests to real servers through IP tunnel in the former, and the load balancer sends request to real servers via network address translation in the latter.
When a user accesses a virtual service provided by the server cluster, a packet destined for virtual IP address (the IP address for the virtual server) arrives. The load balancer examines the packet's destination address and port. If they are matched for the virtual service, a real server is chosen from the cluster according to a connection scheduling algorithm, and the connection is added into the hash table which records connections. Then, the load balancer encapsulates the packet within an IP datagram and forwards it to the chosen server. When an incoming packet belongs to this connection and the chosen server can be found in the hash table, the packet will be again encapsulated and forwarded to that server. When the server receives the encapsulated packet, it decapsulates the packet and processes the request, finally return the result directly to the user according to its own routing table. After a connection terminates or timeouts, the connection record will be removed from the hash table. The workflow is illustrated in the following figure.
Note that real servers can have any real IP address in any network, they can be geographically distributed, but they must support IP encapsulation protocol. Their tunnel devices are all configured up so that the systems can decapsulate the received encapsulation packets properly, and the <Virtual IP Address> must be configured on non-arp devices or any alias of non-arp devices, or the system can be configured to redirect packets for <Virtual IP Address> to a local socket. See the arp problem page for more information.
Finally, when an encapsulated packet arrives, the real server decapsulates it and finds that the packet is destined for <Virtual IP Address>, it says, "Oh, it is for me, so I do it.", it processes the request and returns the result directly to the user in the end.
How to build the kernel
First, get a fresh copy of the Linux kernel source of the right version. Second, get a right version of IP virtual server patch and apply it to the kernel. Third, make sure that some kernel compile options must be selected. Fourth, rebuild the kernel. Once you have your kernel properly built, update your system kernel and reboot.
1. The VS patch for kernel 2.0.36
Kernel Compile Options:
Code maturity level options --->
[*] Prompt for development and/or incomplete code/drivers
Networking options --->
[*] Network firewalls
...
[*] IP: forwarding/gatewaying
...
[*] IP: firewalling
...
[*] IP: masquerading
...
[*] IP: ippfvs(LinuxDirector) masquerading (EXPERIMENTAL)
Virtual server request dispatching technique---
( ) VS-NAT
(X) VS-Tunneling
( ) VS-DRouting
And, you have to choice one scheduling algorithm.
Virtual server scheduling algorithm
(X) WeightedRoundRobin
( ) LeastConnection
( ) WeightedLeastConnection
[ ] IP: enabling ippfvs with the local node feature
Finally, cd the ippfvsadm source and type "make install" to install ippfvsadm into your system directory.
2. The VS patch for kernel 2.2.x
Kernel Compile Options:
Code maturity level options --->
[*] Prompt for development and/or incomplete code/drivers
Networking options --->
[*] Network firewalls
...
[*] IP: forwarding/gatewaying
...
[*] IP: firewalling
...
[*] IP: masquerading
...
[*] IP: masquerading virtual server support (EXPERIMENTAL)
(12) IP masquerading table size (the Nth power of 2)
<M> IPVS: round-robin scheduling
<M> IPVS: weighted round-robin scheduling
<M> IPVS: least-connection scheduling
<M> IPVS: weighted least-connection scheduling
<M> IPVS: locality-based least-connection scheduling
<M> IPVS: locality-based least-connection with replication scheduling
Finally, cd the ipvsadm source and type "make install" to install ipvsadm into your system directory, or install ipvsadm rpm package.
3. The IPVS patch for kernel 2.4.x
Kernel Compile Options:
Code maturity level options --->
[*] Prompt for development and/or incomplete code/drivers
Networking options --->
[*] Network packet filtering (replaces ipchains)
[ ] Network packet filtering debugging
...
IP: Netfilter Configuration --->
IP: Virtual Server Configuration --->
<M> virtual server support (EXPERIMENTAL)
[*] IP virtual server debugging
(12) IPVS connection table size (the Nth power of 2)
--- IPVS scheduler
<M> round-robin scheduling
<M> weighted round-robin scheduling
<M> least-connection scheduling scheduling
<M> weighted least-connection scheduling
<M> locality-based least-connection scheduling
<M> locality-based least-connection with replication scheduling
<M> destination hashing scheduling
<M> source hashing scheduling
--- IPVS application helper
<M> FTP protocol helper
How to use it
Let's give an example to see how to use it. The following table illustrates the rules specified in the Linux box with virtual server via IP tunneling. Note that the services running on the real servers must run on the same port as virtual service, so it is not necessary to specify the service port on the real servers.
| Protocol | Virtual IP Address | Port | Real IP Address | Weight |
| TCP | 202.103.106.5 | 80 | 202.103.107.2 | 1 |
| 202.103.106.3 | 2 |
All traffic destined for IP address 202.103.106.5 Port 80 is load-balanced over real IP address 202.103.107.2 Port 80 and 202.103.106.3 Port 80.
We can use the following commands to specify the rules in the table above in the system.
1. For kernel 2.0.x
ippfvsadm -A -t 202.103.106.5:80 -R 202.103.107.2 -w 1 ippfvsadm -A -t 202.103.106.5:80 -R 202.103.106.3 -w 2
2. For kernel 2.2.x / kernel 2.4.x / kernel 2.6.x
ipvsadm -A -t 202.103.106.5:80 -s wlc ipvsadm -a -t 202.103.106.5:80 -r 202.103.107.2 -i -w 1 ipvsadm -a -t 202.103.106.5:80 -r 202.103.106.3 -i -w 2
My example for testing virtual server via tunneling
Here is my configure example for testing virtual server via tunneling. The configuration is as follows. I hope it can give you some clues. The load balancer has 172.26.20.111 address, and the real server 172.26.20.112. The 172.26.20.110 is the virtual IP address. In all the following examples, "telnet 172.26.20.110" will actually reach the real server.
1. For kernel 2.0.x
The load balancer (LinuxDirector), kernel 2.0.36
ifconfig eth0 172.26.20.111 netmask 255.255.255.0 broadcast 172.26.20.255 up route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0 ifconfig eth0:0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up route add -host 172.26.20.110 dev eth0:0 ippfvsadm -A -t 172.26.20.110:23 -R 172.26.20.112
The real server 1, kernel 2.0.36 (IP forwarding enabled)
ifconfig eth0 172.26.20.112 netmask 255.255.255.0 broadcast 172.26.20.255 up route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0 ifconfig tunl0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up route add -host 172.26.20.110 dev tunl0
When I am on other hosts, 'telnet 172.26.20.110' will actually connect the real server 1.
2. For kernel 2.2.x
The load balancer (LinuxDirector), kernel 2.2.14
ifconfig eth0 172.26.20.111 netmask 255.255.255.0 broadcast 172.26.20.255 up ifconfig eth0:0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up echo 1 > /proc/sys/net/ipv4/ip_forward ipvsadm -A -t 172.26.20.110:23 -s wlc ipvsadm -a -t 172.26.20.110:23 -r 172.26.20.112 -i
The real server 1, kernel 2.0.36 (IP forwarding enabled)
ifconfig eth0 172.26.20.112 netmask 255.255.255.0 broadcast 172.26.20.255 up route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0 ifconfig tunl0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up route add -host 172.26.20.110 dev tunl0
More configuration examples
Here are more configuration examples of virtual server via IP tunneling. In order to save space, just important commands are put and less importants are omitted.
1. Real server running kernel 2.2.14 or later with hidden device
The load balancer (LinuxDirector), kernel 2.2.14
echo 1 > /proc/sys/net/ipv4/ip_forward ipvsadm -A -t 172.26.20.110:23 -s wlc ipvsadm -a -t 172.26.20.110:23 -r 172.26.20.112 -i
The real server 1, kernel 2.2.14
echo 1 > /proc/sys/net/ipv4/ip_forward # insert it if it is compiled as module modprobe ipip ifconfig tunl0 0.0.0.0 up echo 1 > /proc/sys/net/ipv4/conf/all/hidden echo 1 > /proc/sys/net/ipv4/conf/tunl0/hidden ifconfig tunl0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up
Since the kernel 2.2 just has one tunnel device tunl0, you can only have one VIP in this configuration. For multiple VIPs, you can make the tunl0 device up, and configure them on aliases of tunnel/dummy/loopback devices and hide that device. An example is as follows:
echo 1 > /proc/sys/net/ipv4/ip_forward # insert it if it is compiled as module modprobe ipip ifconfig tunl0 0.0.0.0 up ifconfig dummy0 0.0.0.0 up echo 1 > /proc/sys/net/ipv4/conf/all/hidden echo 1 > /proc/sys/net/ipv4/conf/dummy0/hidden ifconfig dummy0:0 172.26.20.110 up route add -host 172.26.20.110 dev dummy0:0 ifconfig dummy0:1 <Another-VIP> up ...
2. Real servers runing kernel 2.2.x with redirect approach
The load balancer's configuration is the same as the example above. Real servers running kernel 2.2.x can be configured as follows:
echo 1 > /proc/sys/net/ipv4/ip_forward # insert it if it is compiled as module modprobe ipip ifconfig tunl0 0.0.0.0 up ipchains -A input -j REDIRECT 23 -d 172.26.20.110 23 -p tcp ...