LVS Cluster Configuration HOWTO

Wayne Sherrill (sherrill@redhat.com)

This document explains how to set up and administer a Linux Virtual Server (LVS) cluster that provides highly available Web and FTP services.

1. Introduction

A Linux Virtual Server (LVS) cluster is a collection of servers that have been specially configured to provide highly available Web and FTP services. The diagram below illustrates how an LVS cluster works. Service requests arriving at an LVS cluster are addressed to a virtual server: a publicly advertised, fully-qualified domain name that is associated with a floating IP address, and which can be migrated to a different host.

Figure 1. LVS Cluster Interactions

_/\__/\_ | | / Internet \ \____/ \/ \/ \/ | |

LVS Cluster Configuration HOWTO

Virtual Server IP/FQDN |——————————————| |eth0 |eth0

——|—– ——-|—–

| Primary | | Backup | | Node | | Node | LVS | | | | routers ——|—– ——-|—–

|eth1 |eth1 |——-|————|—————–|—|

|  |  |
|  |  |
|  |  |

|—-|—-| |—-|—-| |—-|—-|

| Web/FTP | | Web/FTP | | Web/FTP | real | Node#1 | | Node#2 | ... | Node#n | servers |_________| |_________| |_________|

An LVS cluster consists of one or two router nodes (top of figure) and a variable number of Web/FTP servers (bottom). We will refer to the LVS router nodes as LVS routers, and to the pool of Web/FTP servers as real servers. The real servers are connected via a private network. The LVS routers are connected to this private network and also to the public network. The adapters connecting the LVS routers to the public and private networks (eth0 and eth1 in the figure) may be any device but must be the same on each router.

Only one LVS router is active at a time. The role of the active router is to redirect service requests from the virtual server address to the real servers. The redirection is based on one of the four supported load-balancing algorithms (described in Table 1). The active router dynamically monitors the health of the real servers, and the workload on each, via one of three supported methods (described in Table 2). If a real server becomes disabled, the active router stops sending jobs to the server until it returns to normal operation.

Periodically, the LVS routers exchange "I’m alive" heartbeat messages over their public connection. Should the backup node fail to receive a heartbeat message within an expected interval, it initiates failover in order to assume the role of the active router. During failover, the backup router takes over the failed router’s floating IP addresses (as indicated in the cluster configuration file), and uses the technique known as ARP spoofing: starts announcing itself as the destination for IP packets addressed to the failed node. When the failed node returns to service, it assumes the hot backup role.

Currently, the LVS cluster supports one routing method, Network Address Translation (NAT). (In the future, tunneling and direct routing will be added.) The diagram below illustrates how a NAT virtual server works.

Figure 2. An LVS Cluster Implemented with NAT Routing

_/\__/\_ | | / Internet \ \____/ \/ \/ \/ | |

|——————————————| public

| network eth0 | virtual server IP address

——————– | active router | | | ——————–

eth1 | NAT router IP address

|

|——-|————————-|——–| private | | network | | | | |—-|———-| |———–|—| | real server#1 | | real server#2 | | | | | ... |_______________| |_______________|

Client requests for service arrive at a virtual server IP address. This is a publicly-advertised address that an administrator at the site will have associated with a fully-qualified domain name (for example, lvs.ajax.com). The illustration shows only one virtual server address, but there may be many. A unique virtual server address is a triplet comprising a protocol (TCP or UDP), IP address, and port number.

The IP components of virtual server addresses are floating addresses. They may be aliased to the device (for example, eth0:1) that connects the LVS routers to the public network, or each could be associated with a separate device. The NAT router IP address, also a floating IP address, is the default route used by each real server on the private network to communicate with the active router. As with virtual server addresses, the NAT router IP address may be aliased to the device (for example, eth1:1) connecting the virtual server to the network of real servers, or it could be associated with a separate device.

Virtual server and NAT addresses are enabled only on the active router. Thus, should the active router fail, the backup router enables the virtual server and NAT addresses during take-over of the floating IP addresses. In the topology shown in Figure 2, virtual server addresses are enabled on device eth0 and the NAT router address on eth1.

The IPVS table in the kernel maps requests from the virtual server address to a real server on the private network. For example, a TCP request addressed to port 80 on virtual server 1.2.3.1 might be routed to port 80 on real server 192.168.1.2. The actual mapping in the IPVS table of jobs to real servers is based on which load-balancing algorithm is in use. Table 1 describes the supported load-balancing methods.

Table 1. Load-balancing Methods

Name  Description

Name  Description

Round robin Distribute jobs equally among the real servers.

Least-connections Distribute more jobs to real servers with fewer active connections. (The IPVS table stores active connections.)

Weighted round robin Distribute more jobs to servers with greater capacity. Capacity is indicated by the user-assigned weight, which is adjusted upward or downward by dynamic load information.

Weighted least-connections Distribute more jobs to servers with fewer active connections relative to their capacity. Capacity is indicated by the user-assigned weight, which is adjusted upward or downward by dynamic load information.

As a real server processes a request, it returns packets to the active router, where the address of the real server in the packets is replaced by the virtual server address. In this manner, the private network of real servers is masqueraded from requesting clients.

2. Components of an LVS Cluster

The components of an LVS cluster are described below.

2.1. pulse

This is the controlling process that starts the other daemons as needed. It is started on the LVS routers by the /etc/rc.d/init.d/pulse script, normally at boot time . Through pulse, which implements a simple heartbeat, the inactive LVS router determines the health of the active router and whether to initiate failover.

2.2. lvs

The lvs daemon runs on the LVS routers. It reads the configuration file and calls ipvsadm to build and maintain the IPVS routing table.

2.3. nanny

The nanny monitoring daemon runs on the active LVS router. Through this daemon, the active router determines the health of each real server and gets its workload. A separate process runs for each real server used by each virtual server.

2.4. /etc/lvs.cf

This is the LVS cluster configuration file. Directly or indirectly, all daemons get their configuration information from this file.

2.5. piranha

The GUI tool for monitoring, configuring, and administering an LVS cluster. Normally this is the tool you will use to maintain /etc/lvs.cf, restart the running daemons, and monitor an LVS cluster.

2.6. ipvsadm

This tool updates the IPVS routing table in the kernel. The lvs daemon sets up and administers an LVS cluster by calling ipvsadm to add, change or delete

entries in the IPVS routing table.

3. Background of the LVS Cluster

The Red Hat LVS cluster is based either directly on contributions from the Linux community, or on components that were inspired or enriched by Linux community projects.

The primary source of the LVS cluster is Wensong Zhang’s Linux Virtual Server (LVS) kernel routing algorithm (see http://www.linuxvirtualserver.org). The capabilities of the LVS project that the Red Hat LVS cluster currently supports are:

Building virtual servers: floating IP addresses where requests for service arrive from the public internet.

Routing service requests from virtual servers to a pool of real servers.

Load-balancing (see Table 1).

Packet-forwarding Network Address Translation (NAT).

The LVS innovations supported by the Red Hat LVS cluster are based on a number of technologies including Network Address Translation (NAT), IP Masquerading, and Port Forwarding. For a good general discussion and index of the relevant HOWTOs on these and related topics, see http://www.linas.org/linux/load.html.

4. Hardware/Network Requirements

An LVS cluster consists of one or two LVS routers and a collection of real servers providing Web and FTP services. The connection and hardware requirements are described below.

4.1. LVS Routers

A Linux server that will be the primary LVS router is required. This machine needs two network adapters, one connecting it to the public network and the other to a private network of real servers.

If you want failover capability, a second Linux server that will be the backup LVS router is required. This machine also needs two network adapters connecting it to the public network and to the private network of real servers. The adapter devices must match on the two LVS routers. Thus, if devices eth0 and eth1 on the primary LVS router connect it to public and private network, respectively, then these same devices on the backup LVS router must connect it to the public and private network. Note that the backup LVS router is a purely hot standby machine.

4.2. Real Servers

The private network to which the LVS routers are connected includes a variable number of Web/FTP server hosts. Jobs addressed to virtual servers are redirected to these real servers, which may be any kind of computing platform running any operating system or Web server.

During configuration, you assign a weight to each real server. This is an integer reflecting each server’s processing capacity (based on memory, processor speed, number of processors, etc.) relative to that of the others. It is the ratio (2/1, 20/10, 200/100) that is significant. For example, a weight of 2000 assigned to a server indicates that it has twice the computing power of a server assigned a weight of 1000. The assigned weight, which may be dynamically adjusted based on load information, is used by two available job scheduling algorithms (described in Table 1). You should be prepared to assign accurate weights.

5. LVS Router Prerequisites

The LVS routers require Red Hat Linux 6.1 or greater. Also, packet forwarding, packet defragmenting, and IP masquerading must be enabled on the LVS routers.

To enable packet forwarding and defragmenting, make sure the following two

lines are present in /etc/sysconf/network:

FORWARD_IPV4=yes

DEFRAG_IPV4=yes

These lines cause /etc/rc.d/rc.sysinit to execute these two commands whenever the routers boot:

echo 1 > /proc/sys/net/ipv4/ip_forward

echo 1 > /proc/sys/net/ipv4/ip_always_defrag

To enable IP masquerading, issue this command:

ipchains -A forward -j MASQ -s n.n.n.n/type -d 0.0.0.0/0

where:

n.n.n.n is the address of the private subnet to which the real servers are connected.

type is 8, 16, 24, or 32 indicating the address type and mask:

netmask | type | Subnet

~~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~~~~~~~ 255.0.0.0 | 8 | Class A 255.255.0.0 | 16 | Class B 255.255.255.0 | 24 | Class C 255.255.255.255 | 32 | Point-to-point

You will probably want to put the ipchains command in an init script (e.g., /etc/rc.d/rc.local), so that masquerading is configured on the LVS routers at system startup.

ipchains is the tool used to create and manage firewalling rules set in the kernel’s TCP stack. Masquerading is a small subset of these rules that allows machines making use of private IP networks to communicate with the outside world. Using ipchains can have an impact on system security. If you have security concerns, read the ipchains HOWTO (http://www.linuxdoc.org/HOWTO/IPCHAINS-HOWTO.html).

6. Cluster Node Interconnection Prerequisites

During configuration, you select the tool family (rsh or ssh) that will be used to synchronize the /etc/lvs.cf configuration files on the LVS routers. The selected tool must be enabled on the LVS routers, such that root on each router can log in to the other router without administrator intervention.

Also during configuration, you select the tool (uptime, ruptime, or rup) that the active router will use to monitor the workload on the real servers. Enable the selected tool on the real servers. If this cannot be done (for example, one of your real servers is a Windows/NT Web server), the cluster will still provide highly available services. However, the Weighted round robin and Weighted least-connections algorithms (described in Table 1 will be affected. Namely, since load information will not be available, the user-assigned weights will be applied statically rather than dynamically adjusted based on server workload.

Table 2 describes in general terms what you do to enable these tools on the source and destination hosts. For more detailed information, see the man pages. Note that, with rsh and ssh, root must be able to log in over the network. To enable remote root login to a Red Hat Linux system, remove the following line

from the file /etc/pam.d/login:

auth required /lib/security/pam_security.so This is a security hole, albeit small. Make sure you have the LVS nodes properly firewalled so that logins are allowed only from trusted sources.

Table 2. Enabling Synchronization and Monitoring Tools

Tool  Do This

rsh

Create a .rhosts file with permission 600 in the root on the destination host naming the source host and user (for example, foo.host1.com root).

ssh Obtain/install the tool, which for legal reasons is not released with international Linux distributions. On source and destination hosts, disable remote login via all other methods, set up RSA-based authentication using

.ssh/authorized_keys, and start sshd.

uptime On each real server, enable either rsh or ssh as described above.

ruptime Set up each LVS router and real server to start rwhod whenever it boots.

rup Set up each real server to start rpc.rstatd whenever it boots.

7. Installing the Software

The LVS cluster software is released in three RPM packages:

piranha (programs) piranha-gui (graphical configuration tool) piranha-docs (documentation)

You can install these packages from the 6.1 CD-ROM with this command:

#rpm -Uvh /mnt/cdrom/RedHat/RPMS/piranha*.rpm

To obtain possible updates of these (or other) packages for your hardware platform, visit http://www.redhat.com/corp/support/errata/rh61-errata-updates.html on the Red Hat website.

8. Configuring an LVS Cluster

You set up and maintain an LVS cluster from the LVS routers, by editing the configuration file and starting or restarting the pulse daemon. Specifically, the steps are:

1. On the primary router, edit the configuration file /etc/lvs.cf.

2. Copy the edited configuration file to the backup router.

3. Start (or restart) the pulse daemon first on the primary router, then on the backup router.

You can perform these steps from the shell, by editing the configuration file with the editor of your choice. The shell commands for starting, restarting, and stopping the pulse daemon are:

/etc/rc.d/init.d/pulse start /etc/rc.d/init.d/pulse restart /etc/rc.d/init.d/pulse stop

The pulse daemon starts or restarts the other LVS cluster daemons as needed, which obtain their configuration information, directly or indirectly, from the current configuration file.

If you stop pulse (in order to shut down the cluster), stop it on the backup router first. This will prevent the backup router from possibly taking over the active role.

Alternatively, you can use piranha to set up, monitor, and administer your LVS cluster. The entry fields on its windows set or change lines in /etc/lvs.cf, and there are buttons to start, stop, and restart the cluster. Use of piranha requires that the X Window System be configured on the LVS routers. An advantage of using piranha is that it synchronizes the configuration file on the primary and backup routers, and you perform all administrative tasks from the active router.

The next section describes the LVS cluster configuration file. Read this section if you want to edit this file manually. If you chose to use piranha, read Section 8.2.

8.1. Editing the Configuration File

The /etc/lvs.cf file has three sections. The global section, described in Table 3, sets up the LVS routers and specifies networking and heartbeat parameters. There is one set of parameters for the cluster. The per-virtual-server section, described in Table 4, defines virtual server addresses, sets up the associations between virtual servers and real servers, and specifies job-scheduling parameters. There is a separate set of parameters for each defined virtual server. The per-real-server section, described in Table 5, defines the real servers routed to from each virtual server. There is one set of these parameters for each virtual server.

Table 3. Setting Global Parameters

Parameter  Description

primary = Enter the IP address of the adapter connecting the primary LVS router to the public network.

backup = Enter the IP address of the adapter connecting the backup backup LVS router to the public network.

Parameter  Description

heartbeat_port = Enter the port number used for the heartbeat on the primary and backup LVS routers.

keepalive = Enter the number of seconds between heartbeats.

deadtime = Enter the number of seconds to wait before declaring a non-responding router dead and initiating failover.

rsh_command = [rsh|ssh] Enter the command family to use for synchronizing the configuration files on the primary and backup routers. Important: as described in Table 2, you must enable the selected command on the primary and backup routers.

network = [nat|direct|tunnel] Currently, only Network Address

Translation (nat) is supported.

nat_router = Enter the floating IP address and device of the NAT router. This IP address must be the default route used by each real server to communicate with the active LVS router. The IP address is aliased to the device (for example, eth1:1) connecting the LVS routers to the private network of real servers. The device must be the same (i.e., eth1) on both LVS routers.

Table 4. Setting Per-Virtual-Server Parameters

Parameter  Description

Parameter  Description

name Enter a unique identifier for the virtual server.

address = Enter the virtual server’s IP address: a floating IP address that has been associated with a fully-qualified domain name.

active = [0|1]

Enable (1) or disable (0) this address.

load_monitor = [uptime|ruptime|rup] Select the tool (default uptime) that will be used by the active router to monitor the workload of real servers. Important: as described in Table 2, unless you enable the selected command on the real servers, the scheduling algorithms that use dynamic load information will apply the assigned weight statically rather than adjust the weight from load information. If you select the default (uptime), the tool you specified for "rsh_command" is used to log in to the real servers. This tool must be enabled on the real servers.

timeout = Enter the number of seconds (default 10) that must lapse before a real server determined to be dead is removed from the routing table.

reentry = Enter the number of seconds (default 180) that a restored real server must remain alive before being re-added to the routing table.

Parameter  Description

port = [http/80|ftp/21] Enter the listening port on this virtual server: http for 80 (the default) or ftp for 21. Or you can enter the numbers.

scheduler = [wlc|lc|wrr|rr] Select the scheduling algorithm (default wlc) for distributing jobs from this virtual server to the real servers. The choices are described in Table 1.

Table 5. Setting Per-Real-Server Parameters

Parameter  Description

name

Enter a unique name for the real server.

address = Enter the IP address of the real server on the private network.

active = [0|1]

Enable (1) or disable (0) the real server.

weight = Enter an integer (default is 1) specifying this server’s processing capacity relative to that of other real servers. For example, a server assigned 2000 has twice the capacity of a server assigned 1000. The weighted scheduling algorithms adjust this number dynamically based on workload.

8.2. Using the Piranha Configuration Tool

To start piranha, become root and type piranha &. When you do, its main window opens. The window has four tabs:

Controls/Monitoring – The initial tab. Used to start/stop/restart the cluster daemons and monitor the runtime status. See Section 8.2.1.

Global Settings – Use to set the IP address of the primary LVS router and NAT router. See Section 8.2.2.

Redundancy – Use to set the address of the backup LVS router and set the heartbeating parameters. See Section 8.2.3.

Virtual Servers – Use to set service addresses and set up routing between service addresses and real Web/FTP server hosts. See Section 8.2.4.

Piranha windows have one or more of these buttons along the bottom:

OK – Apply changes and exit from the window.

Apply – Apply changes without exiting the window.

Close – Exit from the window without applying changes.

8.2.1. Controls/Monitoring Tab

Field/Button

Description

Start/Stop When cluster daemons are not running, this button is labeled Start: click it to start the cluster. When the daemons are running, the button is labeled Stop: click it to stop the cluster daemons.

Add pulse daemon to this runlevel Select to start pulse at system boot at

this runlevel.

Update information now Click to display current information in

the kernel routing table.

Auto-update Select to display kernel table routing information automatically at the specified interval.

8.2.2. Global Settings Tab

Field/Button

Description

Primary LVS server IP Contains the public IP address of the

primary LVS router.

NAT Router IP Contains the floating IP address associated with the network adapter connecting the virtual server with the network of real servers. This address is the gateway used by the real servers to communicate with the virtual server. If this node fails, this address migrates to the backup LVS node.

NAT Router Device Names the device with which the NAT router IP address is associated. This is done via address aliasing. For example, the NAT router IP may be aliased to physical device eth1 as eth1:1.

Sync tool Select the tool (rsh or ssh) for synchronizing files on the primary and backup LVS routers. The tool you select must be enabled, such that the LVS routers can log in to one another without needing to enter a password. For general procedures, see Table 2.

8.2.3. Redundancy Tab

Field/Button

Description

Enable redundant server Select to enable failover.

Field/Button

Description

Redundant LVS server IP Contains the public IP address of the

backup LVS node.

Heartbeat interval (seconds) Contains the number of seconds between heartbeats: the interval at which the backup LVS node checks to see if the primary LVS node is alive.

Assume dead after (seconds) If this number of seconds lapses without a response from the primary LVS node, the backup router initiates failover. During failover, the backup router claims all virtual server IPs being processed by the primary LVS router, and broadcasts gratuitous ARPs advertising its MAC address as the destination of packets addressed to the primary router.

Heartbeat port Enter the port used for the heartbeat on the primary and backup LVS routers.

8.2.4. Virtual Servers

This screen displays a row of information for each currently defined virtual server. Click a row to select it. The buttons on the right side of the screen apply to the currently selected virtual server. Click Delete to remove the selected virtual server.

Field/Button Description Field/Button

Description

Status Displays Active or Down. Click Disable to take down a selected active virtual server; click Activate to enable a selected down virtual server. After changing the status, you must restart the pulse daemon to make the change take affect. To do this, go to the Controls/Monitoring tab, click the Stop button (its name changes to Start), then click Start.

Name Contains a unique identifier for the virtual server.

Port Contains the number of the listen port for incoming requests for service.

Protocol

Currently, only TCP is supported.

8.2.4.1. Add/Edit a Virtual Server

Click the Add button to create a new, undefined virtual server. Click the Edit button (or double-click the row) to define or modify a virtual server. Click the Real Servers tab to view or modify the real servers associated with the selected virtual server.

Field/Button

Description

Name Enter a unique identifier for this virtual server.

Application

Click to select HTTP or FTP.

Port Enter the number of the listen port for incoming requests for service.

Field/Button

Description

Address Enter the floating IP address where requests for service arrive. This address will have been associated with the fully-qualified domain name to which requests for service are addressed.

Device Name the adapter device connecting the LVS routers to the public network, with which the floating IP address is associciated via IP aliasing (For example, eth0:1). If the active router fails, the virtual server address and port are failed-over to this device on the backup router, which becomes the active router.

Re-entry Time Enter the number of seconds (default is 180) that a failed real server associated with this virtual server must remain alive before it will be re-added to the cluster.

Service timeout Enter the number of seconds (default is 10) within which a real server must respond to a redirected request. If a host takes longer than this, it is declared dead and removed from the cluster.

Field/Button Description Every two seconds the nanny process on the active router sends a heartbeat (essentially a ping) to each real server. If this succeeds, then a service connect (essentially a telnet conection) is sent out on the specified service port. If nanny gets back any response from a real server, it and the service are assumed to be alive. If "Service timeout" seconds lapse without a response, then the real server is presumed to be dead and is dropped from the routing table. But nanny continues to monitor the server/service by sending out heartbeats every two seconds. If, after a successful ping, a dropped server remains alive for "Re-entry Time" number of seconds, the server is re-added to the kernel routing table.

Load monitoring tool Select the tool to use (uptime, rup, or ruptime) for determining the workload on the real servers. The general procedures for enabling these tools are described in the table in Table 2.

Scheduling Select the algorithm for routing requests from this virtual server to the real servers. The choices are described in Table 1.

8.2.5. Real Servers

This screen displays a row of information for each currently defined real server. Click a row to select it. The buttons on the right side of the screen apply to the currently selected row. Click Delete to remove the selected real server.

Field/Button

Description

Status Displays Active or Down. Click Disable to take down a selected active virtual server; click Activate to enable a selected down virtual server. After changing the status, you must restart the pulse daemon to make the change take affect. To do this, go to the Controls/Monitoring tab, click the Stop button (its name changes to Start), then click Start.

Name Displays the server’s name in the cluster.

Address

Displays the server’s IP address.

8.2.5.1. Add/Edit a Real Server

Click the Add button to create a new, undefined real server. Click the Edit button (or double-click the row) to define or change the selected real server.

Field/Button

Description

Name

Enter a descriptive name.

Address Enter the IP address of the real server on the private network.

Field/Button

Description

Weight Assign an integer to indicate this host’s processing capacity relative to that of other hosts in the pool.

9. Example – Setting Up a Five-node Cluster

This section describes step by step how to create a cluster of two LVS routers and three Web/FTP servers. First, collect information and set up the five systems as explained in the next section. Then, implement the example either from a shell (explained in Section 9.2) or by starting the GUI configuration tool (explained in Section 9.3).

Figure 3 shows the network that will exist after you’ve set up the LVS routers and real servers. All network addresses shown are for purposes of illustration only.

Figure 3. Layout of the Example Network

|——-|——————————————|———| Public network |eth0=1.2.3.2 |eth0=1.2.3.3 |eth0:1=1.2.3.1 (vs1) |

——|—– ——-|—– | active | | backup | | router | | router | | | | | ——|—– ——-|—–

|eth1=192.168.1.1 |eth1=192.168.1.2 |eth1:1=192.168.1.254 (NAT router) |

|——-|-|——————|—————–|—|———-| Private network |eth0=192.168.1.3 |eth0=192.168.1.4 |eth0=192.168.1.5

|  |  |

|———|  |———|  |———|
| rs1 |  | rs2 |  | rs3 |
|_________|  |_________|  |_________|

9.1. Preliminary Setup

1. >From the webmaster, obtain a virtual server IP address. In our example this will be 1.2.3.1. Requests for service at the LVS cluster will be addressed to a fully-qualified domain name associated with this address.

2. Locate 5 servers and designate their roles: 1 primary LVS router, 1 backup LVS router, 3 real servers. The LVS routers must be Linux boxes running Red Hat 6.1 or later. The real servers may be any platform running any operating system and Web server.

Steps 3-7 set up the LVS routers.

3. On each LVS router, install two ethernet adapter cards, eth0 and eth1. Create a public IP interface on eth0 and a private IP interface on eth1. The public interface device (eth0) is the heartbeat device. The virtual server address is aliased to this device.

Primary node  Backup node
eth0  1.2.3.2  1.2.3.3
eth1  192.168.1.1  192.168.1.2

4. Designate an IP address (192.168.1.254) for the router device (eth1) connecting the active LVS router to the private network. This floating IP address will be aliased to the router device as eth1:1, and will be the gateway to the private network and the default route used by each real server to communicate with the active router.

5. On each LVS router: a. Enable packet forwarding. To do this at system boot, make sure the file

/etc/sysconf/network contains the line FORWARD_IPV4=yes. To enable

packet forwarding without rebooting, as root issue this command:

echo "1" > /proc/sys/net/ipv4/ip_forward

b. Enable packet defragmenting. To do this at system boot, make sure the

file /etc/sysconf/network contains the line DEFRAG_IPV4=yes. To enable

packet defragmenting without rebooting, as root issue this command:

echo "1" > /proc/sys/net/ipv4/ip_always_defrag

c. Masquerade the private network. Issue this command and put it in

/etc/rc.d/rc.local:

ipchains -A forward -j MASQ -s 192.168.1.0/24 -d 0.0.0.0

6. Decide whether to use the rsh or ssh families for synchronizing LVS cluster files. Verify that your choice is installed such that the LVS routers can log in to one another as root without administrator intervention (see Table 2). In this example, we will choose rsh.

7. On each LVS router, verify that the LVS cluster software is installed (see Section 7.

Steps 8-11 set up real servers.

8. On each real server, install an ethernet network card, eth0, create an IP address on the same private subnet as in Step 3, and assign a weight to each server indicating its processing capacity relative to that of the others. In this example, rs1 has twice the capacity (two processors) of rs2 and rs3.

rs1  rs2  rs3
eth0  192.168.1.3  192.168.1.4  192.168.1.5
weight  2000  1000  1000

9. On each real server, verify that the address named in Step 4 (192.168.1.254)

is its default route for communicating with the active LVS router.

10. Decide which program (uptime, ruptime, rup) will be used by the active router to monitor the workload on the real servers. If you choose uptime, each LVS router must be able to connect with each real server without administrator intervention, using the tool family you selected in Step 6. See Table 2 for general enablement instructions. If the selected tool cannot be enabled (for example, one of the real servers is an NT box), the scheduling algorithms that use dynamic load information will still work but the user-assigned weights will be statically applied rather than dynamically adjusted based on load.

11. Verify that each real server runs an installed and configured httpd server. Note that the real servers must listen on the same port (80 in the example) as the corresponding virtual server.

12. Verify (for example, using telnet or ping) that each real server can reach hosts on the public LAN. If a real server on the private network cannot reach a host on your LAN, this probably indicates a communication failure between the server and the active router. See Steps 8 and 9.

13. Determine the runtime parameters. For some of these, you may need to experiment over time to obtain optimal values. In this example, we will use the values listed.

Value

Parameter Description

1050 Number of the heartbeat listening port on the primary and backup routers.

2 Number of seconds between heartbeats.

10 Number of seconds to wait for a non-responding router to respond before initiating failover.

Value

Parameter Description

10 Number of seconds to wait for a non-responding real server to respond before removing it from the routing table.

180 When a real server that has been removed from the routing table starts responding again, wait this number of seconds before re-adding the server to the routing table.

wlc Use the Weighted least-connections load-balancing algorithm (assign more jobs to servers that are least busy relative to their load-adjusted weight). See Table 1 for a description of the choices.

http

Application. The alternative is ftp.

80 Virtual server port number. The listening port selected for the virtual server is used on the real servers as well.

Now we are ready to implement the example. You can do this from a shell as explained in the next section. Or you can use the GUI configuration tool as explained in Section 9.3.

9.2. Implementing the Example from the Shell

1. With your favorite editor, open /etc/lvs.cf and set the values shown below. The number on the right is a link to the step in Section 9.1 that discusses this setting.

# Global section primary = 1.2.3.2 3 backup = 1.2.3.3 3 keepalive = 2 13 deadtime = 10 13 heartbeat_port = 1050 13 rsh_command = rsh 6 network = nat nat_router = 192.168.1.254 eth1:1 4 # Per-virtual-server section virtual server vs1 { 8 address = 1.2.3.1 1 active = 1 load_monitor = ruptime 10 timeout = 10 13 reentry = 180 13 port = 80 13 scheduler = wlc 13 # Per-real-server section server rs1 { 8 address = 192.168.1.3 8 active = 1 weight = 2000 8 } server rs2 { 8 address = 192.168.1.4 8 active = 1 weight = 1000 8 } server rs3 { 8 address = 192.168.1.5 8 active = 1 weight = 1000 8 } }

2. Copy the edited configuration file to the backup router.

3. On the primary router, start the pulse daemon with this command:

/etc/rc.d/init.d/pulse start

4. Start pulse on the backup router.

9.3. Implementing the Example with Piranha

1. On the primary LVS router, as root, type piranha & to start the GUI configuration tool.

2. Click the Global Settings tab and enter the values shown:

Field  Enter:  See Step:

Primary LVS server IP 1.2.3.2

3

NAT Router IP 192.168.1.254

4

NAT Router Device eth1:1

4

Sync tool  rsh  6

3. Click the Redundancy tab and enter the values shown:

Field  Enter:  See Step:

Enable Redundant server (select)

Field  Enter:  See Step:

Redundant LVS server IP 1.2.3.3

3

Heartbeat interval 2

13

Assume dead after 10

13

Heartbeat port 1050

13

4. Click the Virtual Servers tab. On the tab, click Add, then Edit, and enter the values shown:

Field  Enter:  See Step:
Name  vs1
Application  http  13
Port  80  13
Address  1.2.3.1  1
Device  eth0:1  3

Re-entry time 180

13

Service timeout 10

13

Load monitoring tool ruptime

10

Scheduling weighted least-connections

13

5. Click the Real Servers tab. On the tab, click Add, then Edit, and enter the values shown:

Field  Enter:  See Step:
Name  rs1  8
Address  192.168.1.3  8
Weight  2000  8

6. Repeat Step 5 for rs2 (192.168.1.4) and rs3 (192.168.1.5). For them, enter a weight of 1000, which will indicate that rs1 has twice the computing capacity of rs2 and rs3.

7. Click Close to return to the Real Servers tab. There, select each real server and click Activate to activate it.

8. Click Close to return to the Virtual Servers tab. There, select the vs1 virtual server and click Activate to activate it.

9. Click the Controls/Monitoring tab and:

a. Click the Start button (name changes to Stop).

b. If you check "Add pulse daemon to this runlevel" the cluster will be started at boot time and, on startup of piranha, the button above will usually be labeled Stop.

10. Log in to the backup router and start pulse with this command:

/etc/rc.d/init.d/pulse start

If you enter the values exactly as shown above, piranha generates at the end of the configuration file the structures shown below for this virtual server and its real servers.

network = nat nat_router = 192.168.1.254 eth1:1

virtual vs1 { address = 1.2.3.1 eth0:1 active = 1 scheduler = wlc load_monitor = ruptime timeout = 5

server rs1 { address = 192.168.1.3 active = 1 weight = 2000

} server rs2 {

address = 192.168.1.4 active = 1 weight = 1000

} server rs3 {

address = 192.168.1.5 active = 1 weight = 1000

} }