Network Load Balancing 2003

Providing high-availability for Shibboleth Identity Provider and Central Authentication System Authors: Brusten Philip & Van der Velpen Jan Last modified: Friday, 09-Jun-2006 15:25:18 CEST Table of contents

•

Introduction

•

Problem space

•

Solution

•

Implementation

o

Setup cluster hosts

o

Managing Network Load Balancing

Introduction On this page we will first describe the problem space of the current setup (Shibboleth Identity Provider and CAS (Central Authentication System)). Afterwards we give a solution to this problem followed by the actual implementation. Problem space With the new Authentication and Authorisation Infrastructure we have centralised all authentication traffic to one machine and this 24 hours a day, 7 days a week! In our other documentation (http://shib.kuleuven.be/docs/idp/install-idp-1.3.shtml) we suggested a configuration of both the Shibboleth Identity Provider (IdP) and the Central Authentication Service (CAS) on the same server. We have illustrated this configuration below:

The host idp.example.be (ip: 134.58.1.1) has two different services running on his system: IdP and CAS. All authentication requests will be handled by this host. Because all authentication will be centralised on this one host, we must provide high availability so we can guarantee uptime to the applications using authentication. Problems like network, hardware & infrastructure problems need to be solved! Solution We need to provide high availability for both IdP and CAS. High availability is the ability to provide user access to the IdP and CAS for a high percentage of time by attempting to reduce unscheduled outages, to decrease the impact of scheduled downtime of IdP and CAS.

To provide this high availability we make an identical system in failover. Failover means that this second system can take over when the primary system failes, due to problems or maintenance operations. This situation is illustrated below:

Both servers are the same as for functionality because they are identical to each other. One disadvantage of this setup is that the client's data on one system will not be shared with the other system. One could setup replication of client's data, but that is a very complex operation. In case of failover, the services will still be available but the client's state will be lost. Network Load Balancing of Microsoft® offers a possible solution called affinity, this will bind a client to one of the servers based on his ip address. When a client has multiple connections to one of these servers he will always return to the same server, that way the client's data will be preserved. This solution isn't possible for the functionality of the IdP, because there will be a callback from the SP's to the IdP, and the client will be the SP. There is no way to couple the client's address to the SP's address. Another idea is trivial: in this case there are two completely separate components: IdP and CAS. Instead of putting all the load on one machine and let the other one do nothing, why not split up those two components? Unfortunately also this causes problems. The CAS-client -which does authentication for the IdP servlet- does an out-of-band callback to CAS-server. When using NLB, a callback from IdP to CAS will be serviced on the SAME machine. This means the CAS-client callback will be handled by the CASserver that should be in standby. Off course this operation will fail all the time. Replicating state of CAS-server might be a solution here. Another solution to this would be to somehow force the callback to be handled by the NLB mechanism, and thus the active CAS-server. The conclusion we take from all this is that we cannot balance the load equally over both servers, unless this we share or replicate all client's data. The best solution is a hot-standby setup where one system takes over when the other one has problems or is being maintained.

Implementation We can achieve the failover setup on a server running Microsoft Windows Server 2003® with Network Load Balancing. We based this document on http://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/TechRef/1611cae3-58654897-a186-7e6ebd8855cb.mspx This document contains all information applicable to our setup of Network Load Balancing, further details can be found in the document mentioned above. We assume you have setup two servers with both IdP and CAS, as described in http://shib.kuleuven.be/docs/idp/install-idp-1.3windows2003.shtml. In our example we called both servers idp.example.be, the first server has a unique (intern) ip address 192.168.0.1, the second has a unique (intern) ip address 192.168.0.2. All configurations need to be exactly the same, so you could setup the first system, then take an image of the first an put in on the second one (only possible if they are exactly the same). Setup cluster hosts We will run over some steps to activate NLB on each host. You can first do this for the first server and afterwards for the second one. We've taken screenshots of the configuration of both servers. The screenshots of the first server are located on the left, the other, server two specific screenshots, are located on the right. The first step will be to verify the ip address you set up for the server. Go to the properties of your network interface:

Go to the properties of the Internet Protocol (TCP/IP)

Check the static ip address for the server. This ip address will be used later as dedicated ip address. In normal situations this ip address wont be available from the outside.

Activate Network Load Balancing and go to the properties:

Now we will configure the Cluster Parameters. This configuration is the same for both servers. The ip address you configure here must be added to the list of ip addresses of the Internet Protocol (TCP/IP). You will have to choose your cluster operation mode, the possibility's are unicast or multicast. Both possibility's have advantages and disadvantages.

•

Unicast mode The unicast mode changes the cluster adapter's MAC address to the cluster MAC address. This cluster address is the same MAC address that is used on all cluster hosts. When this change is made, clients can no longer address the cluster adapters by their original MAC addresses.

•

Multicast mode When using multicast mode, NLB will add a multicast MAC access to the cluster adapters on all of the cluster hosts. At the same time, the cluster adapters retain their original MAC addresses. This way each host could be addressed individually

The Network Load Balancing driver does not support a mixed unicast an multicast environment. All cluster hosts must be either multicast or unicast; otherwise, the cluster will not function properly. If clients are accessing a Network Load Balancing cluster through a router when the cluster has been configured to operate in multicast mode, be sure that the router meets the following requirements:

•

Accepts an Address Resolution Protocol (ARP) reply that has one MAC address in the payload of the ARP structure but appears to arrive from a station with another MAC address, as judged by the Ethernet header.

•

Accepts an ARP reply that has a multicast MAC address in the payload of the ARP structure.

If your router does not meet these requirements, you can create a static ARP entry in the router. For example, some routers require a static ARP entry because they do not support the resolution of unicast IP addresses to multicast MAC addresses. You can also enable Internet Group Management Protocol (IGMP) support on the cluster hosts to control switch flooding when operating in multicast mode. IGMP is a protocol used by Internet Protocol version 4 (IPv4) hosts to report their multicast group memberships to any immediately neighboring multicast routers. When you enable remote control you can manage all NLB configuration on one host with the NLB manager (this will be covered later). When you enable remote control, you must provide a identical password on both cluster hosts.

The previous tab is identical for both cluster hosts, the tab Host parameter is unique for each cluster host. Each host must have its dedicated ip address, as mentioned before. With the unique host identifier you can specify which cluster host will be the default host, when no rules apply to the request. You also need to specify which is the initial host state.

You can now setup the port rules for the requests. You must keep in mind that you have to setup an equal amount of port rules on each cluster host. These rules must also match in port range, protocol and filtering mode. In the setup that we propose, it is not necessary to setup port rules. The priority is defined in the "Host Parameters" tab.

When you press "Ok", you will receive the following notice:

You need to add the cluster ip address to Internet Protocol (TCP/IP) stack. Go to the properties of Internet Protocol (TCP/IP), next press the Advanced button.

Add the cluster ip 134.58.1.1 to the Ip addresses list. You must do this for both cluster hosts.

The implementation of NLB is now finished. Managing Network Load Balancing When you activated remote control on the Cluster Parameters, you can make use of the Network Load Balancing Manager. The NLB Manager can be started from command line: Nlbmgr.exe or by going to the Administrative tools in the Start menu. Network Load Balancing Manager is used to create and manage Network Load Balancing clusters and all cluster hosts from a single computer, and you can also replicate the cluster configuration to other hosts. By centralizing administration tasks, Network Load Balancing Manager helps eliminate many common configuration errors. Microsoft, Windows, Windows Server 2003 are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

Interview Quotations: Network Load Balancing is a clustering technology offered by Microsoft as part of all Windows Server 2000 and Windows Server 2003 family operating systems. Network Load Balancing uses adistributed algorithm to load balance network traffic across the number of hosts, helping to enhance the scalability and availability of IP-based services, such as Web, Virtual Private Networking, Streaming Media, Terminal Services, Proxy, etc. It also provides high availability by detecting host failures and automatically redistributing traffic to operational hosts.

IT Interview Questions:What are the differences between Windows Clustering, Network Load Balancing and Round

Robin, and scenarios for each use? Cluster technologies are becoming increasingly important to ensure service offerings meet the requirements of the enterprise. Windows 2000 and Windows Server 2003 support three cluster technologies to provide high availability, reliability and scalability. These technologies are: NLB, CLB and Server cluster. These technologies have a specific purpose and are designed to meet different requirements. • Server cluster provides failover support for applications and services that require high availability, scalability and reliability, and is ideally suited for back-end applications and services, such as database servers. Server cluster can use various combinations of active and passive nodes to provide failover support for mission critical applications and services. • NLB provides failover support for IP-based applications and services that require high scalability and availability, and is ideally suited for Web tier and front-end services. NLB clusters can use multiple adapters and different broadcast methods to assist in the load balancing of TCP, UDP and GRE traffic requests. • Component Load Balancing provides dynamic load balancing of middle-tier application components that use COM+ and is ideally suited for application servers. CLB clusters use two clusters. The routing cluster can be configured as a routing list on the front-end Web servers or as separate servers that run Server cluster. Cluster technologies by themselves are not enough to ensure that high availability goals can be met. Multiple physical locations may be necessary to guard against natural disasters and other events that may cause complete service outage. Effective processes and procedures, in addition to good architecture, are the keys to high availability. Round robin is a local balancing mechanism used by DNS servers to share and distribute network resource loads. You can use it to rotate all resource record (RR) types contained in a query answer if multiple RRs are found. By default, DNS uses round robin to rotate the order of RR data returned in query answers where multiple RRs of the same type exist for a queried DNS domain name. This feature provides a simple method for load balancing client use of Web servers and other frequently ueried multihomed computers. If round robin is disabled for a DNS server, the order of the response for these queries is based on a static ordering of RRs in the answer list as they are stored in the zone (either its zone file or Active Directory). Applies To: Windows Server 2003 with SP1 Q. What is Network Load Balancing? A. Network Load Balancing is a clustering technology offered by Microsoft as part of all Windows 2000 Server and Windows Server 2003 family operating systems. NLB uses a distributed algorithm to load balance network traffic across a number of hosts, helping to enhance the scalability and availability of mission critical, IP-based services, such as Web, Virtual Private Networking, Streaming Media, Terminal Services, Proxy, etc. It also provides high availability by detecting host failures and automatically redistributing traffic to operational hosts. Q. What is a Cluster? A. A cluster is a group of independent computers that work together to run a common set of applications and provide the image of a single system to the client and application. The computers are physically connected by cables and programmatically connected by cluster software. These connections allow computers to use problemsolving features such as failover in Server clusters and load balancing in Network Load Balancing (NLB) clusters. What is the Difference Between NLB and Server Clusters? A. A server cluster (MSCS) is a collection of servers that together provide a single, highly available platform for hosting applications. Applications can be failed over to ensure high availability in the event of planned downtime due to maintenance or unplanned downtime due to hardware, Operating System or application failures. Server clusters provide a highly available platform for applications such as SQL Server, Exchange Server data stores, file and print servers, etc. Server clusters are used for stateful applications that rely on some state context from one request to the next.

Network Load Balancing (NLB) clusters dynamically distribute the flow of incoming TCP and UDP traffic among the cluster nodes according to a set of traffic-handling rules. NLB clusters provide a highly available and scalable platform for applications such as IIS, ISA server, etc. NLB is used for stateless applications; i.e. those that do not build any state as a result of a request. NLB and server clusters compliment each other in complex architectures: NLB is used for load balancing requests between front-end web servers while server clusters provide high availability for backend database access. NLB and server clusters cannot be used on the same set of servers (see Question Can I use NLB and server clusters on the same set of servers?). Q. Can I Use NLB and Server Clusters on the Same Set of Servers? A. No, Microsoft Server clusters (MSCS) and Network Load Balancing (NLB) are not supported on the same set of nodes. Both Server clusters and Network Load Balancing clusters control and configure network adapters. Since they are not aware of each other, configuring one can interfere with the other. How Large Should My Cluster Be? A. The size of the cluster is determined by the application that is being hosted and load balanced and on the system resources on the host machines available to service client requests sent to the application. If you notice that client requests are slowing down as more and more clients make connection requests to the application, it may be time to add more hosts to the cluster. A single NLB cluster can have up to 32 hosts in it. For more information and examples of how to determine the size of your cluster refer to Chapter 10 Designing Network Load Balancing of the Windows Server 2003 Deployment Guide, (http://go.microsoft.com/fwlink/? LinkId=4298). Q. Are There Any Performance Concerns as My Cluster Grows? A. Tests have shown the NLB performance begins to deviate from the linear as the cluster grows beyond 20 to 25 nodes. But, this really depends on the amount of traffic that the application sees in the form of client requests. The following graphs give a better idea of NLB performance as the cluster size and number of users increase.

Q. We Expect Our Front-End Cluster to Grow Beyond 32 Nodes. Should I Use NLB Even Though it Has a Cluster Size Limit of 32 nodes? A. NLB can be used to scale beyond 32 machines by using Round Robin DNS between multiple NLB Clusters. This is shown in the following figure.

How Does NLB Detect a Server Failure? A. While servicing client requests, each NLB Cluster host emits heartbeats to the other hosts in the cluster. If a host fails and stops emitting heartbeats, then after a default time period of five seconds, the remaining hosts in the cluster undergo a process called convergence to remove the failed host from the cluster and have new client connection requests mapped to remaining hosts in the cluster. So, new connections made by the client get handled by one of the remaining hosts in the cluster. Q. How Long Does it Take For a Failed Server to be Removed From the Cluster? A. By default, five seconds are required to detect a failed host. Once a failure is detected, the convergence process takes an additional two and a half to three seconds to evict the failed host and redistribute its load to the surviving hosts. Applies To: Windows Server 2003 with SP1 Q. Do the Heartbeat Packets Consume a Lot of Bandwidth? A. No, the heartbeat packets, which are emitted every second by each host, consume less than 1,500 bytes. Q. Do the Heartbeats Need to Go on a Back-End Network? A. No, the heartbeat packets are always sent out on the same network interface on which data packets are received and sent. There is no need for an additional back-end network for the control (heartbeat) packets. In fact, in order to be able to detect connectivity failures on the load-balanced network adapter, NLB actually requiresthat heartbeats traverse the load-balanced network adapter. Q. Is NLB a Kernel Component? A. Yes, NLB has a kernel component called wlbs.sys. This is an intermediate NDIS driver. NLB also has user-mode components for management purposes. Q. What are the Benefits of NLB Over Simple Round Robin Domain Name Service (RRDNS)? A.

• Automatic recovery within 5 seconds • More even load balancing Q. How Does the NLB Load Balancing Algorithm Work?

A. NLB employs a fully distributed filtering algorithm to map incoming clients to the cluster hosts. This algorithm enables cluster hosts to independently and quickly make a load balancing decision for each incoming packet. It is optimized to deliver statistically even load balance for a large client population making numerous, relatively small requests, such as those typically made to Web servers. When the client population is small and/or the client connections produce widely varying loads on the server, the load-balancing algorithm is less effective. However, the simplicity and speed of NLBs algorithm allows it to deliver very high performance, including both high throughput and low response time, in a wide range of useful client/server applications. If No Affinity is set, NLB load balances incoming client requests so as to direct a selected percentage of new requests to each cluster host; the load percentage for each host is set in the NLB Properties dialog for each port range to be load balanced. The algorithm does not dynamically respond to changes in the load on each cluster host (such as the CPU load or memory usage). However, the load distribution is modified when the cluster membership changes, and load percentages are renormalized accordingly. When inspecting an arriving packet, all hosts simultaneously perform a mapping to quickly determine which host should handle the packet. The mapping uses a randomization function that calculates a host priority based on their IP address, port, and other information. The corresponding host forwards the packet up the network stack to TCP/IP, and the other cluster hosts discard it. The mapping remains unchanged unless the membership of cluster hosts changes, ensuring that a given clients IP address and port will always map to the same cluster host. However, the particular cluster host to which the clients IP address and port map cannot be predetermined since the randomization function takes into account the current and past clusters membership to minimize remappings. In general, the quality of load balance is statistically determined by the number of clients making requests. This behavior is analogous to dice throws where the number of cluster hosts determines the number of sides of a die, and the number of client requests corresponds to the number of throws. The load distribution improves with the number of client requests just as the fraction of throws of an N-sided die resulting in a given face approaches 1/N with an increasing number of throws. As a rule of thumb, with client affinity set, there must be at least five times more clients than cluster hosts to begin to observe even load balance. The Network Load Balancing client affinity settings are implemented by modifying the statistical mapping algorithms input data. When client affinity is selected in the NLB Properties dialog, the clients port information is not used as part of the mapping. Hence, all requests from the same client always map to the same host within the cluster. Note that this constraint has no timeout value and persists until there is a change in cluster membership. When single affinity is selected, the mapping algorithm uses the clients full IP address. However, when Class C affinity is selected, the algorithm uses only the Class C portion (upper 24 bits) of the clients IP address. This ensures that all clients within the same Class C address space map to the same cluster host. Q. How Does NLB Cluster Convergence Work? A. Convergence involves computing a new cluster membership list and recalculating the statistical mapping of client requests to the cluster hosts. There are two instances in which cluster traffic has to be remapped due to a change in cluster membership: when a host leaves the cluster and when a host joins the cluster. A convergence can also be initiated when several other events take place on a cluster, such as changing the load balancing weight on a host or implementing port rule changes. Removing a Member. Two situations cause a host to leave the cluster or go offline. First, the host can fail, an event that is detected by the NLB heartbeat. Second, a system administrator can explicitly remove a host out of the load-balancing cluster or stop NLB on that host. The NLB heartbeat. NLB uses a heartbeat mechanism to determine the state of the hosts that are load balanced. This message is an Ethernet-level broadcast that goes to every load-balanced cluster host. NLB assumes that a host is functioning normally within a cluster as long as it participates in the normal exchange of heartbeat messages between it and the other hosts. If the other hosts do not receive a message from a host for several periods of heartbeat exchange, they initiate convergence. The number of missed messages required to initiate convergence is set to five by default (but can be changed). During convergence, NLB reduces the heartbeat period by one-half to expedite completion of the convergence process. Server Failure. When a cluster host fails, the client sessions associated with the host are dropped.

After convergence occurs, client connections to the failed host are remapped among the remaining cluster hosts, who are unaffected by the failure and continue to satisfy existing client requests during convergence. Convergence ends when all the hosts report a consistent view of the cluster membership and distribution map for several heartbeat periods. Q. Can NLB Balance Load Based on CPU/Memory Usage? A. No, NLB does not respond to changes in the server load (such as CPU usage or memory utilization) or the health of an application. 1.

How many nodes supported failover cluster

The Windows Server Failover Clustering feature supports a certain maximum number of servers in a cluster. Servers in a Failover Cluster are also called nodes. The maximum number of nodes varies depending on the version of the operating system. Windows Server 2000 supported Windows Server 2003, Enterprise Edition Windows Server 2003, Enterprise x64 Edition Windows Server 2003, Enterprise Edition for Itanium-based Systems Windows Server 2003, Datacenter Edition Windows Server 2008, Enterprise x64 Edition Windows Server 2008, Datacenter x64 Edition Windows Server 2008 R2 Enterprise Windows Server 2008 R2 Datacenter Microsoft Hyper-V Server 2008 R2

1-4 Supported

1-8

1-16