Geek(Wisdom).com

I was recently tasked with increasing the up time of my employer's main Web site. The site uses a content management system that lives on two Windows/IIS servers. (I know, the system was purchased before I was hired.) One server is for making changes to content (design-time server) and the other is the public web site (run-time server). The design-time server has a complete copy of the site which is replicated to the run-time server. Unfortunately the run-time server has a habit of refusing to serve pages at the most inopportune times, usually when I'm on vacation or somewhere without a computer.

I decided to place a load balancer in front of the two servers. Not really for load balancing but for failover. This way when the run-time server decides to act up connections will automatically be redirected to the design-time server until the run-time servers tantrum is finished (which is usually after a a spanking [read reboot]). I looked at a number of different tools for handling the load balancing/failover process: pen, pound, crossroads, LVS and HAProxy. All are very capable. Ultimately I chose to use HAProxy.

I configured a Linux machine and installed Apache, HAProxy, and SEC (simple event correlator). I configured Apache to listen on a high number port and respond only to connections from localhost. I also set Apache up to redirect all 403 and 404 errors to a single page explaining that a system outage has occurred. I configured HAProxy to listen on port 80 and direct requests to the run-time server, then fail over to the design time server, then fail over to the local Apache server. In the event both web servers are unable to serve requests users will receive an informative message instead of connection or routing errors.

Unfortunately, HAProxy does not provide a notification mechanism. So I configured SEC to watch the HAProxy log and send an e-mail to the appropriate network administration group whenever a web server goes down or comes up.

HAProxy like most good proxies supports the X-Forwarded-For HTTP header. However, IIS does not support it out of the box. The problem, of course, is that when requests come through HAProxy the web servers will record the IP address of the balancer instead of the requesters address. I found IIS X-Forward-For ISAPI Filter which solves this problem very nicely. Once installed it checks all requests for the X-Forwarded-For header and, if present, replaces the value of c-ip with the value of X-Forwarded-For.

We aren't doing any load balancing yet but if we want to it's as simple as putting more web servers on our network and adding them to the HAProxy configuration. HAProxy supports round robin and weighted round robin algorithms. Apparently there's a least connections patch out there as well.

The end result is an automated solution which fails over to secondary servers and keeps administrators notified of the health of the servers. Maybe this will mean I can take a vacation without restarting a web server.