For this example, the following services are present, or will be configured here:

• 192.168.1.34: Syslog server
• 192.168.1.68: Web backend #1
• 192.168.1.69: Web backend #2
• 192.168.1.80: HAProxy host #1 management address
• 192.168.1.81: HAProxy host #2 management address
• 192.168.1.84: Load balancer “front end” – the IP address users will ultimately connect to, managed by Keepalived
• 192.168.1.119: SMTP server

Note that all interfaces are on the same subnet (192.168.1.0/24); the load balancer communicates with the backends on the same interface it uses for client connections.

HAProxy

To install HAProxy on Ubuntu, simply install the “haproxy” package; as described above, this example uses two hosts with IP addresses of 192.168.1.80 and 192.168.1.81. Set “ENABLED=1″ in /etc/default/haproxy to have the init script that comes with the package start HAProxy. HAProxy’s configuration lives at /etc/haproxy/haproxy.cfg:

# this config needs haproxy-1.1.28 or haproxy-1.2.1

global
        # log 127.0.0.1 local0
        # log 127.0.0.1 local1 notice
        log 192.168.1.34 local0
        user haproxy
        group haproxy
        daemon
        maxconn 20000

defaults
        log global
        option dontlognull
        balance leastconn
        clitimeout 60000
        srvtimeout 60000
        contimeout 5000
        retries 3
        option redispatch

listen stats 192.168.1.80:80 # or 192.168.1.81:80 for second HAProxy host
        mode http
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth admin:supersecret

listen http 192.168.1.84:80
        mode tcp
        option tcplog
        balance source
        maxconn 10000
        server web01 192.168.1.68:80 maxconn 5000
        server web02 192.168.1.69:80 maxconn 5000

 listen https 192.168.1.84:443
        mode tcp
        option tcplog
        balance roundrobin
        maxconn 10000
        server web01 192.168.1.68:443 maxconn 5000
        server web02 192.168.1.69:443 maxconn 5000

Config Notes
06. Send log messages to the syslog server at 192.168.1.34 using the local0 facility
22. Only stats are provided on this interface – eth0 on the host.
27. Replace “supersecret” with your password for the statistics interface.
29. HTTP listener – 192.168.1.84 is the IP address Keepalived will manage.
30. We are passing HTTP through – not terminating it on – the HAProxy host.
31. In case you’re wondering why we’re not using Keepalived alone, given that we’re not terminating HTTP at HAProxy: The log format options available with HAProxy. (This also gives us the option of changing to HTTP termination in HAProxy later, without installing new software, of course.)
32. Balance based on source address.
34-35. Two HTTP backends.
37. HTTPS listener, again on the Keepalived IP address.
38. HTTPS is also passed through to the web back-ends.
39. See 31 above.
40. Use a round-robin balancing algorithm.
42-43. Two HTTPS backends.

Start HAProxy as follows:

# service haproxy start

Keepalived

Keepalived is installed via the Ubuntu “keepalived” package, intuitively enough. The following entry needs to be made in /etc/sysctl.conf:

net.ipv4.ip_nonlocal_bind=1

(net.ipv4.ip_forward does not need to be set to “1″ for this type of configuration.) Run “sysctl -p” or reboot to apply the sysctl setting.

The keepalived file – /etc/keepalived/keepalived.conf – contains:

global_defs {
    notification_email {
        sysadmin@example.com
    }
    notification_email_from keepalived@haproxy01.example.com
    smtp_server 192.168.1.119
    smtp_connect_timeout 30
}

vrrp_script chk_haproxy { # Requires keepalived-1.1.13
    script "killall -0 haproxy" # widely used idiom
    interval 2 # check every 2 seconds
    weight 2 # add 2 points of prio if OK
}

vrrp_instance VI_1 {
    interface eth0
    state MASTER # or "BACKUP" on backup
    priority 101 # 101 on master, 100 on backup
    virtual_router_id 51

    smtp_alert # Activate SMTP notifications

    authentication {
        auth_type AH
        auth_pass supersecret
    }

    virtual_ipaddress {
        192.168.1.84
    }

    track_script {
        chk_haproxy
    }
}

Config Notes
02-07. Configuration for email notifications on master/backup state changes.
10-14. Script to check if HAProxy is still alive. (“killall -0” is a common idiom seen in Keepalived configurations; I’m curious to know the original source.)
18-19. Configure master (or backup).
22. Send SMTP notifications on master/backup state change for this interface.
26. Set authentication password to something better than “supersecret” here.
30. IP address of the virtual interface – in this case, the IP address in front of the load balancer.
34. Use the chk_haproxy script defined above to check whether to fail over this interface or not.

Start the keepalived daemon:

service keepalived start

Acknowledgement: My co-worker Will Winslow helped with much of the research for this post.
Reference: HAProxy Architecture Guide

From the f.lux website: “f.lux makes your computer screen look like the room you’re in, all the time. When the sun sets, it makes your computer look like your indoor lights. In the morning, it makes things look like sunlight again.” For me, the decrease in eye strain is palpable; at night, picking up a mobile phone or even looking at a TV after using a machine with f.lux installed is painful. (Obviously, if you need accurate color representation, you can’t have f.lux on; thankfully, temporarily disabling it is a breeze.)

There are are Windows, OS X and Linux versions available; installation on Ubuntu is made easy through a PPA. Here’s a little Puppet module for Ubuntu Maverick hosts, if Puppet is your thing:

class flux {

  exec { "add f.lux ppa":
    command => "/usr/bin/apt-add-repository ppa:kilian/f.lux",
    creates => "/etc/apt/sources.list.d/kilian-f_lux-maverick.list",
    notify => Exec["apt-get update"],
  }

  package { "fluxgui":
    ensure => present,
    require => Exec["add f.lux ppa"],
  }

}

> ack --php phpinfo
utilities.php
141:	   Gather phpinfo into a string variable - This has to be done before
147:	phpinfo(INFO_MODULES);

phpinfo.php
1:<?php phpinfo(); ?>

I have a fork of ack on Github – a very trivial fork, adding only support for Puppet (.pp) files. (I did submit a pull request to the author, but he quite reasonably rejected it as he is focused on version two, which will have new features for handling file types.) (Update: See comment below from ack’s creator on a better way to do this.)

Autojump is “a cd command that learns.” As in, with minimal input, it picks the right directory to jump to based on your previous actions:

> pwd
/tmp
> j git
/haiku/home/anl/git
> pwd
/haiku/home/anl/git

“jumpstat” tells you the relative ranking of directories in the autojump database:

> jumpstat
1.0:	/haiku/home/anl/git/hrc-puppet/modules/puppet
2.0:	/haiku/photos/andy/201012
2.0:	/cloud/leonard/photos/201012
2.0:	/opt
2.0:	/haiku/home/anl/git/hrc-puppet/modules/apt
2.0:	/home/anl/Desktop
[...]

Tab completion works, too; typing “j user” and three tabs yields the following for me:

> j user__
user__1__/haiku/home/anl/git/hrc-puppet/modules/users
user__2__/haiku/home/anl/git/ext/puppet/modules/users

Installation is straightforward, with an install script and manual instructions. To scratch my own particular itch, I have a fork on Github that adds an install script for non-root users. (I haven’t submitted a pull request, and I doubt I will, as I don’t think my code is particularly useful to most folks; it also isn’t particularly polished.)

Despite its many desirable features and continuing popularity, Drupal is not without its shortcomings, as many readers are also likely aware. Although Drupal has an active and responsive security team, the software has a long track record of requiring frequent security patches – Secunia has seven 2009 advisories for Drupal 6.x listed as of this writing. Although by its nature an apples-to-oranges comparison, this ranks Drupal behind similarly large and complex PHP projects such as WordPress 2.x (5) and Gallery 2.x (0) – and the number for Drupal does not include dozens of additional advisories for Drupal modules. Further, Drupal has struggled and lagged with support for PHP 5.3.x, suggesting to this outside observer that the project is having difficulties maintaining its codebase.

All that being said, I do not personally believe that the above issues rule out using Drupal; the benefits outweigh the shortcomings. So, assuming the question is not whether to deploy Drupal, but how to do so most securely and efficiently, my recommendations from a systems administration perspective are below.

Goals

The recommendations described stem from three goals for a Drupal installation:

1. Security – Avoid compromise of your site and system.
2. Flexibility – Create an environment that allows for easy modification of the OS, server and application, and provides for straightforward roll-back of those changes should those need arise.
3. Stability and Security over Performance – Although not mutually exclusive, designing for stability and security can entail trade-offs that may impact performance. I don’t directly address performance in these recommendations.

The below recommendations are written assuming that the sysadmin (you) responsible for the Drupal server and the developer are two different people. If you wear both hats, the recommendations can obviously adapted to fit your dual role.

Recommendations

1. Insist on the latest version of Drupal, and plan to upgrade as Drupal releases come out. Some developers may be hesitant to use the latest (stable) version of Drupal, but Drupal’s security track record dictates this; anything else is not serving the site owner’s interests.
2. Use the latest supported version of PHP. Drupal’s PHP version requirements are more brittle than most applications; the latest version of PHP is almost always the most stable and secure version. Combining the two is obvious: Aggressively track latest version of PHP that Drupal supports. Additionally, use the latest version of your preferred web server and patch as it is updated. Note that this implies a source-based installation of PHP and your web server instead of a package-based installation from your distribution. Carefully consider your build and deployment strategies with an eye towards reproducibility, documentation and ease of roll-back should a problem arise.
3. Isolate Drupal from other applications. Run Drupal on its own system – physical or virtual – and in doing so, reduce operating complexity while limiting collateral damage if your Drupal instance should be compromised. Depending on your backup strategy, don’t overlook the possibility that running a VM may have obvious recovery and forensic advantages over a physical install should your Drupal site get broken into.
4. mod_security – Consider integrating a “web application firewall” such as mod_security directly into your web server. Carefully evaluate the trade-off of additional complexity for enhanced security in your environment.
5. Deploy a host-based firewall for inbound and outbound traffic – This has two functions – prevent your server from being compromised through a hole in a service other than your web server, and limit damage to other systems if/when you are compromised. There’s a decent chance that the only external service you need exposed other than HTTP for your Drupal site is SSH for remote management – and you can probably lock that down to a very limited set of IP addresses. And, in the situation where your server is compromised, outbound rules can reduce the ability of your server to attack other machines: Consider, for example, the effect of an outbound rule on port 25 if an attacker attempts to use your compromised server as a spam bot.
6. Use SELinux – Security-Enhanced Linux provides a the ability to audit and possibly deny actions on your system through mandatory access control policies. You will likely want to run SELinux in “permissive” mode before your site is publicly available, at which point you would switch to “enforcing” mode; “audit2allow” and “audit2why” can be very helpful tools when developing policies. See “man selinux” for more information.
7. Use a Sysadmin Staging Server – The site developer likely has a staging instance of Drupal for testing out their code changes; deploy a similar environment for testing Sysadmin changes, such as PHP updates or the latest version of Drupal with your code. Consider using a software testing framework such as Selenium to automate the tests you run on the sysadmin staging site.