But perhaps there’s good reason for hiding the download away: There are still large gaps in the API. For instance, say you want to change the security mode of a qtree? You’re out of luck. (Makes one wonder how NetApp implements this functionality in OnCommand System Manager – are they eating their own dogfood?)

That said, if you’re willing to venture off the beaten (and supported) path, you can use the undocumented system-cli API call. Here’s how I’m using it in a Python wrapper I’m working on that makes the SDK feel a little bit less like handling thinly-varnished XML:

    def invoke_cli(self, *cli_args):
        """
        Call the unsupported/undocumented system-cli API.

        cli_args, joined with spaces, would represent the command line 
        if executing in the CLI.

        Return the NaElement result of executing the command.
        """

        args = NaElement('args')
        for arg in cli_args:
            args.child_add(NaElement('arg', arg))

        cli = NaElement('system-cli')
        cli.child_add(args)
        out = self.api.invoke_elem(cli)
        if out.results_status() == 'failed':
            raise OntapApiException(out.results_errno(), out.results_reason())
        return out

On lines 11-13, I’m building up an NaElement (itself imported from the eponymous NaElement module) representation of the command line arguments; this is then added as a child to a “system-cli” NaElement. On line 17, I call “NaServer.invoke_elem()” on the “system-cli” NaElement – and from here on out, it’s just like using the ordinary (documented) API. The “cli-output” element in “out” contains the output for you to parse, and “cli-result-value” is a return code.

Here’s hoping newer versions of the API will do away with the need for this workaround.

This pre-commit hook’s structure is based heavily on a Puppet 2.7 pre-commit published elsewhere. Without further ado:

#!/bin/bash

# Adapted from a puppet pre-commit hook at:
# http://blog.snijders-it.nl/2011/12/example-puppet-27-git-pre-commit-script.html
#
# install this as .git/hooks/pre-commit to check DNS zone files
# for errors before committing changes.

rc=0

[ "$SKIP_PRECOMMIT_HOOK" = 1 ] && exit 0

# Make sure we're at top level of repository.
cd $(git rev-parse --show-toplevel)

trap 'rm -rf $tmpdir $tmpfile1' EXIT INT HUP
tmpdir=$(mktemp -d precommitXXXXXX)
tmpfile1=$(mktemp errXXXXXX)

echo "$(basename $0): Validating changes."

# Here we copy files out of the index into a temporary directory. This
# protects us from a the situation in which we have staged an invalid
# zone file using ``git add`` but corrected the changes in the
# working directory. If we checked the files "in place", we would
# fail to detect the errors.

git diff-index --cached --name-only HEAD |
grep '^db\.' |
git checkout-index --stdin --prefix=$tmpdir/

find $tmpdir -type f -name 'db.*' |
while read zonefile; do
    zone=`echo $zonefile | sed -e "s/^${tmpdir}\/db\.\(.*\)$/\1/"`
    named-checkzone -q $zone $zonefile
    # If named-checkzone reports an error, get some output:
    if [ $? -ne 0 ]; then
	named-checkzone $zone $zonefile | sed "s#$tmpdir/##" >> $tmpfile1 2>&1
    fi
done

if [ -s "$tmpfile1" ]; then
echo
echo Error: Zone file problem:
echo ----------------------------
cat $tmpfile1
echo ----------------------------
echo

rc=1
fi

exit $rc

Update: Now available as a Gist.

Step 1. Create a bare Git repository on your DNS server. Using Fabric, you’d do it something like this:

def config_git():

    # Create bare git repo for direct DNS data pushes:
    sudo('/bin/mkdir /srv/bind.git')
    sudo('/bin/chown ubuntu:ubuntu /srv/bind.git')
    with cd('/srv/bind.git'):
        run('/usr/bin/git init --bare .')
    git_post_receive()

(The above assumes an Ubuntu system, where the “ubuntu” user has sudo privileges, such as on EC2; adjust to your environment as needed.)

Step 2. Add a post-receive hook. Notice that “git_post_receive()” in the fabfile snippet above? That function is nothing more than something like this:

def git_post_receive():
    put('git/post-receive', '/srv/bind.git/hooks/post-receive', mode=0755)

“git/post-receive” – which is, not surprisingly, a post-receive Git hook – is in turn something like this:

#!/bin/sh
sudo GIT_WORK_TREE=/etc/bind /usr/bin/git checkout -f
sudo chmod 0440 /etc/bind/rndc.key

(Again, assuming an Ubuntu environment, where BIND lives in /etc/bind.)

Step 3. Add some orchestration. You could tack on an automatic DNS reload to your post-receive hook, but you may prefer to separate control of BIND into distinct functions in your fabfile, perhaps with some tests run before applying changes:

def test():
    test_zones()
    test_conf()

def test_conf():
    run('/usr/sbin/named-checkconf')

def test_zones():
    with cd('/etc/bind'):
        run('for db in db.*; do zone=`/bin/grep SOA $db | /usr/bin/awk \'{ print $1 }\'`; echo ${zone}: $db; /usr/sbin/named-checkzone $zone $db; done')

def reload_bind():
    sudo('/usr/sbin/service bind9 reload')

def restart_bind():
    sudo('/usr/sbin/service bind9 restart')

Downsides. Note that you’re putting your rndc.key file (used to secure rndc) into Git if you put all your config files into your repository. In that case, you’ll obviously want to restrict access to the repository.

Of course, it isn’t hard to imagine an adaptation of this system where the rndc.key file is not stored in Git, but is perhaps put in place by the post-receive hook. (Call this an “exercise for the reader.”)

Alternatives. If you are using DNSSEC, a tool like nsdiff might be a better fit to manage updates to your zones.

Second, let’s get the most common complaint about OpenDNS – one that isn’t going to be discussed here any further – out of the way: Their practice of returning ads on blocked or non-existent sites in your browser, via a bogus A RR of 67.215.65.132 (if you don’t go with one of their paid options). OpenDNS is upfront about doing this, so you can decide if the trade-off is worthwhile before you sign up – and you can quit using them any time you want.

Those two preliminaries covered, here’s a case study of what I think is a serious problem with OpenDNS, plus some thoughts on how they could fix it.

Background: Recently, I helped my wife buy a domain for a personal blog; it turns out OpenDNS has tagged this domain as pornographic. Currently, the domain hosts nothing beyond an empty WordPress blog.

I can think of three possible scenarios that could be in effect here:

1. The site has been compromised. This was my first thought, although I now consider it unlikely. I’ve gone through the site and the logs pretty closely, and have been unable to find anything amiss. I’m willing to leave the possibility open that I missed something, though.
2. The domain used to be pornographic, but is no longer. This also appears unlikely, at least based on archive.org and other sites; the domain name itself also doesn’t exactly suggest a porn site, either, unless there’s some obscure slang I’m unaware of.
3. The OpenDNS classification is incorrect. The domain was accidentally or intentionally mislabeled. It appears most classification – especially for small sites like the one in question – is crowd-sourced, but only a few members of the “crowd” might tag a small site.

It’s unclear which of the above went wrong, but there are several obvious ideas that OpenDNS could implement to address this class of problems:

1. More metadata, please. For my “pornographic” domain, it would have been nice to know when it was tagged as such, what the DNS servers and relevant records were when it was tagged, what a screenshot looked like then (not now), and what the registrar data was. If it was a site compromise, this data would make my finding it that much easier. If the classification is stale, metadata would make this obvious as well.
2. Automatically-triggered classification reviews. If there are substantial changes to a domain’s registrar entry, that should trigger an automatic review of the site – especially so if the domain registration lapses, and another party purchases the domain after a period of time. The volume of users whitelisting a site in the control panel should be another signal of misclassification, although this probably won’t help obscure sites. This is an obvious, simple fix, and perhaps OpenDNS has something like this in place, but I don’t see any signs of it.
3. A better user interface. When I first visited their tagging interface, the thumbnail viewer was broken, and the “flag for review” link was missing. Both later reappeared, apparently functional, although I have no idea if or when a review will actually take place.
4. Better policing of community members. Some of the OpenDNS Community’s most prolific members tag in excess of 1,000 domains on an average day. Many of these tags appear to be of dubious quality, going far beyond anything that could be called a difference of opinion into the realm of flat-out wrong. Poor quality – or malicious – tagging is damaging OpenDNS’s product; I’m surprised they don’t appear more strongly motivated to address this.

For now, I’m going to continue to use OpenDNS’s services at home, although I’m on the lookout for a better product elsewhere. But until I see some signs that they’re addressing the issue of incorrect tags, I can no longer recommend them for professional use, which is too bad; as I said in the first paragraph, there’s a lot to like about OpenDNS.

Update, 1/2/2012: It’s been a couple weeks since I flagged the domain for OpenDNS’s review – and nothing has happened. I think it’s fair to update my conclusion: OpenDNS has a garbage-in, garbage-out problem and does not appear to be invested in the quality of their product; look elsewhere for a content-filtering tool that goes beyond hobbyist quality.

Running a cpu-bound workload (building Perl modules) on an Ubuntu 11.10 t1.micro instance in us-west-2 tonight, I noticed the following curious CPU usage pattern of approximately 15 seconds on, 60 seconds off:

> vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0  38528  29524 370540    0    0    86   423   84  216 12  5 35  4
 1  0      0   6800  30288 388856    0    0  5356    26  660 1433 27 27  6 40
 5  0      0  21752  27624 378088    0    0    30   211  150  159 40 22  0  8
 6  0      0  21256  27636 378104    0    0     0    27    9    7  1  1  0  0
 7  0      0  21256  27644 378108    0    0     0    10    9    9  1  1  0  0
 7  0      0  21256  27652 378112    0    0     0     8    9    9  2  1  0  0
 7  0      0  20256  27652 378228    0    0     0     0    8   13  1  1  0  0
 8  0      0  20016  27660 378072    0    0     0   218   15   29  0  2  0  3
 6  0      0  37884  27672 378048    0    0     0    14    9   11  3  1  0  0
 4  0      0  30808  27684 378048    0    0     0    11    9   10  1  1  0  0
 4  0      0  23740  27692 378056    0    0     0    10    8    8  2  1  0  0
 4  0      0  30676  27692 378104    0    0     0     0   10   10  1  1  0  0
 5  0      0  26220  27700 378064    0    0     0     9    7   14  6  2  0  1
 5  0      0  21012  27712 378120    0    0     0    10    9   10  1  0  0  0
 5  0      0  27336  27720 378064    0    0     0    21   13   10  1  1  0  0
 1  0      0  29444  27732 378064    0    0     0    14  149   97 39 19  0  0
 1  0      0  33420  27744 378084    0    0     6    12  250  166 67 30  0  0
 2  0      0  41108  27756 378100    0    0     0    37  207  148 60 29  0  0
 6  0      0  33668  27768 378068    0    0     0    14    8    9  1  1  0  0
 5  0      0  37008  27780 378068    0    0     0    10   10   15  4  1  0  0
 4  0      0  30808  27788 378072    0    0     0    18   11    9  2  0  0  0
 5  0      0  24360  27796 378092    0    0     0     9    8    7  2  0  0  0
 2  0      0  19896  27796 378140    0    0     0     0    8    9  1  1  0  0
 6  0      0  27584  27804 378152    0    0     0     7    8   12  1  1  0  0
 6  0      0  22864  27812 378148    0    0     0     9   10   12  2  1  0  0
 7  0      0  19136  27820 378152    0    0     0    10    8    9  1  1  0  0
 6  0      0  26096  27828 378148    0    0     0    12   10    7  2  1  0  0
 6  0      0  20640  27828 378156    0    0     0    19   13    8  2  1  0  0
 6  0      0  27956  27836 378156    0    0     0    11    9   12  1  1  0  0
 6  0      0  22864  27844 378156    0    0     0     6    9   12  2  1  0  0
 6  0      0  19020  27844 378156    0    0     0     1    9    9  1  1  0  0
 2  0      0  46896  21504 368588    0    0   518    18  261  291 47 29  1  7
 1  0      0  35372  21692 368788    0    0     0    43  253  174 65 32  0  0
 1  0      0  43060  21796 368600    0    0     0    62  149  112 66 32  0  1
 5  0      0  38100  21808 368600    0    0     0    46   11   10  1  1  0  0
 5  0      0  45788  21816 368592    0    0     0     7    8   12  2  1  0  0
 7  0      0  38464  21816 368600    0    0     0     0    7    8  2  1  0  0
 7  0      0  45912  21824 368596    0    0     0    11    9    9  2  1  0  0
 7  0      0  39216  21832 368600    0    0     0     7    9    8  1  0  0  0
 4  0      0  35496  21840 368596    0    0     0    19   11    9  4  1  0  0
 5  0      0  43060  21848 368600    0    0     0    29   10   10  2  1  0  0
 5  0      0  37480  21856 368592    0    0     0    11    9   10  1  1  0  0
 5  0      0  45044  21864 368596    0    0     0     7    9   10  1  1  0  0
 5  0      0  38340  21872 368600    0    0     0     8    8    8  2  1  0  0
 4  0      0  46284  21880 368596    0    0     0    10   10   11  1  1  0  0
 6  0      0  38836  21888 368592    0    0     0     8    8    8  2  1  0  0
 1  0      0  38340  21888 368544    0    0     0    15   53   41 12  7  0  0
 1  0      0  40828  21900 368568    0    0     2    46  255  218 66 33  0  0
 1  0      0  39960  21912 368608    0    0     0    26  237  153 63 28  0  0
 3  0      0  50632  21924 368540    0    0     0    16   58   44 32 15  0  0
 4  0      0  46284  21932 368540    0    0     0     7    8   11  1  1  0  0
 4  0      0  45400  21940 368540    0    0     0     6    9   10  1  1  0  0
 5  0      0  45292  21948 368552    0    0     0    11    8   14  0  1  0  0
 6  0      0  37720  21948 368584    0    0     0    17   12    6  2  1  0  0

Apparently, the “small amount of consistent CPU resources” is about 3% of the CPU.

Moral of the story for me? Next time, pay the big bucks and launch an m1.small spot instance.

Assuming you already have the Fog gem installed:

First, as a prerequisite and as Jeff Gran notes, you’ll need to create a Rackspace image with the cloud-init package installed.

Next, similar to what Vladimir Vuksan describes, create a config.rb file, and populate the following values as appropriate for your environment:

#!/usr/bin/env ruby

@flavor_id = 3
@image_id = 1234567

@rackspace_username =  'example'
@rackspace_api_key = '1234....'

@private_key_path = './ssh/id_rsa'
@public_key_path = './ssh/id_rsa.pub'

The flavor_id values and image_id specify the instance size and the image you built with cloud-init installed (see the “fog” executable’s “Compute[:rackspace].flavors” and “Compute[:rackspace].images”, respectively); the Rackspace username and api_key can be retrieved from within the console under “Your Account: API Access.” The SSH key pair will be what you use to access the new instance as root.

Third, create a cloud-init user_data file ERB template; place it in ./cloud-init/user_data.erb; for example:

#cloud-config
apt_upgrade: true
hostname: <%= hostname %>

packages:
- emacs
- git
- puppet

#runcmd:

#ssh_keys:
#  rsa_private: |
#  rsa_public:
#  dsa_private: |
#  dsa_public:

Finally, the script you launch to deploy Rackspace images is as follows:

#!/usr/bin/env ruby

# Based on:
# http://blog.vuksan.com/2010/07/20/provision-to-cloud-in-5-minutes-using-fog/
# as well as:
# http://jeffgran.com/276/blog/ubuntu-cloud-init-rackspace-fog-ruby

require 'erb'
require 'optparse'

require 'rubygems'

require 'fog'
require 'mime'

# Parse options:
options = {}

optparse = OptionParser.new do|opts|
  opts.banner = "Usage new_instance.rb [options]"

  options[:name] = 'rax.example.com'
  opts.on( '-n', '--name INSTANCE_NAME', 'Instance name (default: rax.example.com)' ) do|n|
    options[:name] = n || 'rax.example.com'
  end

  options[:user_data] = './cloud-init/user_data.erb'
  opts.on( '-u', '--userdata USER_DATA', 'Path to Cloud-Init user_data ERB template (default: ./cloud-init/user_data.erb)' ) do |u|
    options[:user_data] = u || './cloud-init/user_data.erb'
  end

  opts.on( '-h', '--help', 'Display this help message' ) do
    puts opts
    exit
  end
end

optparse.parse!

hostname = options[:name] # to facilitate erb cloud-init template

# Create cloud-init data:

f = File.new(options[:user_data])
e = ERB.new(f.read)
user_data = MIME::MultipartMedia::Mixed.new
user_data.add_entity(MIME::TextMedia.new(e.result, 'text/plain'))

# Import Rackspace credentials:
require './config.rb'

# Connect to Rackspace:
connection = Fog::Compute.new({ 
  :provider           => 'Rackspace',
  :rackspace_api_key  => @rackspace_api_key, 
  :rackspace_username => @rackspace_username
})

# Launch instance:
puts "Launching instance '#{options[:name]}'..."

server = connection.servers.bootstrap({
  :flavor_id        => @flavor_id,
  :image_id         => @image_id,
  :name             => hostname,
  :personality      => [ { 'path'     => '/var/lib/cloud/seed/nocloud-net/user-data',
                           'contents' => user_data.to_s }, 
                         { 'path'     => '/var/lib/cloud/seed/nocloud-net/meta-data',
                           'contents' => ' ' } ],
  :private_key_path => @private_key_path,
  :public_key_path  => @public_key_path
})

puts "Instance launched at #{server.public_ip_address()}"

Launch it with the “-h” flag to see usage; otherwise, launch it with no arguments to launch an instance with your default options.

In this example, I’ll be installing a replacement drive pulled from aggr0 on another filer. Note that this procedure is not relevant for drive failures covered by a support contract, where you will receive a zeroed replacement drive directly from NetApp.

• Physically remove failed drive and replace with working drive. This will generate log messages similar to the following:

  May 27 11:02:36 filer01 [raid.disk.missing: info]: Disk 1b.51 Shelf 3 Bay 3 [NETAPP   X268_SGLXY750SSX AQNZ] S/N [5QD599LZ] is missing from the system
  May 27 11:03:00 filer01 [monitor.globalStatus.ok: info]: The system's global status is normal. 
  May 27 11:03:16 filer01 [scsi.cmd.notReadyCondition: notice]: Disk device 0a.51: Device returns not yet ready: CDB 0x12: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x0)(7715).
  May 27 11:03:25 filer01 [sfu.firmwareUpToDate: info]: Firmware is up-to-date on all disk shelves.
  May 27 11:03:27 filer01 [diskown.changingOwner: info]: changing ownership for disk 0a.51 (S/N P8G9SMDF) from unowned (ID -1) to filer01 (ID 135027165)
  May 27 11:03:27 filer01 [raid.assim.rg.missingChild: error]: Aggregate foreign:aggr0, rgobj_verify: RAID object 0 has only 1 valid children, expected 14.
  May 27 11:03:27 filer01 [raid.assim.plex.missingChild: error]: Aggregate foreign:aggr0, plexobj_verify: Plex 0 only has 0 working RAID groups (2 total) and is being taken offline
  May 27 11:03:27 filer01 [raid.assim.mirror.noChild: ALERT]: Aggregate foreign:aggr0, mirrorobj_verify: No operable plexes found.
  May 27 11:03:27 filer01 [raid.assim.tree.foreign: error]: raidtree_verify: Aggregate aggr0 is a foreign aggregate and is being taken offline. Use the 'aggr online' command to bring it online.
  May 27 11:03:27 filer01 [raid.assim.tree.dupName: error]: Duplicate aggregate names found, an instance of foreign:aggr0 is being renamed to foreign:aggr0(1).
  May 27 11:03:28 filer01 [sfu.firmwareUpToDate: info]: Firmware is up-to-date on all disk shelves.
  May 27 11:04:40 filer01 [asup.smtp.sent: notice]: System Notification mail sent: System Notification from filer01 (RAID VOLUME FAILED) ERROR
  May 27 11:04:42 filer01 [asup.post.sent: notice]: System Notification message posted to NetApp: System Notification from filer01 (RAID VOLUME FAILED) ERROR
  

  Note line 6, where it identifies the newly-added disk as part of “foreign:aggr0″ and missing the rest of its RAID group; “foreign:aggr0″ is taken offline in line 9. In line 10, “foreign:aggr0″ is renamed to “foreign:aggr0(1)” because the filer already has an aggr0, as you might expect. Be sure to note the new aggregate name, as you will need it for later steps.

• Verify aggregate status and names:

  filer01> aggr status
             Aggr State           Status            Options
            aggr0 online          raid_dp, aggr     root
            aggr1 online          raid_dp, aggr     
         aggr0(1) failed          raid_dp, aggr     diskroot, lost_write_protect=off,
                                  foreign           
                                  partial           
            aggr2 online          raid_dp, aggr     nosnap=on
  

• Double-check the name of the foreign, offline aggregate that was brought in with the replacement drive, and destroy it:

  filer01> aggr destroy aggr0(1)
  Are you sure you want to destroy this aggregate? yes
  Aggregate 'aggr0(1)' destroyed.
  

• Verify that the aggregate has been removed:

  netapp03> aggr status          
             Aggr State           Status            Options
            aggr0 online          raid_dp, aggr     root
            aggr1 online          raid_dp, aggr     
            aggr2 online          raid_dp, aggr     nosnap=on
  

• Zero the new spare. First, confirm it is un-zeroed:

  filer01> vol status -s
  
  Spare disks
  
  RAID Disk	Device	HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
  ---------	------	------------- ---- ---- ---- ----- --------------    --------------
  Spare disks for block or zoned checksum traditional volumes or aggregates
  spare   	0a.53	0a    3   5   FC:B   -  ATA   7200 635555/1301618176 635858/1302238304 (not zeroed)
  spare   	0a.69	0a    4   5   FC:B   -  ATA   7200 635555/1301618176 635858/1302238304 
  spare   	1b.51	1b    3   3   FC:A   -  ATA   7200 635555/1301618176 635858/1302238304 (not zeroed)
  spare   	1b.61	1b    3   13  FC:A   -  ATA   7200 635555/1301618176 635858/1302238304 
  spare   	1b.87	1b    5   7   FC:A   -  ATA   7200 847555/1735794176 847827/1736350304 
  spare   	1b.89	1b    5   9   FC:A   -  ATA   7200 847555/1735794176 847827/1736350304 
  

  In this example, we actually have two un-zeroed spares – the newly replaced drive (1b.51) and another drive (0a.53). Zero them both:

  filer01> disk zero spares
  

  And verify that they have been zeroed:

  filer01> vol status -s
  
  Spare disks
  
  RAID Disk	Device	HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
  ---------	------	------------- ---- ---- ---- ----- --------------    --------------
  Spare disks for block or zoned checksum traditional volumes or aggregates
  spare   	0a.53	0a    3   5   FC:B   -  ATA   7200 635555/1301618176 635858/1302238304 
  spare   	0a.69	0a    4   5   FC:B   -  ATA   7200 635555/1301618176 635858/1302238304 
  spare   	1b.51	1b    3   3   FC:A   -  ATA   7200 635555/1301618176 635858/1302238304 
  spare   	1b.61	1b    3   13  FC:A   -  ATA   7200 635555/1301618176 635858/1302238304 
  spare   	1b.87	1b    5   7   FC:A   -  ATA   7200 847555/1735794176 847827/1736350304 
  spare   	1b.89	1b    5   9   FC:A   -  ATA   7200 847555/1735794176 847827/1736350304 
  

• Done. You have replaced a failed drive with a zeroed spare.

For this example, the following services are present, or will be configured here:

• 192.168.1.34: Syslog server
• 192.168.1.68: Web backend #1
• 192.168.1.69: Web backend #2
• 192.168.1.80: HAProxy host #1 management address
• 192.168.1.81: HAProxy host #2 management address
• 192.168.1.84: Load balancer “front end” – the IP address users will ultimately connect to, managed by Keepalived
• 192.168.1.119: SMTP server

Note that all interfaces are on the same subnet (192.168.1.0/24); the load balancer communicates with the backends on the same interface it uses for client connections.

HAProxy

To install HAProxy on Ubuntu, simply install the “haproxy” package; as described above, this example uses two hosts with IP addresses of 192.168.1.80 and 192.168.1.81. Set “ENABLED=1″ in /etc/default/haproxy to have the init script that comes with the package start HAProxy. HAProxy’s configuration lives at /etc/haproxy/haproxy.cfg:

# this config needs haproxy-1.1.28 or haproxy-1.2.1

global
        # log 127.0.0.1 local0
        # log 127.0.0.1 local1 notice
        log 192.168.1.34 local0
        user haproxy
        group haproxy
        daemon
        maxconn 20000
 
defaults
        log global
        option dontlognull
        balance leastconn
        clitimeout 60000
        srvtimeout 60000
        contimeout 5000
        retries 3
        option redispatch

listen stats 192.168.1.80:80 # or 192.168.1.81:80 for second HAProxy host
        mode http
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth admin:supersecret
 
listen http 192.168.1.84:80
        mode tcp
        option tcplog
        balance source
        maxconn 10000
        server web01 192.168.1.68:80 maxconn 5000
        server web02 192.168.1.69:80 maxconn 5000
 
 listen https 192.168.1.84:443 
        mode tcp
        option tcplog
        balance roundrobin
        maxconn 10000
        server web01 192.168.1.68:443 maxconn 5000
        server web02 192.168.1.69:443 maxconn 5000

Config Notes
06. Send log messages to the syslog server at 192.168.1.34 using the local0 facility
22. Only stats are provided on this interface – eth0 on the host.
27. Replace “supersecret” with your password for the statistics interface.
29. HTTP listener – 192.168.1.84 is the IP address Keepalived will manage.
30. We are passing HTTP through – not terminating it on – the HAProxy host.
31. In case you’re wondering why we’re not using Keepalived alone, given that we’re not terminating HTTP at HAProxy: The log format options available with HAProxy. (This also gives us the option of changing to HTTP termination in HAProxy later, without installing new software, of course.)
32. Balance based on source address.
34-35. Two HTTP backends.
37. HTTPS listener, again on the Keepalived IP address.
38. HTTPS is also passed through to the web back-ends.
39. See 31 above.
40. Use a round-robin balancing algorithm.
42-43. Two HTTPS backends.

Start HAProxy as follows:

# service haproxy start

Keepalived

Keepalived is installed via the Ubuntu “keepalived” package, intuitively enough. The following entry needs to be made in /etc/sysctl.conf:

net.ipv4.ip_nonlocal_bind=1

(net.ipv4.ip_forward does not need to be set to “1″ for this type of configuration.) Run “sysctl -p” or reboot to apply the sysctl setting.

The keepalived file – /etc/keepalived/keepalived.conf – contains:

global_defs {
    notification_email {
        sysadmin@example.com
    }
    notification_email_from keepalived@haproxy01.example.com
    smtp_server 192.168.1.119
    smtp_connect_timeout 30
}

vrrp_script chk_haproxy { # Requires keepalived-1.1.13
    script "killall -0 haproxy" # widely used idiom
    interval 2 # check every 2 seconds
    weight 2 # add 2 points of prio if OK
}

vrrp_instance VI_1 {
    interface eth0
    state MASTER # or "BACKUP" on backup
    priority 101 # 101 on master, 100 on backup
    virtual_router_id 51

    smtp_alert # Activate SMTP notifications

    authentication {
        auth_type AH
        auth_pass supersecret
    }

    virtual_ipaddress {
        192.168.1.84
    }

    track_script {
        chk_haproxy
    }
}

Config Notes
02-07. Configuration for email notifications on master/backup state changes.
10-14. Script to check if HAProxy is still alive. (“killall -0” is a common idiom seen in Keepalived configurations; I’m curious to know the original source.)
18-19. Configure master (or backup).
22. Send SMTP notifications on master/backup state change for this interface.
26. Set authentication password to something better than “supersecret” here.
30. IP address of the virtual interface – in this case, the IP address in front of the load balancer.
34. Use the chk_haproxy script defined above to check whether to fail over this interface or not.

Start the keepalived daemon:

service keepalived start

Acknowledgement: My co-worker Will Winslow helped with much of the research for this post.
Reference: HAProxy Architecture Guide

> df -h /cloud/hrc/src/
Filesystem            Size  Used Avail Use% Mounted on
s3fs-1.35             256T     0  256T   0% /cloud/hrc/src

In the words of its authors, “s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files).”

Now, make no mistake about it – since s3fs is backed by object storage in a remote data center, this is not for high- or even moderate-IOPS workloads. Routine tasks like expanding tarballs containing many small files or compiling code on an s3fs file system can be painful. But for “colder” storage applications – think online archives, or possibly some backup applications – it shines.

The installation procedure for s3fs is straightforward. I’ve also put a Puppet module for installing s3fs and managing its mounts on GitHub, although you may want to adapt it to distribute your own package of s3fs instead of building it locally on each machine.

S3fs is licensed under the GPL, as is my Puppet module.