Running a cpu-bound workload (building Perl modules) on an Ubuntu 11.10 t1.micro instance in us-west-2 tonight, I noticed the following curious CPU usage pattern of approximately 15 seconds on, 60 seconds off:

> vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0  38528  29524 370540    0    0    86   423   84  216 12  5 35  4
 1  0      0   6800  30288 388856    0    0  5356    26  660 1433 27 27  6 40
 5  0      0  21752  27624 378088    0    0    30   211  150  159 40 22  0  8
 6  0      0  21256  27636 378104    0    0     0    27    9    7  1  1  0  0
 7  0      0  21256  27644 378108    0    0     0    10    9    9  1  1  0  0
 7  0      0  21256  27652 378112    0    0     0     8    9    9  2  1  0  0
 7  0      0  20256  27652 378228    0    0     0     0    8   13  1  1  0  0
 8  0      0  20016  27660 378072    0    0     0   218   15   29  0  2  0  3
 6  0      0  37884  27672 378048    0    0     0    14    9   11  3  1  0  0
 4  0      0  30808  27684 378048    0    0     0    11    9   10  1  1  0  0
 4  0      0  23740  27692 378056    0    0     0    10    8    8  2  1  0  0
 4  0      0  30676  27692 378104    0    0     0     0   10   10  1  1  0  0
 5  0      0  26220  27700 378064    0    0     0     9    7   14  6  2  0  1
 5  0      0  21012  27712 378120    0    0     0    10    9   10  1  0  0  0
 5  0      0  27336  27720 378064    0    0     0    21   13   10  1  1  0  0
 1  0      0  29444  27732 378064    0    0     0    14  149   97 39 19  0  0
 1  0      0  33420  27744 378084    0    0     6    12  250  166 67 30  0  0
 2  0      0  41108  27756 378100    0    0     0    37  207  148 60 29  0  0
 6  0      0  33668  27768 378068    0    0     0    14    8    9  1  1  0  0
 5  0      0  37008  27780 378068    0    0     0    10   10   15  4  1  0  0
 4  0      0  30808  27788 378072    0    0     0    18   11    9  2  0  0  0
 5  0      0  24360  27796 378092    0    0     0     9    8    7  2  0  0  0
 2  0      0  19896  27796 378140    0    0     0     0    8    9  1  1  0  0
 6  0      0  27584  27804 378152    0    0     0     7    8   12  1  1  0  0
 6  0      0  22864  27812 378148    0    0     0     9   10   12  2  1  0  0
 7  0      0  19136  27820 378152    0    0     0    10    8    9  1  1  0  0
 6  0      0  26096  27828 378148    0    0     0    12   10    7  2  1  0  0
 6  0      0  20640  27828 378156    0    0     0    19   13    8  2  1  0  0
 6  0      0  27956  27836 378156    0    0     0    11    9   12  1  1  0  0
 6  0      0  22864  27844 378156    0    0     0     6    9   12  2  1  0  0
 6  0      0  19020  27844 378156    0    0     0     1    9    9  1  1  0  0
 2  0      0  46896  21504 368588    0    0   518    18  261  291 47 29  1  7
 1  0      0  35372  21692 368788    0    0     0    43  253  174 65 32  0  0
 1  0      0  43060  21796 368600    0    0     0    62  149  112 66 32  0  1
 5  0      0  38100  21808 368600    0    0     0    46   11   10  1  1  0  0
 5  0      0  45788  21816 368592    0    0     0     7    8   12  2  1  0  0
 7  0      0  38464  21816 368600    0    0     0     0    7    8  2  1  0  0
 7  0      0  45912  21824 368596    0    0     0    11    9    9  2  1  0  0
 7  0      0  39216  21832 368600    0    0     0     7    9    8  1  0  0  0
 4  0      0  35496  21840 368596    0    0     0    19   11    9  4  1  0  0
 5  0      0  43060  21848 368600    0    0     0    29   10   10  2  1  0  0
 5  0      0  37480  21856 368592    0    0     0    11    9   10  1  1  0  0
 5  0      0  45044  21864 368596    0    0     0     7    9   10  1  1  0  0
 5  0      0  38340  21872 368600    0    0     0     8    8    8  2  1  0  0
 4  0      0  46284  21880 368596    0    0     0    10   10   11  1  1  0  0
 6  0      0  38836  21888 368592    0    0     0     8    8    8  2  1  0  0
 1  0      0  38340  21888 368544    0    0     0    15   53   41 12  7  0  0
 1  0      0  40828  21900 368568    0    0     2    46  255  218 66 33  0  0
 1  0      0  39960  21912 368608    0    0     0    26  237  153 63 28  0  0
 3  0      0  50632  21924 368540    0    0     0    16   58   44 32 15  0  0
 4  0      0  46284  21932 368540    0    0     0     7    8   11  1  1  0  0
 4  0      0  45400  21940 368540    0    0     0     6    9   10  1  1  0  0
 5  0      0  45292  21948 368552    0    0     0    11    8   14  0  1  0  0
 6  0      0  37720  21948 368584    0    0     0    17   12    6  2  1  0  0

Apparently, the “small amount of consistent CPU resources” is about 3% of the CPU.

Moral of the story for me? Next time, pay the big bucks and launch an m1.small spot instance.

Assuming you already have the Fog gem installed:

First, as a prerequisite and as Jeff Gran notes, you’ll need to create a Rackspace image with the cloud-init package installed.

Next, similar to what Vladimir Vuksan describes, create a config.rb file, and populate the following values as appropriate for your environment:

#!/usr/bin/env ruby

@flavor_id = 3
@image_id = 1234567

@rackspace_username =  'example'
@rackspace_api_key = '1234....'

@private_key_path = './ssh/id_rsa'
@public_key_path = './ssh/id_rsa.pub'

The flavor_id values and image_id specify the instance size and the image you built with cloud-init installed (see the “fog” executable’s “Compute[:rackspace].flavors” and “Compute[:rackspace].images”, respectively); the Rackspace username and api_key can be retrieved from within the console under “Your Account: API Access.” The SSH key pair will be what you use to access the new instance as root.

Third, create a cloud-init user_data file ERB template; place it in ./cloud-init/user_data.erb; for example:

#cloud-config
apt_upgrade: true
hostname: <%= hostname %>

packages:
- emacs
- git
- puppet

#runcmd:

#ssh_keys:
#  rsa_private: |
#  rsa_public:
#  dsa_private: |
#  dsa_public:

Finally, the script you launch to deploy Rackspace images is as follows:

#!/usr/bin/env ruby

# Based on:
# http://blog.vuksan.com/2010/07/20/provision-to-cloud-in-5-minutes-using-fog/
# as well as:
# http://jeffgran.com/276/blog/ubuntu-cloud-init-rackspace-fog-ruby

require 'erb'
require 'optparse'

require 'rubygems'

require 'fog'
require 'mime'

# Parse options:
options = {}

optparse = OptionParser.new do|opts|
  opts.banner = "Usage new_instance.rb [options]"

  options[:name] = 'rax.example.com'
  opts.on( '-n', '--name INSTANCE_NAME', 'Instance name (default: rax.example.com)' ) do|n|
    options[:name] = n || 'rax.example.com'
  end

  options[:user_data] = './cloud-init/user_data.erb'
  opts.on( '-u', '--userdata USER_DATA', 'Path to Cloud-Init user_data ERB template (default: ./cloud-init/user_data.erb)' ) do |u|
    options[:user_data] = u || './cloud-init/user_data.erb'
  end

  opts.on( '-h', '--help', 'Display this help message' ) do
    puts opts
    exit
  end
end

optparse.parse!

hostname = options[:name] # to facilitate erb cloud-init template

# Create cloud-init data:

f = File.new(options[:user_data])
e = ERB.new(f.read)
user_data = MIME::MultipartMedia::Mixed.new
user_data.add_entity(MIME::TextMedia.new(e.result, 'text/plain'))

# Import Rackspace credentials:
require './config.rb'

# Connect to Rackspace:
connection = Fog::Compute.new({
  :provider           => 'Rackspace',
  :rackspace_api_key  => @rackspace_api_key,
  :rackspace_username => @rackspace_username
})

# Launch instance:
puts "Launching instance '#{options[:name]}'..."

server = connection.servers.bootstrap({
  :flavor_id        => @flavor_id,
  :image_id         => @image_id,
  :name             => hostname,
  :personality      => [ { 'path'     => '/var/lib/cloud/seed/nocloud-net/user-data',
                           'contents' => user_data.to_s },
                         { 'path'     => '/var/lib/cloud/seed/nocloud-net/meta-data',
                           'contents' => ' ' } ],
  :private_key_path => @private_key_path,
  :public_key_path  => @public_key_path
})

puts "Instance launched at #{server.public_ip_address()}"

Launch it with the “-h” flag to see usage; otherwise, launch it with no arguments to launch an instance with your default options.

> df -h /cloud/hrc/src/
Filesystem            Size  Used Avail Use% Mounted on
s3fs-1.35             256T     0  256T   0% /cloud/hrc/src

In the words of its authors, “s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files).”

Now, make no mistake about it – since s3fs is backed by object storage in a remote data center, this is not for high- or even moderate-IOPS workloads. Routine tasks like expanding tarballs containing many small files or compiling code on an s3fs file system can be painful. But for “colder” storage applications – think online archives, or possibly some backup applications – it shines.

The installation procedure for s3fs is straightforward. I’ve also put a Puppet module for installing s3fs and managing its mounts on GitHub, although you may want to adapt it to distribute your own package of s3fs instead of building it locally on each machine.

S3fs is licensed under the GPL, as is my Puppet module.

Sign up is straightforward – click a few buttons on aws.amazon.com, and a few moments later, you’ll have an email confirming your access to the service. If you dig into the Getting Started Guide, you’ll note that “Part of the sign-up procedure involves receiving a phone call and entering a PIN using the phone keypad,” however, that wasn’t necessary for me. Perhaps it’s only for new AWS accounts?

There is no user interface in the AWS Console although there are indications one is on its way. The Route 53 developer tools are fairly bare-bones at this point – four Perl scripts. Those scripts require relatively uncommon Perl modules, not included in the default Ubuntu (Lucid) repositories, although they are available through CPAN.

However, the third-party Boto Python interface to Amazon Web Services already includes support, and presumably other tools are also rapidly adding support, if they don’t have it already.

Using the Perl tools, I created a zone for an example domain – gearlister.org – and was given four name servers:

ns-1945.awsdns-51.co.uk (205.251.199.153)
ns-39.awsdns-04.com (205.251.192.39)
ns-690.awsdns-22.net (205.251.194.178)
ns-1344.awsdns-40.org (205.251.197.64)

The cross-section of TLDs increase the likelihood that a glue record for one of the Route 53 name servers will be returned with a query to the TLD name servers, reducing latency for clients:

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <<>> @d0.org.afilias-nst.org gearlister.org
; (2 servers found)
;; global options:  printcmd
;; Got answer:
;; >>HEADER<<- opcode: QUERY, status: NOERROR, id: 51176
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; QUESTION SECTION:
;gearlister.org.                        IN      A

;; AUTHORITY SECTION:
gearlister.org.         86400   IN      NS      ns-39.awsdns-04.com.
gearlister.org.         86400   IN      NS      ns-1945.awsdns-51.co.uk.
gearlister.org.         86400   IN      NS      ns-1344.awsdns-40.org.
gearlister.org.         86400   IN      NS      ns-690.awsdns-22.net.

;; ADDITIONAL SECTION:
ns-1344.awsdns-40.org.  86400   IN      A       205.251.197.64

;; Query time: 161 msec
;; SERVER: 199.19.57.1#53(199.19.57.1)
;; WHEN: Mon Dec  6 12:13:30 2010
;; MSG SIZE  rcvd: 184

While all name servers are in 205.251.192.0/18, there appear to be separate routing table entries for each /23 containing a name server. Further, the servers appear to be anycast to different locations around the world:

traceroute to 205.251.199.153 (205.251.199.153), 30 hops max, 40 byte packets
 1  swiCS5-V108.switch.ch (130.59.108.5)  0.355 ms  0.429 ms  0.546 ms
 2  swiZH2-10GE-3-1.switch.ch (130.59.36.138)  0.437 ms  0.515 ms  0.610 ms
 3  swiIX1-10GE-1-3.switch.ch (130.59.36.129)  6.300 ms  6.391 ms  6.486 ms
 4  zch-b1-geth3-1.telia.net (213.248.79.189)  0.345 ms  0.348 ms  0.347 ms
 5  ffm-bb2-link.telia.net (80.91.249.115)  11.914 ms  11.963 ms  11.991 ms
 6  ffm-b10-link.telia.net (80.91.251.250)  11.846 ms  11.834 ms ffm-b10-link.telia.net (80.91.251.126)  11.834 ms
 7  xe-4-2-0.edge4.Frankfurt1.level3.net (4.68.63.121)  11.960 ms  11.972 ms  11.961 ms
 8  vlan99.csw4.Frankfurt1.Level3.net (4.68.23.254)  20.991 ms vlan89.csw3.Frankfurt1.Level3.net (4.68.23.190)  12.241 ms  12.220 ms
 9  ae-82-82.ebr2.Frankfurt1.Level3.net (4.69.140.25)  13.459 ms  12.333 ms ae-92-92.ebr2.Frankfurt1.Level3.net (4.69.140.29)  12.699 ms
10  ae-24-24.ebr2.London1.Level3.net (4.69.148.197)  24.591 ms ae-21-21.ebr2.London1.Level3.net (4.69.148.185)  26.136 ms ae-22-22.ebr2.London1.Level3.net (4.69.148.189)  25.632 ms
11  ae-22-52.car2.London1.Level3.net (4.69.139.99)  20.163 ms  20.064 ms  19.870 ms
12  AMAZONCOM.car2.London1.Level3.net (212.187.193.2)  19.840 ms  19.868 ms  20.107 ms
13  * * *

Tracing the route to 205.251.199.153

  1 vl-51.uonet1-gw.uoregon.edu (128.223.51.2) [AS 3582] 0 msec 0 msec 0 msec
  2 3.xe-1-3-0.uonet10-gw.uoregon.edu (128.223.3.10) [AS 3582] 0 msec 0 msec 0 msec
  3 vl-3.uonet9-gw.uoregon.edu (128.223.3.9) [AS 3582] 0 msec 0 msec 0 msec
  4 eugn-car1-gw.nero.net (207.98.68.181) [AS 3701] 4 msec 0 msec 0 msec
  5 eugn-core1-gw.nero.net (207.98.64.161) [AS 3701] 0 msec 0 msec 0 msec
  6 eugnor1wce1-gige7-0.wcg.net (64.200.134.197) [AS 3356] 16 msec 8 msec 8 msec
  7 ae-32-52.ebr2.Seattle1.Level3.net (4.68.105.62) [AS 3356] 20 msec 8 msec 16 msec
  8 ae-2-2.ebr2.Denver1.Level3.net (4.69.132.54) [AS 3356] 48 msec 40 msec 36 msec
  9 ae-1-100.ebr1.Denver1.Level3.net (4.69.132.37) [AS 3356] 44 msec 36 msec 36 msec
 10 ae-4-4.car1.StLouis1.Level3.net (4.69.132.181) [AS 3356] 56 msec 52 msec 56 msec
 11 ae-11-11.car2.StLouis1.Level3.net (4.69.132.186) [AS 3356] 52 msec 56 msec 52 msec
 12 AMAZONCOM.car2.StLouis1.Level3.net (4.53.162.66) [AS 3356] 56 msec 56 msec 56 msec
 13  *  *  *

Adapting the Getting Started Guide, I created two A records for “gearlister.org” and “www.gearlister.org”. For reasons I wasn’t able to track down – or reproduce – adding the “gearlister.org” A record failed the first time, although I was able to add it later.

Update 12/7/2010: I received an email from Amazon earlier today explaining the failed A record add:

Here’s what happened. In our “Getting Started Guide” we incorrectly provided an example ChangeResourceRecordSets request that showed a single element that included multiple elements. This was a mistake. In reality, only one element is permitted per element. Our API accepted this request as valid, but silently only processed one of the elements. We have now fixed both the documentation and the configuration API to enforce the proper semantics.

Record changes propagated quickly, although the zone serial number did not increment. Given the lack of support for connecting secondary servers to Route 53 and the API support for checking whether a change has propagated, this may matter little in practice, although it is certainly odd.

Although there is no direct support for secondary servers using Route 53 as primary DNS – or for using Route 53 as a secondary to a non-Amazon primary – the BIND conversion scripts hint that it should be straightforward to have a master script update Route 53 and non-Route 53 zone configuration simultaneously. Also, while Route 53 does support AAAA records (ironic, given that you cannot use IPv6 to address EC2 instances), it does not yet support DNSSEC.

At $1/domain/month and $0.50/millon queries, pricing is extremely low for anycast DNS. However, given the lack of integration with some of Amazon’s other products, such as Elastic Load Balancing – apparently forthcoming – and the limited tools for managing zones, uptake will probably be limited initially. Some heavy AWS users may be hesitant to put their DNS on the same service provider as the rest of their infrastructure – although ultimately, as Amazon adds features, the benefits of Route 53 may outweigh the risks.

However, if you’re willing to accept the risk of unexpected high EBS I/O costs, it’s straightforward to add swap:

# /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
# /sbin/mkswap /var/swap.1
# /sbin/swapon /var/swap.1

Or, if you prefer Puppet:

class swapfile {

  exec { "create swap file":
    command => "/bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024",
    creates => "/var/swap.1",
  }

  exec { "attach swap file":
    command => "/sbin/mkswap /var/swap.1 && /sbin/swapon /var/swap.1",
    require => Exec["create swap file"],
    unless => "/sbin/swapon -s | grep /var/swap.1",
  }

}

Assumptions: An IMAP interface to your current email, basic comptency at managing DNS, and the ability to run the imapsync Perl script (built via FreeBSD ports in my case, but installation should be straightforward under most UNIX or Linux systems).

0. Ensure your domain registration is up to date and with a reputable registrar (Pro tip: if you have to wade through pages of deceptive and needless up-sells with your registrar, they’re not reputable), and fully independent from anything you have set up with Google. Ensure your DNS configuration is completely correct; if you’re unsure whether or not it is, buy a membership at DNSstuff, run its tests on your domains, and fix the relevant problems. If you don’t fully host your own DNS – and chances are, if you’re outsourcing your email, you probably don’t host your own DNS – I recommend using two independent DNS providers. Hosted DNS service that’s worth spending money with will be able to function as a secondary and allow zone transfers – and, while you’re at it, make sure your chosen provider supports DNS NOTIFY – it’ll make changing your MX records in a timely fashion that much easier. Whatever you do, don’t use your registrar’s DNS servers.

You occasionally see sad tales of Google Apps woe which could have been mitigated by independent domain registration and a moderate knowledge of DNS. Don’t risk becoming another bitter messageboard-posting statistic.

In summary: Don’t understand DNS or don’t have time for it? Stop reading now; either hire a consultant to do the work, or accept that outsourced email probably isn’t right for you.

0.1. This goes hand-in-hand with step 0 above: Have an off-Google backup system for your email – one that runs unattended, automatically, just like a “real” backup system would be. Google has its strong points, but customer service for their free products isn’t one of them. If Google should update their unofficial motto to a more succinct “be evil” you want to have an “out” and the ability to migrate your mail elsewhere. Imapsync, linked to above and described below, can be adapted for this purpose.

0.2 Install and test imapsync before beginning this process. This is the best tool that I’ve found for syncing mailboxes between servers; all others I tried were unreliable, and documentation below will be specific to it. In theory, another tool could work – IMAP is IMAP and a standard – but I leave using other migration tools as an exercise for the reader.

1. Drop your MX record TTL to a suitably low value; I used 60 seconds during my migration. Let the older, presumably greater TTL age off prior to making any further changes to your MX records (as in step 7 below).

2. Sign up your domain for Google Apps, and use DNS to verify control.

3. Create users for the domain, and accept terms. A best practice would be to use different passwords on Gmail from what users currently have, and do note that you will need both old and new passwords for the synchronization step. (If you are currently running a Cyrus IMAP server with sasldb, passwords can be extracted by running “db3_dump185 -p sasldb2.db” as a user that can read sasldb.)

4. In the Google Apps control panel, create distribution lists and aliases (“nicknames”) for your users as appropriate.

5. For each Gmail mailbox, enable IMAP: Settings > Forwarding and POP/IMAP: Enable IMAP.

6. If appropriate (and assuming you have other users in your domain), have users verify their logins and familiarize themselves with Gmail’s options.

7. Change your MX records to point to Google; keep your TTL low in case you need to change back to your old servers.

8. Verify delivery to your new Gmail mailboxes by sending test messages from a third-party address.

9. Set an appropriate SPF record for your domain in DNS.

10. Sync your old mailboxes with your new; I recommend running imapsync directly on your mail server, if possible, for best performance. Example imapsync session for a Cyrus inbox:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 user@example.com --password2 [...] --ssl2 --folder INBOX

(Insert passwords as appropriate.)

Example imapsync session for a Cyrus folder other than the inbox:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 user@example.com --password2 [...] --ssl2 --folder INBOX.saved-messages

Note that Gmail has certain reserved labels that you cannot sync directly to, such as “Sent”. In this case, you’ll need to use the “regextrans2″ flag to sync to a different folder:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 use@example.com --password2 [...] --ssl2 --folder INBOX.Sent --regextrans2 's/Sent/Old-Sent/'

11. After double checking that your old MX record’s TTL has passed, and no new mail is being delivered to your old mail server, decommission it as necessary.