Sign up is straightforward – click a few buttons on aws.amazon.com, and a few moments later, you’ll have an email confirming your access to the service. If you dig into the Getting Started Guide, you’ll note that “Part of the sign-up procedure involves receiving a phone call and entering a PIN using the phone keypad,” however, that wasn’t necessary for me. Perhaps it’s only for new AWS accounts?

There is no user interface in the AWS Console although there are indications one is on its way. The Route 53 developer tools are fairly bare-bones at this point – four Perl scripts. Those scripts require relatively uncommon Perl modules, not included in the default Ubuntu (Lucid) repositories, although they are available through CPAN.

However, the third-party Boto Python interface to Amazon Web Services already includes support, and presumably other tools are also rapidly adding support, if they don’t have it already.

Using the Perl tools, I created a zone for an example domain – gearlister.org – and was given four name servers:

ns-1945.awsdns-51.co.uk (205.251.199.153)
ns-39.awsdns-04.com (205.251.192.39)
ns-690.awsdns-22.net (205.251.194.178)
ns-1344.awsdns-40.org (205.251.197.64)

The cross-section of TLDs increase the likelihood that a glue record for one of the Route 53 name servers will be returned with a query to the TLD name servers, reducing latency for clients:

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <<>> @d0.org.afilias-nst.org gearlister.org
; (2 servers found)
;; global options:  printcmd
;; Got answer:
;; >>HEADER<<- opcode: QUERY, status: NOERROR, id: 51176
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; QUESTION SECTION:
;gearlister.org.                        IN      A

;; AUTHORITY SECTION:
gearlister.org.         86400   IN      NS      ns-39.awsdns-04.com.
gearlister.org.         86400   IN      NS      ns-1945.awsdns-51.co.uk.
gearlister.org.         86400   IN      NS      ns-1344.awsdns-40.org.
gearlister.org.         86400   IN      NS      ns-690.awsdns-22.net.

;; ADDITIONAL SECTION:
ns-1344.awsdns-40.org.  86400   IN      A       205.251.197.64

;; Query time: 161 msec
;; SERVER: 199.19.57.1#53(199.19.57.1)
;; WHEN: Mon Dec  6 12:13:30 2010
;; MSG SIZE  rcvd: 184

While all name servers are in 205.251.192.0/18, there appear to be separate routing table entries for each /23 containing a name server. Further, the servers appear to be anycast to different locations around the world:

traceroute to 205.251.199.153 (205.251.199.153), 30 hops max, 40 byte packets
 1  swiCS5-V108.switch.ch (130.59.108.5)  0.355 ms  0.429 ms  0.546 ms
 2  swiZH2-10GE-3-1.switch.ch (130.59.36.138)  0.437 ms  0.515 ms  0.610 ms
 3  swiIX1-10GE-1-3.switch.ch (130.59.36.129)  6.300 ms  6.391 ms  6.486 ms
 4  zch-b1-geth3-1.telia.net (213.248.79.189)  0.345 ms  0.348 ms  0.347 ms
 5  ffm-bb2-link.telia.net (80.91.249.115)  11.914 ms  11.963 ms  11.991 ms
 6  ffm-b10-link.telia.net (80.91.251.250)  11.846 ms  11.834 ms ffm-b10-link.telia.net (80.91.251.126)  11.834 ms
 7  xe-4-2-0.edge4.Frankfurt1.level3.net (4.68.63.121)  11.960 ms  11.972 ms  11.961 ms
 8  vlan99.csw4.Frankfurt1.Level3.net (4.68.23.254)  20.991 ms vlan89.csw3.Frankfurt1.Level3.net (4.68.23.190)  12.241 ms  12.220 ms
 9  ae-82-82.ebr2.Frankfurt1.Level3.net (4.69.140.25)  13.459 ms  12.333 ms ae-92-92.ebr2.Frankfurt1.Level3.net (4.69.140.29)  12.699 ms
10  ae-24-24.ebr2.London1.Level3.net (4.69.148.197)  24.591 ms ae-21-21.ebr2.London1.Level3.net (4.69.148.185)  26.136 ms ae-22-22.ebr2.London1.Level3.net (4.69.148.189)  25.632 ms
11  ae-22-52.car2.London1.Level3.net (4.69.139.99)  20.163 ms  20.064 ms  19.870 ms
12  AMAZONCOM.car2.London1.Level3.net (212.187.193.2)  19.840 ms  19.868 ms  20.107 ms
13  * * *

Tracing the route to 205.251.199.153

  1 vl-51.uonet1-gw.uoregon.edu (128.223.51.2) [AS 3582] 0 msec 0 msec 0 msec
  2 3.xe-1-3-0.uonet10-gw.uoregon.edu (128.223.3.10) [AS 3582] 0 msec 0 msec 0 msec
  3 vl-3.uonet9-gw.uoregon.edu (128.223.3.9) [AS 3582] 0 msec 0 msec 0 msec
  4 eugn-car1-gw.nero.net (207.98.68.181) [AS 3701] 4 msec 0 msec 0 msec
  5 eugn-core1-gw.nero.net (207.98.64.161) [AS 3701] 0 msec 0 msec 0 msec
  6 eugnor1wce1-gige7-0.wcg.net (64.200.134.197) [AS 3356] 16 msec 8 msec 8 msec
  7 ae-32-52.ebr2.Seattle1.Level3.net (4.68.105.62) [AS 3356] 20 msec 8 msec 16 msec
  8 ae-2-2.ebr2.Denver1.Level3.net (4.69.132.54) [AS 3356] 48 msec 40 msec 36 msec
  9 ae-1-100.ebr1.Denver1.Level3.net (4.69.132.37) [AS 3356] 44 msec 36 msec 36 msec
 10 ae-4-4.car1.StLouis1.Level3.net (4.69.132.181) [AS 3356] 56 msec 52 msec 56 msec
 11 ae-11-11.car2.StLouis1.Level3.net (4.69.132.186) [AS 3356] 52 msec 56 msec 52 msec
 12 AMAZONCOM.car2.StLouis1.Level3.net (4.53.162.66) [AS 3356] 56 msec 56 msec 56 msec
 13  *  *  *

Adapting the Getting Started Guide, I created two A records for “gearlister.org” and “www.gearlister.org”. For reasons I wasn’t able to track down – or reproduce – adding the “gearlister.org” A record failed the first time, although I was able to add it later.

Update 12/7/2010: I received an email from Amazon earlier today explaining the failed A record add:

Here’s what happened. In our “Getting Started Guide” we incorrectly provided an example ChangeResourceRecordSets request that showed a single element that included multiple elements. This was a mistake. In reality, only one element is permitted per element. Our API accepted this request as valid, but silently only processed one of the elements. We have now fixed both the documentation and the configuration API to enforce the proper semantics.

Record changes propagated quickly, although the zone serial number did not increment. Given the lack of support for connecting secondary servers to Route 53 and the API support for checking whether a change has propagated, this may matter little in practice, although it is certainly odd.

Although there is no direct support for secondary servers using Route 53 as primary DNS – or for using Route 53 as a secondary to a non-Amazon primary – the BIND conversion scripts hint that it should be straightforward to have a master script update Route 53 and non-Route 53 zone configuration simultaneously. Also, while Route 53 does support AAAA records (ironic, given that you cannot use IPv6 to address EC2 instances), it does not yet support DNSSEC.

At $1/domain/month and $0.50/millon queries, pricing is extremely low for anycast DNS. However, given the lack of integration with some of Amazon’s other products, such as Elastic Load Balancing – apparently forthcoming – and the limited tools for managing zones, uptake will probably be limited initially. Some heavy AWS users may be hesitant to put their DNS on the same service provider as the rest of their infrastructure – although ultimately, as Amazon adds features, the benefits of Route 53 may outweigh the risks.

(See bottom of post for an update running similar tests on OpenDNS.)

Methods: I searched Google for keywords that I believed fell somewhere between obscure and common and collected the first ten hostnames printed on the screen. I then used local installations of dig to query a collection of DNS servers for the hostnames’ A records and collected the response times. The different resolvers used were:

• A local BIND installation (127.0.0.1, cache empty) with Comcast Internet connectivity;
• A Comcast DNS server (68.87.69.150) via Comcast Internet connectivity;
• My employer’s internal caching DNS;
• Google (8.8.8.8) via my employer’s Internet connectivity (mostly Level 3);
• Google (8.8.8.8) via Comcast; and
• Google (8.8.8.8) via an Amazon EC2 instance in us-east-1a.

Anticipating a bimodal distribution of results, I assumed high latency responses were cache misses, while low latency responses were cache hits, and categorized results correspondingly.

Limitations: Chiefly, the small number of hostnames queried. Results from a larger group of domains would be more conclusive.

Results: Given in the format of Server/Connectivity: Cache Miss/Cache Hit

• Local BIND server/Comcast: 319ms/0ms
• Comcast/Comcast: 166ms/14ms
• Google/Comcast: no misses/73ms
• Employer/Level 3: 235ms/30ms
• Google/Level 3: 204ms/44ms
• Google/EC2: 190ms/4ms

I concluded that Google/Comcast had no misses by testing another set of obscure hostnames twice each, noting that the first query was slower (~120ms) and the second query was similar in latency to the results above (70ms). (My belief is that I inadvertently pre-populated the cache for Google/Comcast by my tests elsewhere.)

Discussion:

• It’s all about cache hits. Whichever resolver gives you the most cache hits will give you the best performance; cache misses are at least an order of magnitude slower than cache hits. In this extremely limited test, the cache-hits-champion appears to be Google. Excluding Google/Comcast, where I believe I pre-populated Google’s cache, Google had a 50% cache hit rate, while Comcast and my Employer only hit 20%.
• Location, location, location. Secondary to cache hits, the closer the resolver is to you, the better. Looking at the Comcast results, it’s hard to get closer than localhost, and, as seems logical, Comcast’s resolvers have lower cached latency than Google’s. Running a local caching resolver forwarding to Google may be a desirable configuration.
• Resolver behavior matters. Comcast is notorious for poor behavior. It’s reasonable to expect that Google will be mining your DNS query data. Running a slower but directly-controlled local, non-forwarding server may be preferable for privacy and security reasons.

Update: At the suggestion of @tscalzott, I researched OpenDNS’s performance with the same set of hostnames via the same connectivity on their DNS resolvers at 208.67.222.222. This time, however, I queried the A record for each hostname twice in rapid succession to ascertain how many of my queries were served from OpenDNS’s cache. Results are in the format:

DNS Server/Connectivity: Cache miss/Cache hit – cache hit rate

• OpenDNS/Comcast: 218ms/30ms – 20%
• OpenDNS/EC2: 144ms/2ms – 10%
• OpenDNS/Level 3: 230ms/4ms – 40%

Compared to Google, OpenDNS had similar latency for cache misses and lower latency for cache hits, but appears to possibly have a lower cache hit rate. It seems likely that the latency “winner” for each user’s individual situation will depend on where they are on the Internet relative to the nearest Google and OpenDNS installations. Google’s greater cache hit rate suggests it may offer better service, but testing a larger number of hostnames would be necessary before being able to state that with any certainty.

Disclosure: I use Google’s free Apps services to host personal email, and I use their public sites (Search, Reader, News, Analytics, etc.) extensively. I recently attended a Google Apps for the Enterprise dog-and-pony show where I received a number of small tchotchkes; my wife took the notebook, I kept the pen and binder and threw the rest away. My employer uses Postini. I tried OpenDNS briefly several months back, but did not use them long-term because of limitations in my own configuration.

Assumptions: An IMAP interface to your current email, basic comptency at managing DNS, and the ability to run the imapsync Perl script (built via FreeBSD ports in my case, but installation should be straightforward under most UNIX or Linux systems).

0. Ensure your domain registration is up to date and with a reputable registrar (Pro tip: if you have to wade through pages of deceptive and needless up-sells with your registrar, they’re not reputable), and fully independent from anything you have set up with Google. Ensure your DNS configuration is completely correct; if you’re unsure whether or not it is, buy a membership at DNSstuff, run its tests on your domains, and fix the relevant problems. If you don’t fully host your own DNS – and chances are, if you’re outsourcing your email, you probably don’t host your own DNS – I recommend using two independent DNS providers. Hosted DNS service that’s worth spending money with will be able to function as a secondary and allow zone transfers – and, while you’re at it, make sure your chosen provider supports DNS NOTIFY – it’ll make changing your MX records in a timely fashion that much easier. Whatever you do, don’t use your registrar’s DNS servers.

You occasionally see sad tales of Google Apps woe which could have been mitigated by independent domain registration and a moderate knowledge of DNS. Don’t risk becoming another bitter messageboard-posting statistic.

In summary: Don’t understand DNS or don’t have time for it? Stop reading now; either hire a consultant to do the work, or accept that outsourced email probably isn’t right for you.

0.1. This goes hand-in-hand with step 0 above: Have an off-Google backup system for your email – one that runs unattended, automatically, just like a “real” backup system would be. Google has its strong points, but customer service for their free products isn’t one of them. If Google should update their unofficial motto to a more succinct “be evil” you want to have an “out” and the ability to migrate your mail elsewhere. Imapsync, linked to above and described below, can be adapted for this purpose.

0.2 Install and test imapsync before beginning this process. This is the best tool that I’ve found for syncing mailboxes between servers; all others I tried were unreliable, and documentation below will be specific to it. In theory, another tool could work – IMAP is IMAP and a standard – but I leave using other migration tools as an exercise for the reader.

1. Drop your MX record TTL to a suitably low value; I used 60 seconds during my migration. Let the older, presumably greater TTL age off prior to making any further changes to your MX records (as in step 7 below).

2. Sign up your domain for Google Apps, and use DNS to verify control.

3. Create users for the domain, and accept terms. A best practice would be to use different passwords on Gmail from what users currently have, and do note that you will need both old and new passwords for the synchronization step. (If you are currently running a Cyrus IMAP server with sasldb, passwords can be extracted by running “db3_dump185 -p sasldb2.db” as a user that can read sasldb.)

4. In the Google Apps control panel, create distribution lists and aliases (“nicknames”) for your users as appropriate.

5. For each Gmail mailbox, enable IMAP: Settings > Forwarding and POP/IMAP: Enable IMAP.

6. If appropriate (and assuming you have other users in your domain), have users verify their logins and familiarize themselves with Gmail’s options.

7. Change your MX records to point to Google; keep your TTL low in case you need to change back to your old servers.

8. Verify delivery to your new Gmail mailboxes by sending test messages from a third-party address.

9. Set an appropriate SPF record for your domain in DNS.

10. Sync your old mailboxes with your new; I recommend running imapsync directly on your mail server, if possible, for best performance. Example imapsync session for a Cyrus inbox:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 user@example.com --password2 [...] --ssl2 --folder INBOX

(Insert passwords as appropriate.)

Example imapsync session for a Cyrus folder other than the inbox:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 user@example.com --password2 [...] --ssl2 --folder INBOX.saved-messages

Note that Gmail has certain reserved labels that you cannot sync directly to, such as “Sent”. In this case, you’ll need to use the “regextrans2″ flag to sync to a different folder:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 use@example.com --password2 [...] --ssl2 --folder INBOX.Sent --regextrans2 's/Sent/Old-Sent/'

11. After double checking that your old MX record’s TTL has passed, and no new mail is being delivered to your old mail server, decommission it as necessary.

The tester in the sidebar at DoxPara Research seems to do a good job of testing your configuration end-to-end for this vulnerability.

(File this under “Just another reason why NAT is evil.”)