thinking sysadmin

I recently moved my personal email from a self-managed Exim/Cyrus setup on a dedicated FreeBSD server to Gmail (Google Apps for Domains). This migration was motivated by a desire to reduce expenses, reduce time spent managing mail software and the importance of email (for me, personally) dropping to a level where I was willing to accept the risks inherent in outsourcing it. Details of the exact process I used to migrate mail are below.

Assumptions: An IMAP interface to your current email, basic comptency at managing DNS, and the ability to run the imapsync Perl script (built via FreeBSD ports in my case, but installation should be straightforward under most UNIX or Linux systems).

0. Ensure your domain registration is up to date and with a reputable registrar (Pro tip: if you have to wade through pages of deceptive and needless up-sells with your registrar, they’re not reputable), and fully independent from anything you have set up with Google. Ensure your DNS configuration is completely correct; if you’re unsure whether or not it is, buy a membership at DNSstuff, run its tests on your domains, and fix the relevant problems. If you don’t fully host your own DNS – and chances are, if you’re outsourcing your email, you probably don’t host your own DNS – I recommend using two independent DNS providers. Hosted DNS service that’s worth spending money with will be able to function as a secondary and allow zone transfers – and, while you’re at it, make sure your chosen provider supports DNS NOTIFY – it’ll make changing your MX records in a timely fashion that much easier. Whatever you do, don’t use your registrar’s DNS servers.

You occasionally see sad tales of Google Apps woe which could have been mitigated by independent domain registration and a moderate knowledge of DNS. Don’t risk becoming another bitter messageboard-posting statistic.

In summary: Don’t understand DNS or don’t have time for it? Stop reading now; either hire a consultant to do the work, or accept that outsourced email probably isn’t right for you.

0.1. This goes hand-in-hand with step 0 above: Have an off-Google backup system for your email – one that runs unattended, automatically, just like a “real” backup system would be. Google has its strong points, but customer service for their free products isn’t one of them. If Google should update their unofficial motto to a more succinct “be evil” you want to have an “out” and the ability to migrate your mail elsewhere. Imapsync, linked to above and described below, can be adapted for this purpose.

0.2 Install and test imapsync before beginning this process. This is the best tool that I’ve found for syncing mailboxes between servers; all others I tried were unreliable, and documentation below will be specific to it. In theory, another tool could work – IMAP is IMAP and a standard – but I leave using other migration tools as an exercise for the reader.

1. Drop your MX record TTL to a suitably low value; I used 60 seconds during my migration. Let the older, presumably greater TTL age off prior to making any further changes to your MX records (as in step 7 below).

2. Sign up your domain for Google Apps, and use DNS to verify control.

3. Create users for the domain, and accept terms. A best practice would be to use different passwords on Gmail from what users currently have, and do note that you will need both old and new passwords for the synchronization step. (If you are currently running a Cyrus IMAP server with sasldb, passwords can be extracted by running “db3_dump185 -p sasldb2.db” as a user that can read sasldb.)

4. In the Google Apps control panel, create distribution lists and aliases (“nicknames”) for your users as appropriate.

5. For each Gmail mailbox, enable IMAP: Settings > Forwarding and POP/IMAP: Enable IMAP.

6. If appropriate (and assuming you have other users in your domain), have users verify their logins and familiarize themselves with Gmail’s options.

7. Change your MX records to point to Google; keep your TTL low in case you need to change back to your old servers.

8. Verify delivery to your new Gmail mailboxes by sending test messages from a third-party address.

9. Set an appropriate SPF record for your domain in DNS.

10. Sync your old mailboxes with your new; I recommend running imapsync directly on your mail server, if possible, for best performance. Example imapsync session for a Cyrus inbox:

imapsync --host1 mail.example.com --port1 143 --user1 [email protected] --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 [email protected] --password2 [...] --ssl2 --folder INBOX

(Insert passwords as appropriate.)

Example imapsync session for a Cyrus folder other than the inbox:

imapsync --host1 mail.example.com --port1 143 --user1 [email protected] --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 [email protected] --password2 [...] --ssl2 --folder INBOX.saved-messages

Note that Gmail has certain reserved labels that you cannot sync directly to, such as “Sent”. In this case, you’ll need to use the “regextrans2″ flag to sync to a different folder:

imapsync --host1 mail.example.com --port1 143 --user1 [email protected] --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 [email protected] --password2 [...] --ssl2 --folder INBOX.Sent --regextrans2 's/Sent/Old-Sent/'

11. After double checking that your old MX record’s TTL has passed, and no new mail is being delivered to your old mail server, decommission it as necessary.