Despite its many desirable features and continuing popularity, Drupal is not without its shortcomings, as many readers are also likely aware. Although Drupal has an active and responsive security team, the software has a long track record of requiring frequent security patches – Secunia has seven 2009 advisories for Drupal 6.x listed as of this writing. Although by its nature an apples-to-oranges comparison, this ranks Drupal behind similarly large and complex PHP projects such as Wordpress 2.x (5) and Gallery 2.x (0) – and the number for Drupal does not include dozens of additional advisories for Drupal modules. Further, Drupal has struggled and lagged with support for PHP 5.3.x, suggesting to this outside observer that the project is having difficulties maintaining its codebase.

All that being said, I do not personally believe that the above issues rule out using Drupal; the benefits outweigh the shortcomings. So, assuming the question is not whether to deploy Drupal, but how to do so most securely and efficiently, my recommendations from a systems administration perspective are below.

Goals

The recommendations described stem from three goals for a Drupal installation:

1. Security – Avoid compromise of your site and system.
2. Flexibility – Create an environment that allows for easy modification of the OS, server and application, and provides for straightforward roll-back of those changes should those need arise.
3. Stability and Security over Performance – Although not mutually exclusive, designing for stability and security can entail trade-offs that may impact performance. I don’t directly address performance in these recommendations.

The below recommendations are written assuming that the sysadmin (you) responsible for the Drupal server and the developer are two different people. If you wear both hats, the recommendations can obviously adapted to fit your dual role.

Recommendations

1. Insist on the latest version of Drupal, and plan to upgrade as Drupal releases come out. Some developers may be hesitant to use the latest (stable) version of Drupal, but Drupal’s security track record dictates this; anything else is not serving the site owner’s interests.
2. Use the latest supported version of PHP. Drupal’s PHP version requirements are more brittle than most applications; the latest version of PHP is almost always the most stable and secure version. Combining the two is obvious: Aggressively track latest version of PHP that Drupal supports. Additionally, use the latest version of your preferred web server and patch as it is updated. Note that this implies a source-based installation of PHP and your web server instead of a package-based installation from your distribution. Carefully consider your build and deployment strategies with an eye towards reproducibility, documentation and ease of roll-back should a problem arise.
3. Isolate Drupal from other applications. Run Drupal on its own system – physical or virtual – and in doing so, reduce operating complexity while limiting collateral damage if your Drupal instance should be compromised. Depending on your backup strategy, don’t overlook the possibility that running a VM may have obvious recovery and forensic advantages over a physical install should your Drupal site get broken into.
4. mod_security – Consider integrating a “web application firewall” such as mod_security directly into your web server. Carefully evaluate the trade-off of additional complexity for enhanced security in your environment.
5. Deploy a host-based firewall for inbound and outbound traffic – This has two functions – prevent your server from being compromised through a hole in a service other than your web server, and limit damage to other systems if/when you are compromised. There’s a decent chance that the only external service you need exposed other than HTTP for your Drupal site is SSH for remote management – and you can probably lock that down to a very limited set of IP addresses. And, in the situation where your server is compromised, outbound rules can reduce the ability of your server to attack other machines: Consider, for example, the effect of an outbound rule on port 25 if an attacker attempts to use your compromised server as a spam bot.
6. Use SELinux – Security-Enhanced Linux provides a the ability to audit and possibly deny actions on your system through mandatory access control policies. You will likely want to run SELinux in “permissive” mode before your site is publicly available, at which point you would switch to “enforcing” mode; “audit2allow” and “audit2why” can be very helpful tools when developing policies. See “man selinux” for more information.
7. Use a Sysadmin Staging Server – The site developer likely has a staging instance of Drupal for testing out their code changes; deploy a similar environment for testing Sysadmin changes, such as PHP updates or the latest version of Drupal with your code. Consider using a software testing framework such as Selenium to automate the tests you run on the sysadmin staging site.

(See bottom of post for an update running similar tests on OpenDNS.)

Methods: I searched Google for keywords that I believed fell somewhere between obscure and common and collected the first ten hostnames printed on the screen. I then used local installations of dig to query a collection of DNS servers for the hostnames’ A records and collected the response times. The different resolvers used were:

• A local BIND installation (127.0.0.1, cache empty) with Comcast Internet connectivity;
• A Comcast DNS server (68.87.69.150) via Comcast Internet connectivity;
• My employer’s internal caching DNS;
• Google (8.8.8.8) via my employer’s Internet connectivity (mostly Level 3);
• Google (8.8.8.8) via Comcast; and
• Google (8.8.8.8) via an Amazon EC2 instance in us-east-1a.

Anticipating a bimodal distribution of results, I assumed high latency responses were cache misses, while low latency responses were cache hits, and categorized results correspondingly.

Limitations: Chiefly, the small number of hostnames queried. Results from a larger group of domains would be more conclusive.

Results: Given in the format of Server/Connectivity: Cache Miss/Cache Hit

• Local BIND server/Comcast: 319ms/0ms
• Comcast/Comcast: 166ms/14ms
• Google/Comcast: no misses/73ms
• Employer/Level 3: 235ms/30ms
• Google/Level 3: 204ms/44ms
• Google/EC2: 190ms/4ms

I concluded that Google/Comcast had no misses by testing another set of obscure hostnames twice each, noting that the first query was slower (~120ms) and the second query was similar in latency to the results above (70ms). (My belief is that I inadvertently pre-populated the cache for Google/Comcast by my tests elsewhere.)

Discussion:

• It’s all about cache hits. Whichever resolver gives you the most cache hits will give you the best performance; cache misses are at least an order of magnitude slower than cache hits. In this extremely limited test, the cache-hits-champion appears to be Google. Excluding Google/Comcast, where I believe I pre-populated Google’s cache, Google had a 50% cache hit rate, while Comcast and my Employer only hit 20%.
• Location, location, location. Secondary to cache hits, the closer the resolver is to you, the better. Looking at the Comcast results, it’s hard to get closer than localhost, and, as seems logical, Comcast’s resolvers have lower cached latency than Google’s. Running a local caching resolver forwarding to Google may be a desirable configuration.
• Resolver behavior matters. Comcast is notorious for poor behavior. It’s reasonable to expect that Google will be mining your DNS query data. Running a slower but directly-controlled local, non-forwarding server may be preferable for privacy and security reasons.

Update: At the suggestion of @tscalzott, I researched OpenDNS’s performance with the same set of hostnames via the same connectivity on their DNS resolvers at 208.67.222.222. This time, however, I queried the A record for each hostname twice in rapid succession to ascertain how many of my queries were served from OpenDNS’s cache. Results are in the format:

DNS Server/Connectivity: Cache miss/Cache hit – cache hit rate

• OpenDNS/Comcast: 218ms/30ms – 20%
• OpenDNS/EC2: 144ms/2ms – 10%
• OpenDNS/Level 3: 230ms/4ms – 40%

Compared to Google, OpenDNS had similar latency for cache misses and lower latency for cache hits, but appears to possibly have a lower cache hit rate. It seems likely that the latency “winner” for each user’s individual situation will depend on where they are on the Internet relative to the nearest Google and OpenDNS installations. Google’s greater cache hit rate suggests it may offer better service, but testing a larger number of hostnames would be necessary before being able to state that with any certainty.

Disclosure: I use Google’s free Apps services to host personal email, and I use their public sites (Search, Reader, News, Analytics, etc.) extensively. I recently attended a Google Apps for the Enterprise dog-and-pony show where I received a number of small tchotchkes; my wife took the notebook, I kept the pen and binder and threw the rest away. My employer uses Postini. I tried OpenDNS briefly several months back, but did not use them long-term because of limitations in my own configuration.

Assumptions: An IMAP interface to your current email, basic comptency at managing DNS, and the ability to run the imapsync Perl script (built via FreeBSD ports in my case, but installation should be straightforward under most UNIX or Linux systems).

0. Ensure your domain registration is up to date and with a reputable registrar (Pro tip: if you have to wade through pages of deceptive and needless up-sells with your registrar, they’re not reputable), and fully independent from anything you have set up with Google. Ensure your DNS configuration is completely correct; if you’re unsure whether or not it is, buy a membership at DNSstuff, run its tests on your domains, and fix the relevant problems. If you don’t fully host your own DNS – and chances are, if you’re outsourcing your email, you probably don’t host your own DNS – I recommend using two independent DNS providers. Hosted DNS service that’s worth spending money with will be able to function as a secondary and allow zone transfers – and, while you’re at it, make sure your chosen provider supports DNS NOTIFY – it’ll make changing your MX records in a timely fashion that much easier. Whatever you do, don’t use your registrar’s DNS servers.

You occasionally see sad tales of Google Apps woe which could have been mitigated by independent domain registration and a moderate knowledge of DNS. Don’t risk becoming another bitter messageboard-posting statistic.

In summary: Don’t understand DNS or don’t have time for it? Stop reading now; either hire a consultant to do the work, or accept that outsourced email probably isn’t right for you.

0.1. This goes hand-in-hand with step 0 above: Have an off-Google backup system for your email – one that runs unattended, automatically, just like a “real” backup system would be. Google has its strong points, but customer service for their free products isn’t one of them. If Google should update their unofficial motto to a more succinct “be evil” you want to have an “out” and the ability to migrate your mail elsewhere. Imapsync, linked to above and described below, can be adapted for this purpose.

0.2 Install and test imapsync before beginning this process. This is the best tool that I’ve found for syncing mailboxes between servers; all others I tried were unreliable, and documentation below will be specific to it. In theory, another tool could work – IMAP is IMAP and a standard – but I leave using other migration tools as an exercise for the reader.

1. Drop your MX record TTL to a suitably low value; I used 60 seconds during my migration. Let the older, presumably greater TTL age off prior to making any further changes to your MX records (as in step 7 below).

2. Sign up your domain for Google Apps, and use DNS to verify control.

3. Create users for the domain, and accept terms. A best practice would be to use different passwords on Gmail from what users currently have, and do note that you will need both old and new passwords for the synchronization step. (If you are currently running a Cyrus IMAP server with sasldb, passwords can be extracted by running “db3_dump185 -p sasldb2.db” as a user that can read sasldb.)

4. In the Google Apps control panel, create distribution lists and aliases (“nicknames”) for your users as appropriate.

5. For each Gmail mailbox, enable IMAP: Settings > Forwarding and POP/IMAP: Enable IMAP.

6. If appropriate (and assuming you have other users in your domain), have users verify their logins and familiarize themselves with Gmail’s options.

7. Change your MX records to point to Google; keep your TTL low in case you need to change back to your old servers.

8. Verify delivery to your new Gmail mailboxes by sending test messages from a third-party address.

9. Set an appropriate SPF record for your domain in DNS.

10. Sync your old mailboxes with your new; I recommend running imapsync directly on your mail server, if possible, for best performance. Example imapsync session for a Cyrus inbox:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 user@example.com --password2 [...] --ssl2 --folder INBOX

(Insert passwords as appropriate.)

Example imapsync session for a Cyrus folder other than the inbox:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 user@example.com --password2 [...] --ssl2 --folder INBOX.saved-messages

Note that Gmail has certain reserved labels that you cannot sync directly to, such as “Sent”. In this case, you’ll need to use the “regextrans2″ flag to sync to a different folder:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 use@example.com --password2 [...] --ssl2 --folder INBOX.Sent --regextrans2 's/Sent/Old-Sent/'

11. After double checking that your old MX record’s TTL has passed, and no new mail is being delivered to your old mail server, decommission it as necessary.

Crash Symptoms

A CentOS 4.x or 5.x guest will crash with a message similar to the following on its console:

CentOS 4.x:

[<f883b299>] .text.lock.scsi_error+0x19/0x34 [scsi_mod]
[<f88c19ce>] mptscsih_io_done+0x5ee/0x608 [mptscsi] (…)
[<c02de564>] common_interrupt+0x18/0x20
[<c02ddb54>] system_call+0x0/0x30

CentOS 5.x:

RIP  [<ffffffff8014c562>] list_del+0x48/0x71 RSP <ffffffff80425d00> <0>Kernel Panic - not syncing: Fatal exception

A hard reset (i.e. pressing the reset button on the VM’s console) is required to reboot the guest.

Further Details

Five different VMs have encountered this issue, running at a mix of close-to-current CentOS 4.x and 5.x patch levels. Guest kernel versions when the crash occurred were 2.6.18-128.7.1.el5 and 2.6.18-128.1.10.el5 (5.x) and 2.6.9-89.0.9.ELsmp (4.x). Memory allocations on affected guests range from 512MB to 3072MB. Notably, all affected VMs are using SMP – each has 2 vCPUs – having been created before our in-house practices followed VMware guidelines and discouraged use of SMP on ESX guests when unnecessary. One VM was created via P2V; the rest were created de novo on virtual hardware.

All crashes have happened on a single node in an ESX 3.5 HA cluster composed of four Dell PowerEdge 1950s. ESX hosts have tracked the latest VMware patches closely. COS memory on the ESX host in question was increased from the default to 800MB prior to the three most recent crashes; in other words, the COS memory increase appears to have had no effect on the crashes. DRS is in use, set to “fully automated” and “apply recommendations with three or more stars” and no virtual machine rules have been created to control DRS host placement.

All guests are on the same NFS data store, served from a NetApp filer running ONTAP 7.2.x. One guest had its vmswap placed separately on an iSCSI data store; the rest have their swap stored on NFS with the VM. No log messages were seen on the filer during the event, although the a log message similar to the following has been seen several times on the ESX host:

vmkernel: 43:07:27:51.725 cpu2:2185)WARNING: NFS: 4590: Can't find call with serial number -2146566055

Curiously, all crashes have happened in the evening, in the 10 o’clock hour, after nightly backups have been completed. Backups are created using a combination of VMware and NetApp snapshots via a script similar to one detailed on vmwaretips.com. No substantial load or latency has been recorded on the NetApp during the crashes, and weeks have passed between events.

Speculation

Explanations I’m leaning towards, ranked by my judgment of their likelihood:

1) Hardware issue. Assuming a random distribution of VMs – recall that DRS is in use and no virtual machine rules are in place – the odds of all five crashes happening on one host out of four are slim: 1 in 1024. Unfortunately, by all measures we’ve used, including the VI Client’s “Health Status” and Dell OMSA, there are no hardware issues with the host.

Further, the distribution of VMs is not truly random. DRS migrations are infrequent in this cluster, and the largest determinant of guest location is migration following hosts being placed into maintenance mode for patching.

If it is a hardware issue, it’s subtle, and possibly only brought to the fore by the following issues.

2) Red Hat Enterprise Linux bug – which, by extension, is typically equivalent to a CentOS bug. In fact, this issue appears to have been raised with Red Hat already in bugs 197158 and 228108 – but, according the bug reports, the issue is resolved, and the patches have since been ported downstream to CentOS. However, perhaps the issue is not truly resolved – see comment 35 in 228108.

3) vSMP Bug. The majority of our Linux VMs are uniprocessor and appear so far to be immune to this issue; it is striking that the crash has only occurred on dual processor guests. I cannot articulate a mechanism for multiple vCPUs causing this crash, however.

4) NetApp issue. This appears to be a storage issue at some level, considering the mptscsi and NFS messages noted above, so performance of the NetApp filer would be a natural place for further investigation. However, we monitor the performance of our filer relatively closely, using the ONTAP SDK and Cacti, and nothing unusual was recorded during any crash. It seems unusual that all VMs reside on the same data store, but that data store shares an aggregate with multiple other unaffected data stores, and several LUNs are served from the same aggregate to non-ESX machines without complaint.

I have not yet opened a case with VMware on this issue – or Dell, or NetApp, for that matter – but if and when I do, I’ll update here to the extent possible.

Update 11/20/2009: Prompted by a helpful comment from nate below, I looked up and verified the NFS settings across the cluster. They are the same across all hosts, and are as follows:

NFS.IndirectSend 0
NFS.DiskFileLockUpdate 10
NFS.LockUpdateTimeout 5
NFS.LockRenewMaxFailureNumber 3
NFS.LockDisable 0
NFS.HeartbeatFrequency 12
NFS.HeartbeatTimeout 5
NFS.HeartbeatDelta 5
NFS.HeartbeatMaxFailures 10
NFS.MaxVolumes 8
NFS.SendBufferSize 264
NFS.ReceiveBufferSize 128
NFS.VolumeRemountFrequency 30
NFS.UDPRetransmitDelay 700

The only values that are changed from default are HeartbeatFrequency and HeartbeatMaxFailures, to match NetApp’s recommendations in TR-3428.

> rpm -q slocate
slocate-2.7-13.el4.8.i386

or

> rpm -q mlocate
mlocate-0.15-1.el5.2.x86_64

If so, multiple RHEL VMs plus mlocate or slocate may be adding up to an array-crushing 4:02am shared storage load and latency spike for you. Before being addressed, this spike was bad enough at my place of employment (when combined with a NetApp Sunday-morning disk scrub) to cause a Windows VM to crash with I/O errors. Ouch.

Details and ideas for resolution:

By default, a line in /etc/crontab runs the scripts within /etc/cron.daily at 4:02am each morning:

02 4 * * * root run-parts /etc/cron.daily

One of those scripts – mlocate.cron or slocate.cron, depending on your OS version – launches updatedb; as the man page says, “updatedb creates or updates a database used by locate(1).” (The “locate” binary is a filesystem search tool, see “man locate” for more information.) Updatedb refreshes its database by walking the filesystem, generating a fair amount of I/O on a single system. Imagine upwards of thirty of these running in parallel through VMDKs on one shared storage system carrying out internal maintenance at the same time, and you’re pretty much picturing the problem my employer had.

I see three options for addressing this issue:

1) Uninstall mlocate or slocate. If you don’t currently use “locate” and you’re not interested in learning to use a tool that will likely make you more effective at your job (again, see “man locate”), this is probably the best option. (Yeah, I know, people that fit this bill generally don’t read blogs more technical than this one, so I could probably have skipped it here. Consider it an option for completeness, or if you really need to strip down an install.)

2) Disable the scheduled job by removing mlocate.cron or slocate.cron from /etc/cron.daily. This keeps locate available for your use, but requires that you update locate’s database ad-hoc and interactively by running the following as root:

# updatedb

This will take a few minutes to return, depending on the size of your file systems.

I don’t recommend this option either; at least it doesn’t fit the way I work. I often find myself using locate in high-pressure situations in which I need to quickly get a file location on a system. Waiting minutes for updatedb to return is extra painful when every second counts.

3) Stagger when updatedb runs by inserting a random delay into the script.. This is my preferred alternative; locate’s database is kept current automatically, and your storage doesn’t have to bear a sudden spike in load. I implemented this by adding the lines in bold (lines 2-7 if your browser doesn’t display the bold text clearly):

#!/bin/sh
# sleep up to two hours before launching job:
value=$RANDOM
while [ $value -gt 7200 ] ; do
value=$RANDOM
done
sleep $value
nodevs=$(< /proc/filesystems awk '$1 == "nodev" { print $2 }')
renice +19 -p $$ >/dev/null 2>&1
/usr/bin/updatedb -f "$nodevs"


The added code inserts a pseudo-random sleep delay of up to two hours before updatedb runs, with the key being the built-in Bash function $RANDOM. In our environment, this removed a 2000 IOPS spike at 4:02am, and eliminated a corresponding jump in filer latency. Obviously, adjust the delay period as appropriate for your environment. Additionally, be sure to add this change to your configuration management or installation management tools so that all of your RHEL and RHEL-derived VMs get the updated script.

Using $RANDOM to avoid this variant of the thundering herd problem also works nicely for a range of similar problems; I believe I first saw it at Moundalexis.com.

(This problem may apply to other Linux distributions being run as VMs, and FreeBSD does something equivalent – weekly – with /etc/periodic/weekly/310.locate. A similar solution can be applied to these environments, if necessary.)

If you suspect that your system has almost used all of its free space, or if you use thin provisioning, you should check the amount of space in use by each aggregate. If any aggregate is 97 percent full or more, do not proceed with the upgrade until you have used the Upgrade Advisor or aggrSpaceCheck tools to determine your system capacity and plan your upgrade.

Upgrade Advisor is a great tool, and I heartily recommend you use it for your upgrade. However, it doesn’t give you a lot of visibility into what’s being checked for here. Lucky for us, NetApp offers an alternative tool: aggrSpaceCheck (NOW login required).

AggrSpaceCheck, as written, relies on you having rsh turned on for access to the filer – something that you probably locked down when you were still wearing acid-washed jeans. Assuming you don’t have rsh access, you’ll see an error like this when you attempt to run aggrSpaceCheck:

> perl aggrSpaceCheck.pl -filer toaster01

Could not retrieve Aggregate free space. Could not get any aggregates in this filer


The fix is easy, however, if you’re using SSH: Edit the aggrSpaceCheck.pl file, replacing “rsh” with “ssh” (you only need to actually edit it in the line where “$remCmd” is defined, but changing rsh to ssh elsewhere won’t hurt). You will be prompted for root’s SSH password repeatedly – once for each command run remotely on the filer:

> perl aggrSpaceCheck.pl -filer toaster01

Aggregate aggr0 requires 54.18GB for volume metadata; 137.76GB is available.
Aggregate aggr0 has enough free space for you to upgrade to
Data ONTAP 7.3 or later.


There’s a solution for this too, of course: Enable SSH key pair authentication from your management host to your filer – no more password prompts:

> perl aggrSpaceCheck.pl -filer toaster01

Aggregate aggr0 requires 54.18GB for volume metadata; 137.76GB is available.
Aggregate aggr0 has enough free space for you to upgrade to
Data ONTAP 7.3 or later.


Beginning with Data ONTAP 7.3.1, FAS2020 systems support aggregates up to 16 TB raw capacity,
provided that the root volume is hosted in a dedicated aggregate (that is, one that contains only the root
volume and no user data).

The release notes go on to point out an alternative to the dedicated root aggregate – having two spare disks per controller.

It’s nice to see the FAS2020 finally getting a maximum aggregate size on par with the rest of NetApp’s product line. However, in an era where 2TB drives are available from Western Digital – and presumably other manufacturers before too long – ONTAP’s 16TB aggregate limit grows increasingly anachronistic.

We recently acquired SnapManager for Exchange (SME) at my place of employment. We have an existing NetApp deployment consisting of two primary filers in a SnapVault arrangement with a third filer. The SME install is part of an upgrade from Exchange 2003 (on DAS) to 2007 (on Fibre Channel storage).

What we missed prior to purchasing SME: If you want to use SnapVault with SME, you need two additional pieces of software: Protection Manager and NetApp Management Console (part of DataFabric Manager, apparently). Here’s what p. 408 of the SnapManager® 5.0 for Microsoft® Exchange Installation and Administration Guide (NOW login required) says:

The following are the software dependencies for integrating SnapManager with
data set and SnapVault:

◆ Protection Manager 3.7 and later
◆ NetApp Management Console 3.7 and later
◆ SnapDrive for Windows 6.0 and later
◆ Data ONTAP 7.3 or later

Wish I’d known that sooner.

(This is the point where some random NetApp fanboy pops down to the comments and fires off something about how NetApp is the greatest storage company ever, and if I’d done appropriate due diligence, I wouldn’t have missed this requirement. My advice: Spare us, smart guy. I’m writing this post to make it easier for other NetApp customers to do their “due diligence”.)

Here’s our in-house procedure for recovery; note that we do not have FlexClone licensed on our filers.

Prerequisites

• An existing VMware ESX infrastructure, connected to a NetApp filer NFS datastore; SnapRestore speeds the recovery process but is not mandatory – see discussion below.
• A backup script or system which coordinates VMware snapshots with NetApp snapshots – perhaps something along the lines of Rick Scherer’s script.
• A dedicated Linux restore VM, at a similar version level to the rest of your Linux VM infrastructure. This VM should have LVM support, but should not have any volume groups (VGs) or logical volumes (LVs) configured – volume group and logical volume names on the VMDK you are restoring from must not conflict with VGs and LVs already in use on the restore system; the simplest way to guarantee this is to simply not have any VGs or LVs.

Restore Procedure

• Restore the VMDK file from the appropriate snapshot to a new location in the datastore. With SnapRestore, this can be done as follows (one line in the filer CLI, restoring from snapshot sv_daily.0 to a new file – again, be extremely careful not to overwrite the current version of the VMDK in your datastore, consider restoring to an entirely different directory in the FlexVol):

  snap restore -t file -s sv_daily.0
-r /vol/vmware04_sis/system.example.com/system.example.com-restore.vmdk
/vol/vmware04_sis/system.example.com/system.example.com.vmdk

  Follow the prompts, verifying the restore path is correct and is not the path to your existing VMDK. Do the same for the flat VMDK file (again, one line, and, as before, use caution to make sure you do not clobber an existing file):

  snap restore -t file -s sv_daily.0
-r /vol/vmware04_sis/system.example.com/system.example.com-restore-flat.vmdk
/vol/vmware04_sis/system.example.com/system.example.com-flat.vmdk

  Without SnapRestore, you can simply mount the NFS export of the datastore on a Linux machine and use “cp” to copy the files out of the snapshot. For flat VMDK files, expect this copy to run for a substantial amount of time compared the nearly-instant recovery SnapRestore offers.

• Manually edit the line below “# Extent description” in the recovered .vmdk file to match the path to the recovered flat VMDK. In this case, it would look something like this:
  # Extent description
RW 20971520 VMFS "system.example.com-restore-flat.vmdk"
• Attach the recovered VMDK to your powered-off restore host. Boot the restore host.
• Once your restore host is up, use “pvscan”, “vgscan” and “lvscan” (each without arguments) as root to examine available LVM components. Then, use the “lvchange” command to activate the necessary volume group (in this case, “VolGroup00″):
  # lvchange -ay VolGroup00
• Mount the appropriate logical volume – for example, LogVol00 in VolGroup00:
  mount -o ro /dev/VolGroup00/LogVol00 /mnt
  Restore files by copying them out of /mnt.

Cleanup

• Shut down the Linux restore host.
• Remove the recovery VMDK – the files restored with SnapRestore or by “cp” above – from the restore host in the VMware Infrastructure Client.
• Delete the recovery .vmdk and -flat.vmdk files in the NFS datastore. Don’t screw up here: Be sure to delete the recovery files only, not the working VMDK.

I see echoes of Toyota teaching its Toyota Production System in Google’s recent release of information about their data centers. Relatively straightforward concepts – the challenge is in adapting your existing systems to them.