netapp01> lun show /vol/nexenta01/lun01/lun
        /vol/nexenta01/lun01/lun      10g (10737418240)   (r/w, online, mapped)

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no

root@nexenta01:/volumes# stmsboot -L
stmsboot: MPXIO disabled
root@nexenta01:/volumes# stmsboot -e -D fp
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y)
updating //platform/i86pc/boot_archive
updating //platform/i86pc/amd64/boot_archive
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y)

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes

Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

root@nexenta01:/volumes# stmsboot -L
stmsboot: No STMS devices have been found

netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
netapp01> igroup set nexenta01 alua yes
netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee(logged in on: 0b, 0d, vtic)
        ALUA: Yes

nmc@nexenta01:/$ lunsync -r
Cleanup obsolete (dangling) device links?  Yes
Re-scanning HBAs... done.
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c4t6*469d0  sd9       disk         10GB                  no      mpxio  60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

root@nexenta01:/volumes# stmsboot -L
non-STMS device name                    STMS device name
------------------------------------------------------------------
/dev/rdsk/c3t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0

Hat Tip to @complex on Twitter.

Reference: Is it possible to use I/O multipathing? How?

See also: f5vpn-login.py, described here, and brought to my attention by sh4k3sph3r3. A CLI FirePass client is quite likely a better solution than separate browser instances, etc.

Preliminaries: Although the F5 FirePass SSL VPN product supports Linux, as best as I can tell, that support is somewhat limited: My understanding is that they officially claim support for 32-bit installs only, and they do not appear to track new distribution releases particularly aggressively. F5 has also been somewhat slow in supporting new browser versions: They announced support for Firefox 3 on October 6, 2008, nearly four months after its release and with only two months to go before Firefox 2 was end-of-lifed. For Firefox 3.6 support, a comment on the post linked above states that you need to request a special hot fix from F5 (which my site has not applied). There is no Google Chrome support that I am aware of.

Further, F5’s automated client installation tools have unfortunately never worked for me on Linux, even when the architecture and browser are in their support matrix. The manual download instruction links are also broken on the FirePass install I connect to.

Solution: Install a dedicated, 32-bit version of Firefox in a supported version; create a single-purpose Firefox profile for VPN use. Add the FirePass client to that browser and the operating system.

For the Firefox install, follow the “Manual Installation” instructions from the Ubuntu Community Documentation site. Install version 3.5 if your site does not have the hotfix mentioned above.

Be sure to create a new Firefox profile in your account for use with the FirePass; however, I recommend modifying the script in the Ubuntu documentation to automatically take you to your FirePass site (https://firepass.example.com/ for the purposes of this post):

#!/bin/bash
exec "\$HOME/firefox/firefox" -P mozilla-build https://firepass.example.com/

Next, download the client components from your F5 site; again, assuming firepass.example.com, retrieve and save:

https://firepass.example.com/vdesk/vpn/nogzip/downloads.php/linux/np_F5_SSL_VPN.so

and

https://firepass.example.com/vdesk/vpn/nogzip/downloads.php/linux/SSLVpn.tgz

Move np_F5_SSL_VPN.so to the plugins directory of the new Firefox installation – ~/firefox/plugins if following the Ubuntu documentation. Based on file layout, it appears that F5 intended for you to extract SSLVpn.tgz at the root of your file system. Instead of following this bad practice, in scratch space and as root, extract the SSLVpn.tgz tarball and manually move the files into place:

cp SSLVpn.tgz /tmp
cd /tmp
sudo tar -xvpzf SSLVpn.tgz
# inspect extracted files here...
cd /usr/local/lib
mkdir -p F5Networks/SSLVPN
cd /tmp/usr/local/lib/F5Networks/SSLVPN
cp -Rp etc svpn var /

Using the bash script above, you should now be able to launch your purpose-built FirePass browser installation and have it “just work” for Network Access. Good luck!

#!/usr/local/bin/bash

# Path to ZFS executable:
ZFS=/sbin/zfs

# Parse arguments:
TARGET=$1
SNAP=$2
COUNT=$3

# Function to display usage:
usage() {
    scriptname=`/usr/bin/basename $0`
    echo "$scriptname: Take and rotate snapshots on a ZFS file system"
    echo
    echo "  Usage:"
    echo "  $scriptname target snap_name count"
    echo
    echo "  target:    ZFS file system to act on"
    echo "  snap_name: Base name for snapshots, to be followed by a '.' and"
    echo "             an integer indicating relative age of the snapshot"
    echo "  count:     Number of snapshots in the snap_name.number format to"
    echo "             keep at one time.  Newest snapshot ends in '.0'."
    echo
    exit
}

# Basic argument checks:
if [ -z $COUNT ] ; then
    usage
fi

if [ ! -z $4 ] ; then
    usage
fi

# Snapshots are number starting at 0; $max_snap is the highest numbered
# snapshot that will be kept.
max_snap=$(($COUNT -1))

# Clean up oldest snapshot:
if [ -d /${TARGET}/.zfs/snapshot/${SNAP}.${max_snap} ] ; then
    $ZFS destroy -r ${TARGET}@${SNAP}.${max_snap}
fi

# Rename existing snapshots:
dest=$max_snap
while [ $dest -gt 0 ] ; do
    src=$(($dest - 1))
    if [ -d /${TARGET}/.zfs/snapshot/${SNAP}.${src} ] ; then
	$ZFS rename -r ${TARGET}@${SNAP}.${src} ${TARGET}@${SNAP}.${dest}
    fi
    dest=$(($dest - 1))
done

# Create new snapshot:
$ZFS snapshot -r ${TARGET}@${SNAP}.0

From the command line, call the script something like the following:

./zfs-snapshot.sh tank weekly 5

This would take a recursive snapshot of the “tank” zpool with the basename weekly, rotating through five snapshots with names “weekly.0″ through “weekly.4″. This allows you to implement a snapshot scheme approximately similar to NetApp’s hourly-daily-weekly scheme, if you like. Because my FreeBSD workstation isn’t on 24×7, I run hourly snapshots out of /etc/crontab:

# Automated ZFS backups (hourly):
0 * * * * root /root/bin/zfs-snapshot.sh tank hourly 25

And daily/weekly/monthly snapshots out of /usr/local/etc/anacrontab (from the sysutils/anacron port):

PATH=/bin:/sbin:/usr/bin:/usr/sbin

# days		make sure the command is executed at least every 'days' days
# delay		delay in minutes, before a command starts
# id		unique id of a command

# days	delay	id		command
1	5	daily		periodic daily
1	10	daily_snap	/root/bin/zfs-snapshot.sh tank daily 8
7	15	weekly		periodic weekly
7	30	weekly_snap	/root/bin/zfs-snapshot.sh tank weekly 5
30	60	monthly		periodic monthly
30	90	monthly_snap	/root/bin/zfs-snapshot.sh tank monthly 13

(Of course, this isn’t as cool as the Gnome-integrated Time Slider in OpenSolaris, but it scratches my itch sufficiently.)

Despite its many desirable features and continuing popularity, Drupal is not without its shortcomings, as many readers are also likely aware. Although Drupal has an active and responsive security team, the software has a long track record of requiring frequent security patches – Secunia has seven 2009 advisories for Drupal 6.x listed as of this writing. Although by its nature an apples-to-oranges comparison, this ranks Drupal behind similarly large and complex PHP projects such as Wordpress 2.x (5) and Gallery 2.x (0) – and the number for Drupal does not include dozens of additional advisories for Drupal modules. Further, Drupal has struggled and lagged with support for PHP 5.3.x, suggesting to this outside observer that the project is having difficulties maintaining its codebase.

All that being said, I do not personally believe that the above issues rule out using Drupal; the benefits outweigh the shortcomings. So, assuming the question is not whether to deploy Drupal, but how to do so most securely and efficiently, my recommendations from a systems administration perspective are below.

Goals

The recommendations described stem from three goals for a Drupal installation:

1. Security – Avoid compromise of your site and system.
2. Flexibility – Create an environment that allows for easy modification of the OS, server and application, and provides for straightforward roll-back of those changes should those need arise.
3. Stability and Security over Performance – Although not mutually exclusive, designing for stability and security can entail trade-offs that may impact performance. I don’t directly address performance in these recommendations.

The below recommendations are written assuming that the sysadmin (you) responsible for the Drupal server and the developer are two different people. If you wear both hats, the recommendations can obviously adapted to fit your dual role.

Recommendations

1. Insist on the latest version of Drupal, and plan to upgrade as Drupal releases come out. Some developers may be hesitant to use the latest (stable) version of Drupal, but Drupal’s security track record dictates this; anything else is not serving the site owner’s interests.
2. Use the latest supported version of PHP. Drupal’s PHP version requirements are more brittle than most applications; the latest version of PHP is almost always the most stable and secure version. Combining the two is obvious: Aggressively track latest version of PHP that Drupal supports. Additionally, use the latest version of your preferred web server and patch as it is updated. Note that this implies a source-based installation of PHP and your web server instead of a package-based installation from your distribution. Carefully consider your build and deployment strategies with an eye towards reproducibility, documentation and ease of roll-back should a problem arise.
3. Isolate Drupal from other applications. Run Drupal on its own system – physical or virtual – and in doing so, reduce operating complexity while limiting collateral damage if your Drupal instance should be compromised. Depending on your backup strategy, don’t overlook the possibility that running a VM may have obvious recovery and forensic advantages over a physical install should your Drupal site get broken into.
4. mod_security – Consider integrating a “web application firewall” such as mod_security directly into your web server. Carefully evaluate the trade-off of additional complexity for enhanced security in your environment.
5. Deploy a host-based firewall for inbound and outbound traffic – This has two functions – prevent your server from being compromised through a hole in a service other than your web server, and limit damage to other systems if/when you are compromised. There’s a decent chance that the only external service you need exposed other than HTTP for your Drupal site is SSH for remote management – and you can probably lock that down to a very limited set of IP addresses. And, in the situation where your server is compromised, outbound rules can reduce the ability of your server to attack other machines: Consider, for example, the effect of an outbound rule on port 25 if an attacker attempts to use your compromised server as a spam bot.
6. Use SELinux – Security-Enhanced Linux provides a the ability to audit and possibly deny actions on your system through mandatory access control policies. You will likely want to run SELinux in “permissive” mode before your site is publicly available, at which point you would switch to “enforcing” mode; “audit2allow” and “audit2why” can be very helpful tools when developing policies. See “man selinux” for more information.
7. Use a Sysadmin Staging Server – The site developer likely has a staging instance of Drupal for testing out their code changes; deploy a similar environment for testing Sysadmin changes, such as PHP updates or the latest version of Drupal with your code. Consider using a software testing framework such as Selenium to automate the tests you run on the sysadmin staging site.

(See bottom of post for an update running similar tests on OpenDNS.)

Methods: I searched Google for keywords that I believed fell somewhere between obscure and common and collected the first ten hostnames printed on the screen. I then used local installations of dig to query a collection of DNS servers for the hostnames’ A records and collected the response times. The different resolvers used were:

• A local BIND installation (127.0.0.1, cache empty) with Comcast Internet connectivity;
• A Comcast DNS server (68.87.69.150) via Comcast Internet connectivity;
• My employer’s internal caching DNS;
• Google (8.8.8.8) via my employer’s Internet connectivity (mostly Level 3);
• Google (8.8.8.8) via Comcast; and
• Google (8.8.8.8) via an Amazon EC2 instance in us-east-1a.

Anticipating a bimodal distribution of results, I assumed high latency responses were cache misses, while low latency responses were cache hits, and categorized results correspondingly.

Limitations: Chiefly, the small number of hostnames queried. Results from a larger group of domains would be more conclusive.

Results: Given in the format of Server/Connectivity: Cache Miss/Cache Hit

• Local BIND server/Comcast: 319ms/0ms
• Comcast/Comcast: 166ms/14ms
• Google/Comcast: no misses/73ms
• Employer/Level 3: 235ms/30ms
• Google/Level 3: 204ms/44ms
• Google/EC2: 190ms/4ms

I concluded that Google/Comcast had no misses by testing another set of obscure hostnames twice each, noting that the first query was slower (~120ms) and the second query was similar in latency to the results above (70ms). (My belief is that I inadvertently pre-populated the cache for Google/Comcast by my tests elsewhere.)

Discussion:

• It’s all about cache hits. Whichever resolver gives you the most cache hits will give you the best performance; cache misses are at least an order of magnitude slower than cache hits. In this extremely limited test, the cache-hits-champion appears to be Google. Excluding Google/Comcast, where I believe I pre-populated Google’s cache, Google had a 50% cache hit rate, while Comcast and my Employer only hit 20%.
• Location, location, location. Secondary to cache hits, the closer the resolver is to you, the better. Looking at the Comcast results, it’s hard to get closer than localhost, and, as seems logical, Comcast’s resolvers have lower cached latency than Google’s. Running a local caching resolver forwarding to Google may be a desirable configuration.
• Resolver behavior matters. Comcast is notorious for poor behavior. It’s reasonable to expect that Google will be mining your DNS query data. Running a slower but directly-controlled local, non-forwarding server may be preferable for privacy and security reasons.

Update: At the suggestion of @tscalzott, I researched OpenDNS’s performance with the same set of hostnames via the same connectivity on their DNS resolvers at 208.67.222.222. This time, however, I queried the A record for each hostname twice in rapid succession to ascertain how many of my queries were served from OpenDNS’s cache. Results are in the format:

DNS Server/Connectivity: Cache miss/Cache hit – cache hit rate

• OpenDNS/Comcast: 218ms/30ms – 20%
• OpenDNS/EC2: 144ms/2ms – 10%
• OpenDNS/Level 3: 230ms/4ms – 40%

Compared to Google, OpenDNS had similar latency for cache misses and lower latency for cache hits, but appears to possibly have a lower cache hit rate. It seems likely that the latency “winner” for each user’s individual situation will depend on where they are on the Internet relative to the nearest Google and OpenDNS installations. Google’s greater cache hit rate suggests it may offer better service, but testing a larger number of hostnames would be necessary before being able to state that with any certainty.

Disclosure: I use Google’s free Apps services to host personal email, and I use their public sites (Search, Reader, News, Analytics, etc.) extensively. I recently attended a Google Apps for the Enterprise dog-and-pony show where I received a number of small tchotchkes; my wife took the notebook, I kept the pen and binder and threw the rest away. My employer uses Postini. I tried OpenDNS briefly several months back, but did not use them long-term because of limitations in my own configuration.

Assumptions: An IMAP interface to your current email, basic comptency at managing DNS, and the ability to run the imapsync Perl script (built via FreeBSD ports in my case, but installation should be straightforward under most UNIX or Linux systems).

0. Ensure your domain registration is up to date and with a reputable registrar (Pro tip: if you have to wade through pages of deceptive and needless up-sells with your registrar, they’re not reputable), and fully independent from anything you have set up with Google. Ensure your DNS configuration is completely correct; if you’re unsure whether or not it is, buy a membership at DNSstuff, run its tests on your domains, and fix the relevant problems. If you don’t fully host your own DNS – and chances are, if you’re outsourcing your email, you probably don’t host your own DNS – I recommend using two independent DNS providers. Hosted DNS service that’s worth spending money with will be able to function as a secondary and allow zone transfers – and, while you’re at it, make sure your chosen provider supports DNS NOTIFY – it’ll make changing your MX records in a timely fashion that much easier. Whatever you do, don’t use your registrar’s DNS servers.

You occasionally see sad tales of Google Apps woe which could have been mitigated by independent domain registration and a moderate knowledge of DNS. Don’t risk becoming another bitter messageboard-posting statistic.

In summary: Don’t understand DNS or don’t have time for it? Stop reading now; either hire a consultant to do the work, or accept that outsourced email probably isn’t right for you.

0.1. This goes hand-in-hand with step 0 above: Have an off-Google backup system for your email – one that runs unattended, automatically, just like a “real” backup system would be. Google has its strong points, but customer service for their free products isn’t one of them. If Google should update their unofficial motto to a more succinct “be evil” you want to have an “out” and the ability to migrate your mail elsewhere. Imapsync, linked to above and described below, can be adapted for this purpose.

0.2 Install and test imapsync before beginning this process. This is the best tool that I’ve found for syncing mailboxes between servers; all others I tried were unreliable, and documentation below will be specific to it. In theory, another tool could work – IMAP is IMAP and a standard – but I leave using other migration tools as an exercise for the reader.

1. Drop your MX record TTL to a suitably low value; I used 60 seconds during my migration. Let the older, presumably greater TTL age off prior to making any further changes to your MX records (as in step 7 below).

2. Sign up your domain for Google Apps, and use DNS to verify control.

3. Create users for the domain, and accept terms. A best practice would be to use different passwords on Gmail from what users currently have, and do note that you will need both old and new passwords for the synchronization step. (If you are currently running a Cyrus IMAP server with sasldb, passwords can be extracted by running “db3_dump185 -p sasldb2.db” as a user that can read sasldb.)

4. In the Google Apps control panel, create distribution lists and aliases (“nicknames”) for your users as appropriate.

5. For each Gmail mailbox, enable IMAP: Settings > Forwarding and POP/IMAP: Enable IMAP.

6. If appropriate (and assuming you have other users in your domain), have users verify their logins and familiarize themselves with Gmail’s options.

7. Change your MX records to point to Google; keep your TTL low in case you need to change back to your old servers.

8. Verify delivery to your new Gmail mailboxes by sending test messages from a third-party address.

9. Set an appropriate SPF record for your domain in DNS.

10. Sync your old mailboxes with your new; I recommend running imapsync directly on your mail server, if possible, for best performance. Example imapsync session for a Cyrus inbox:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 user@example.com --password2 [...] --ssl2 --folder INBOX

(Insert passwords as appropriate.)

Example imapsync session for a Cyrus folder other than the inbox:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 user@example.com --password2 [...] --ssl2 --folder INBOX.saved-messages

Note that Gmail has certain reserved labels that you cannot sync directly to, such as “Sent”. In this case, you’ll need to use the “regextrans2″ flag to sync to a different folder:

imapsync --host1 mail.example.com --port1 143 --user1 user@example.com --password1 [...] --prefix1 INBOX. --host2 imap.gmail.com --port2 993 --user2 use@example.com --password2 [...] --ssl2 --folder INBOX.Sent --regextrans2 's/Sent/Old-Sent/'

11. After double checking that your old MX record’s TTL has passed, and no new mail is being delivered to your old mail server, decommission it as necessary.

Crash Symptoms

A CentOS 4.x or 5.x guest will crash with a message similar to the following on its console:

CentOS 4.x:

[<f883b299>] .text.lock.scsi_error+0x19/0x34 [scsi_mod]
[<f88c19ce>] mptscsih_io_done+0x5ee/0x608 [mptscsi] (…)
[<c02de564>] common_interrupt+0x18/0x20
[<c02ddb54>] system_call+0x0/0x30

CentOS 5.x:

RIP  [<ffffffff8014c562>] list_del+0x48/0x71 RSP <ffffffff80425d00> <0>Kernel Panic - not syncing: Fatal exception

A hard reset (i.e. pressing the reset button on the VM’s console) is required to reboot the guest.

Further Details

Five different VMs have encountered this issue, running at a mix of close-to-current CentOS 4.x and 5.x patch levels. Guest kernel versions when the crash occurred were 2.6.18-128.7.1.el5 and 2.6.18-128.1.10.el5 (5.x) and 2.6.9-89.0.9.ELsmp (4.x). Memory allocations on affected guests range from 512MB to 3072MB. Notably, all affected VMs are using SMP – each has 2 vCPUs – having been created before our in-house practices followed VMware guidelines and discouraged use of SMP on ESX guests when unnecessary. One VM was created via P2V; the rest were created de novo on virtual hardware.

All crashes have happened on a single node in an ESX 3.5 HA cluster composed of four Dell PowerEdge 1950s. ESX hosts have tracked the latest VMware patches closely. COS memory on the ESX host in question was increased from the default to 800MB prior to the three most recent crashes; in other words, the COS memory increase appears to have had no effect on the crashes. DRS is in use, set to “fully automated” and “apply recommendations with three or more stars” and no virtual machine rules have been created to control DRS host placement.

All guests are on the same NFS data store, served from a NetApp filer running ONTAP 7.2.x. One guest had its vmswap placed separately on an iSCSI data store; the rest have their swap stored on NFS with the VM. No log messages were seen on the filer during the event, although the a log message similar to the following has been seen several times on the ESX host:

vmkernel: 43:07:27:51.725 cpu2:2185)WARNING: NFS: 4590: Can't find call with serial number -2146566055

Curiously, all crashes have happened in the evening, in the 10 o’clock hour, after nightly backups have been completed. Backups are created using a combination of VMware and NetApp snapshots via a script similar to one detailed on vmwaretips.com. No substantial load or latency has been recorded on the NetApp during the crashes, and weeks have passed between events.

Speculation

Explanations I’m leaning towards, ranked by my judgment of their likelihood:

1) Hardware issue. Assuming a random distribution of VMs – recall that DRS is in use and no virtual machine rules are in place – the odds of all five crashes happening on one host out of four are slim: 1 in 1024. Unfortunately, by all measures we’ve used, including the VI Client’s “Health Status” and Dell OMSA, there are no hardware issues with the host.

Further, the distribution of VMs is not truly random. DRS migrations are infrequent in this cluster, and the largest determinant of guest location is migration following hosts being placed into maintenance mode for patching.

If it is a hardware issue, it’s subtle, and possibly only brought to the fore by the following issues.

2) Red Hat Enterprise Linux bug – which, by extension, is typically equivalent to a CentOS bug. In fact, this issue appears to have been raised with Red Hat already in bugs 197158 and 228108 – but, according the bug reports, the issue is resolved, and the patches have since been ported downstream to CentOS. However, perhaps the issue is not truly resolved – see comment 35 in 228108.

3) vSMP Bug. The majority of our Linux VMs are uniprocessor and appear so far to be immune to this issue; it is striking that the crash has only occurred on dual processor guests. I cannot articulate a mechanism for multiple vCPUs causing this crash, however.

4) NetApp issue. This appears to be a storage issue at some level, considering the mptscsi and NFS messages noted above, so performance of the NetApp filer would be a natural place for further investigation. However, we monitor the performance of our filer relatively closely, using the ONTAP SDK and Cacti, and nothing unusual was recorded during any crash. It seems unusual that all VMs reside on the same data store, but that data store shares an aggregate with multiple other unaffected data stores, and several LUNs are served from the same aggregate to non-ESX machines without complaint.

I have not yet opened a case with VMware on this issue – or Dell, or NetApp, for that matter – but if and when I do, I’ll update here to the extent possible.

Update 11/20/2009: Prompted by a helpful comment from nate below, I looked up and verified the NFS settings across the cluster. They are the same across all hosts, and are as follows:

NFS.IndirectSend 0
NFS.DiskFileLockUpdate 10
NFS.LockUpdateTimeout 5
NFS.LockRenewMaxFailureNumber 3
NFS.LockDisable 0
NFS.HeartbeatFrequency 12
NFS.HeartbeatTimeout 5
NFS.HeartbeatDelta 5
NFS.HeartbeatMaxFailures 10
NFS.MaxVolumes 8
NFS.SendBufferSize 264
NFS.ReceiveBufferSize 128
NFS.VolumeRemountFrequency 30
NFS.UDPRetransmitDelay 700

The only values that are changed from default are HeartbeatFrequency and HeartbeatMaxFailures, to match NetApp’s recommendations in TR-3428.

> rpm -q slocate
slocate-2.7-13.el4.8.i386

or

> rpm -q mlocate
mlocate-0.15-1.el5.2.x86_64

If so, multiple RHEL VMs plus mlocate or slocate may be adding up to an array-crushing 4:02am shared storage load and latency spike for you. Before being addressed, this spike was bad enough at my place of employment (when combined with a NetApp Sunday-morning disk scrub) to cause a Windows VM to crash with I/O errors. Ouch.

Details and ideas for resolution:

By default, a line in /etc/crontab runs the scripts within /etc/cron.daily at 4:02am each morning:

02 4 * * * root run-parts /etc/cron.daily

One of those scripts – mlocate.cron or slocate.cron, depending on your OS version – launches updatedb; as the man page says, “updatedb creates or updates a database used by locate(1).” (The “locate” binary is a filesystem search tool, see “man locate” for more information.) Updatedb refreshes its database by walking the filesystem, generating a fair amount of I/O on a single system. Imagine upwards of thirty of these running in parallel through VMDKs on one shared storage system carrying out internal maintenance at the same time, and you’re pretty much picturing the problem my employer had.

I see three options for addressing this issue:

1) Uninstall mlocate or slocate. If you don’t currently use “locate” and you’re not interested in learning to use a tool that will likely make you more effective at your job (again, see “man locate”), this is probably the best option. (Yeah, I know, people that fit this bill generally don’t read blogs more technical than this one, so I could probably have skipped it here. Consider it an option for completeness, or if you really need to strip down an install.)

2) Disable the scheduled job by removing mlocate.cron or slocate.cron from /etc/cron.daily. This keeps locate available for your use, but requires that you update locate’s database ad-hoc and interactively by running the following as root:

# updatedb

This will take a few minutes to return, depending on the size of your file systems.

I don’t recommend this option either; at least it doesn’t fit the way I work. I often find myself using locate in high-pressure situations in which I need to quickly get a file location on a system. Waiting minutes for updatedb to return is extra painful when every second counts.

3) Stagger when updatedb runs by inserting a random delay into the script.. This is my preferred alternative; locate’s database is kept current automatically, and your storage doesn’t have to bear a sudden spike in load. I implemented this by adding the lines in bold (lines 2-7 if your browser doesn’t display the bold text clearly):

#!/bin/sh
# sleep up to two hours before launching job:
value=$RANDOM
while [ $value -gt 7200 ] ; do
value=$RANDOM
done
sleep $value
nodevs=$(< /proc/filesystems awk '$1 == "nodev" { print $2 }')
renice +19 -p $$ >/dev/null 2>&1
/usr/bin/updatedb -f "$nodevs"


The added code inserts a pseudo-random sleep delay of up to two hours before updatedb runs, with the key being the built-in Bash function $RANDOM. In our environment, this removed a 2000 IOPS spike at 4:02am, and eliminated a corresponding jump in filer latency. Obviously, adjust the delay period as appropriate for your environment. Additionally, be sure to add this change to your configuration management or installation management tools so that all of your RHEL and RHEL-derived VMs get the updated script.

Using $RANDOM to avoid this variant of the thundering herd problem also works nicely for a range of similar problems; I believe I first saw it at Moundalexis.com.

(This problem may apply to other Linux distributions being run as VMs, and FreeBSD does something equivalent – weekly – with /etc/periodic/weekly/310.locate. A similar solution can be applied to these environments, if necessary.)

If you suspect that your system has almost used all of its free space, or if you use thin provisioning, you should check the amount of space in use by each aggregate. If any aggregate is 97 percent full or more, do not proceed with the upgrade until you have used the Upgrade Advisor or aggrSpaceCheck tools to determine your system capacity and plan your upgrade.

Upgrade Advisor is a great tool, and I heartily recommend you use it for your upgrade. However, it doesn’t give you a lot of visibility into what’s being checked for here. Lucky for us, NetApp offers an alternative tool: aggrSpaceCheck (NOW login required).

AggrSpaceCheck, as written, relies on you having rsh turned on for access to the filer – something that you probably locked down when you were still wearing acid-washed jeans. Assuming you don’t have rsh access, you’ll see an error like this when you attempt to run aggrSpaceCheck:

> perl aggrSpaceCheck.pl -filer toaster01

aggrSpaceCheck V1.0.0 Copyright (c) 2008 NetApp 

Could not retrieve Aggregate free space. Could not get any aggregates in this filer

The fix is easy, however, if you’re using SSH: Edit the aggrSpaceCheck.pl file, replacing “rsh” with “ssh” (you only need to actually edit it in the line where “$remCmd” is defined, but changing rsh to ssh elsewhere won’t hurt). You will be prompted for root’s SSH password repeatedly – once for each command run remotely on the filer:

> perl aggrSpaceCheck.pl -filer toaster01

aggrSpaceCheck V1.0.0 Copyright (c) 2008 NetApp
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password: 

Aggregate aggr0 requires 54.18GB for volume metadata; 137.76GB is available.
Aggregate aggr0 has enough free space for you to upgrade to
Data ONTAP 7.3 or later.

There’s a solution for this too, of course: Enable SSH key pair authentication from your management host to your filer – no more password prompts:

> perl aggrSpaceCheck.pl -filer toaster01

aggrSpaceCheck V1.0.0 Copyright (c) 2008 NetApp 

Aggregate aggr0 requires 54.18GB for volume metadata; 137.76GB is available.
Aggregate aggr0 has enough free space for you to upgrade to
Data ONTAP 7.3 or later.

Beginning with Data ONTAP 7.3.1, FAS2020 systems support aggregates up to 16 TB raw capacity,
provided that the root volume is hosted in a dedicated aggregate (that is, one that contains only the root
volume and no user data).

The release notes go on to point out an alternative to the dedicated root aggregate – having two spare disks per controller.

It’s nice to see the FAS2020 finally getting a maximum aggregate size on par with the rest of NetApp’s product line. However, in an era where 2TB drives are available from Western Digital – and presumably other manufacturers before too long – ONTAP’s 16TB aggregate limit grows increasingly anachronistic.