netapp01> lun show /vol/nexenta01/lun01/lun
        /vol/nexenta01/lun01/lun      10g (10737418240)   (r/w, online, mapped)

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no

root@nexenta01:/volumes# stmsboot -L
stmsboot: MPXIO disabled
root@nexenta01:/volumes# stmsboot -e -D fp
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y)
updating //platform/i86pc/boot_archive
updating //platform/i86pc/amd64/boot_archive
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y)

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes

Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

root@nexenta01:/volumes# stmsboot -L
stmsboot: No STMS devices have been found

netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
netapp01> igroup set nexenta01 alua yes
netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee(logged in on: 0b, 0d, vtic)
        ALUA: Yes

nmc@nexenta01:/$ lunsync -r
Cleanup obsolete (dangling) device links?  Yes
Re-scanning HBAs... done.
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c4t6*469d0  sd9       disk         10GB                  no      mpxio  60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

root@nexenta01:/volumes# stmsboot -L
non-STMS device name                    STMS device name
------------------------------------------------------------------
/dev/rdsk/c3t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0

Hat Tip to @complex on Twitter.

Reference: Is it possible to use I/O multipathing? How?

Crash Symptoms

A CentOS 4.x or 5.x guest will crash with a message similar to the following on its console:

CentOS 4.x:

[<f883b299>] .text.lock.scsi_error+0x19/0x34 [scsi_mod]
[<f88c19ce>] mptscsih_io_done+0x5ee/0x608 [mptscsi] (…)
[<c02de564>] common_interrupt+0x18/0x20
[<c02ddb54>] system_call+0x0/0x30

CentOS 5.x:

RIP  [<ffffffff8014c562>] list_del+0x48/0x71 RSP <ffffffff80425d00> <0>Kernel Panic - not syncing: Fatal exception

A hard reset (i.e. pressing the reset button on the VM’s console) is required to reboot the guest.

Further Details

Five different VMs have encountered this issue, running at a mix of close-to-current CentOS 4.x and 5.x patch levels. Guest kernel versions when the crash occurred were 2.6.18-128.7.1.el5 and 2.6.18-128.1.10.el5 (5.x) and 2.6.9-89.0.9.ELsmp (4.x). Memory allocations on affected guests range from 512MB to 3072MB. Notably, all affected VMs are using SMP – each has 2 vCPUs – having been created before our in-house practices followed VMware guidelines and discouraged use of SMP on ESX guests when unnecessary. One VM was created via P2V; the rest were created de novo on virtual hardware.

All crashes have happened on a single node in an ESX 3.5 HA cluster composed of four Dell PowerEdge 1950s. ESX hosts have tracked the latest VMware patches closely. COS memory on the ESX host in question was increased from the default to 800MB prior to the three most recent crashes; in other words, the COS memory increase appears to have had no effect on the crashes. DRS is in use, set to “fully automated” and “apply recommendations with three or more stars” and no virtual machine rules have been created to control DRS host placement.

All guests are on the same NFS data store, served from a NetApp filer running ONTAP 7.2.x. One guest had its vmswap placed separately on an iSCSI data store; the rest have their swap stored on NFS with the VM. No log messages were seen on the filer during the event, although the a log message similar to the following has been seen several times on the ESX host:

vmkernel: 43:07:27:51.725 cpu2:2185)WARNING: NFS: 4590: Can't find call with serial number -2146566055

Curiously, all crashes have happened in the evening, in the 10 o’clock hour, after nightly backups have been completed. Backups are created using a combination of VMware and NetApp snapshots via a script similar to one detailed on vmwaretips.com. No substantial load or latency has been recorded on the NetApp during the crashes, and weeks have passed between events.

Speculation

Explanations I’m leaning towards, ranked by my judgment of their likelihood:

1) Hardware issue. Assuming a random distribution of VMs – recall that DRS is in use and no virtual machine rules are in place – the odds of all five crashes happening on one host out of four are slim: 1 in 1024. Unfortunately, by all measures we’ve used, including the VI Client’s “Health Status” and Dell OMSA, there are no hardware issues with the host.

Further, the distribution of VMs is not truly random. DRS migrations are infrequent in this cluster, and the largest determinant of guest location is migration following hosts being placed into maintenance mode for patching.

If it is a hardware issue, it’s subtle, and possibly only brought to the fore by the following issues.

2) Red Hat Enterprise Linux bug – which, by extension, is typically equivalent to a CentOS bug. In fact, this issue appears to have been raised with Red Hat already in bugs 197158 and 228108 – but, according the bug reports, the issue is resolved, and the patches have since been ported downstream to CentOS. However, perhaps the issue is not truly resolved – see comment 35 in 228108.

3) vSMP Bug. The majority of our Linux VMs are uniprocessor and appear so far to be immune to this issue; it is striking that the crash has only occurred on dual processor guests. I cannot articulate a mechanism for multiple vCPUs causing this crash, however.

4) NetApp issue. This appears to be a storage issue at some level, considering the mptscsi and NFS messages noted above, so performance of the NetApp filer would be a natural place for further investigation. However, we monitor the performance of our filer relatively closely, using the ONTAP SDK and Cacti, and nothing unusual was recorded during any crash. It seems unusual that all VMs reside on the same data store, but that data store shares an aggregate with multiple other unaffected data stores, and several LUNs are served from the same aggregate to non-ESX machines without complaint.

I have not yet opened a case with VMware on this issue – or Dell, or NetApp, for that matter – but if and when I do, I’ll update here to the extent possible.

Update 11/20/2009: Prompted by a helpful comment from nate below, I looked up and verified the NFS settings across the cluster. They are the same across all hosts, and are as follows:

NFS.IndirectSend 0
NFS.DiskFileLockUpdate 10
NFS.LockUpdateTimeout 5
NFS.LockRenewMaxFailureNumber 3
NFS.LockDisable 0
NFS.HeartbeatFrequency 12
NFS.HeartbeatTimeout 5
NFS.HeartbeatDelta 5
NFS.HeartbeatMaxFailures 10
NFS.MaxVolumes 8
NFS.SendBufferSize 264
NFS.ReceiveBufferSize 128
NFS.VolumeRemountFrequency 30
NFS.UDPRetransmitDelay 700

The only values that are changed from default are HeartbeatFrequency and HeartbeatMaxFailures, to match NetApp’s recommendations in TR-3428.

If you suspect that your system has almost used all of its free space, or if you use thin provisioning, you should check the amount of space in use by each aggregate. If any aggregate is 97 percent full or more, do not proceed with the upgrade until you have used the Upgrade Advisor or aggrSpaceCheck tools to determine your system capacity and plan your upgrade.

Upgrade Advisor is a great tool, and I heartily recommend you use it for your upgrade. However, it doesn’t give you a lot of visibility into what’s being checked for here. Lucky for us, NetApp offers an alternative tool: aggrSpaceCheck (NOW login required).

AggrSpaceCheck, as written, relies on you having rsh turned on for access to the filer – something that you probably locked down when you were still wearing acid-washed jeans. Assuming you don’t have rsh access, you’ll see an error like this when you attempt to run aggrSpaceCheck:

> perl aggrSpaceCheck.pl -filer toaster01

aggrSpaceCheck V1.0.0 Copyright (c) 2008 NetApp 

Could not retrieve Aggregate free space. Could not get any aggregates in this filer

The fix is easy, however, if you’re using SSH: Edit the aggrSpaceCheck.pl file, replacing “rsh” with “ssh” (you only need to actually edit it in the line where “$remCmd” is defined, but changing rsh to ssh elsewhere won’t hurt). You will be prompted for root’s SSH password repeatedly – once for each command run remotely on the filer:

> perl aggrSpaceCheck.pl -filer toaster01

aggrSpaceCheck V1.0.0 Copyright (c) 2008 NetApp
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password: 

Aggregate aggr0 requires 54.18GB for volume metadata; 137.76GB is available.
Aggregate aggr0 has enough free space for you to upgrade to
Data ONTAP 7.3 or later.

There’s a solution for this too, of course: Enable SSH key pair authentication from your management host to your filer – no more password prompts:

> perl aggrSpaceCheck.pl -filer toaster01

aggrSpaceCheck V1.0.0 Copyright (c) 2008 NetApp 

Aggregate aggr0 requires 54.18GB for volume metadata; 137.76GB is available.
Aggregate aggr0 has enough free space for you to upgrade to
Data ONTAP 7.3 or later.

Beginning with Data ONTAP 7.3.1, FAS2020 systems support aggregates up to 16 TB raw capacity,
provided that the root volume is hosted in a dedicated aggregate (that is, one that contains only the root
volume and no user data).

The release notes go on to point out an alternative to the dedicated root aggregate – having two spare disks per controller.

It’s nice to see the FAS2020 finally getting a maximum aggregate size on par with the rest of NetApp’s product line. However, in an era where 2TB drives are available from Western Digital – and presumably other manufacturers before too long – ONTAP’s 16TB aggregate limit grows increasingly anachronistic.

We recently acquired SnapManager for Exchange (SME) at my place of employment. We have an existing NetApp deployment consisting of two primary filers in a SnapVault arrangement with a third filer. The SME install is part of an upgrade from Exchange 2003 (on DAS) to 2007 (on Fibre Channel storage).

What we missed prior to purchasing SME: If you want to use SnapVault with SME, you need two additional pieces of software: Protection Manager and NetApp Management Console (part of DataFabric Manager, apparently). Here’s what p. 408 of the SnapManager® 5.0 for Microsoft® Exchange Installation and Administration Guide (NOW login required) says:

The following are the software dependencies for integrating SnapManager with
data set and SnapVault:

◆ Protection Manager 3.7 and later
◆ NetApp Management Console 3.7 and later
◆ SnapDrive for Windows 6.0 and later
◆ Data ONTAP 7.3 or later

Wish I’d known that sooner.

(This is the point where some random NetApp fanboy pops down to the comments and fires off something about how NetApp is the greatest storage company ever, and if I’d done appropriate due diligence, I wouldn’t have missed this requirement. My advice: Spare us, smart guy. I’m writing this post to make it easier for other NetApp customers to do their “due diligence”.)

Here’s our in-house procedure for recovery; note that we do not have FlexClone licensed on our filers.

Prerequisites

• An existing VMware ESX infrastructure, connected to a NetApp filer NFS datastore; SnapRestore speeds the recovery process but is not mandatory – see discussion below.
• A backup script or system which coordinates VMware snapshots with NetApp snapshots – perhaps something along the lines of Rick Scherer’s script.
• A dedicated Linux restore VM, at a similar version level to the rest of your Linux VM infrastructure. This VM should have LVM support, but should not have any volume groups (VGs) or logical volumes (LVs) configured – volume group and logical volume names on the VMDK you are restoring from must not conflict with VGs and LVs already in use on the restore system; the simplest way to guarantee this is to simply not have any VGs or LVs.

Restore Procedure

• Restore the VMDK file from the appropriate snapshot to a new location in the datastore. With SnapRestore, this can be done as follows (one line in the filer CLI, restoring from snapshot sv_daily.0 to a new file – again, be extremely careful not to overwrite the current version of the VMDK in your datastore, consider restoring to an entirely different directory in the FlexVol):

  snap restore -t file -s sv_daily.0
-r /vol/vmware04_sis/system.example.com/system.example.com-restore.vmdk
/vol/vmware04_sis/system.example.com/system.example.com.vmdk

  Follow the prompts, verifying the restore path is correct and is not the path to your existing VMDK. Do the same for the flat VMDK file (again, one line, and, as before, use caution to make sure you do not clobber an existing file):

  snap restore -t file -s sv_daily.0
-r /vol/vmware04_sis/system.example.com/system.example.com-restore-flat.vmdk
/vol/vmware04_sis/system.example.com/system.example.com-flat.vmdk

  Without SnapRestore, you can simply mount the NFS export of the datastore on a Linux machine and use “cp” to copy the files out of the snapshot. For flat VMDK files, expect this copy to run for a substantial amount of time compared the nearly-instant recovery SnapRestore offers.

• Manually edit the line below “# Extent description” in the recovered .vmdk file to match the path to the recovered flat VMDK. In this case, it would look something like this:
  # Extent description
RW 20971520 VMFS "system.example.com-restore-flat.vmdk"
• Attach the recovered VMDK to your powered-off restore host. Boot the restore host.
• Once your restore host is up, use “pvscan”, “vgscan” and “lvscan” (each without arguments) as root to examine available LVM components. Then, use the “lvchange” command to activate the necessary volume group (in this case, “VolGroup00″):
  # lvchange -ay VolGroup00
• Mount the appropriate logical volume – for example, LogVol00 in VolGroup00:
  mount -o ro /dev/VolGroup00/LogVol00 /mnt
  Restore files by copying them out of /mnt.

Cleanup

• Shut down the Linux restore host.
• Remove the recovery VMDK – the files restored with SnapRestore or by “cp” above – from the restore host in the VMware Infrastructure Client.
• Delete the recovery .vmdk and -flat.vmdk files in the NFS datastore. Don’t screw up here: Be sure to delete the recovery files only, not the working VMDK.

This articles describes changing the default location for your virtual machines [sic] swap files. You may need to use this if you are running virtual machines from NFS storage.
Note: VMware recommends storing your swap on a VMFS3 volume, when running virtual machines on NFS storage.

(Emphasis mine.) The article provides none of the reasoning behind this recommendation, and a search of the VMware Knowledge Base for “nfs swap” only returns this article. There are quite a few mentions of not putting swap on NFS in the VMware Communities forums, citing performance reasons, but, as far as I could find, no support for the performance issue assertion.

Personally, I’m not buying the performance claims without substantiation – NFS and iSCSI to the same filer should be nearly equally performant. I could imagine a difference in locking behavior, but I’m having trouble imagining a scenario where lock contention with a VM swap file would become an issue.

On the other side of the issue, I’ve been hearing from a few folks – mostly NetApp employees or users – that placing swap on NFS is okay. On the viops site, Steve Chambers writes in a preview of VMworld TA 2784 (which I did not attend) that “there is no need to place swap space on non IP storage.” IP storage could mean iSCSI here, of course, but in the VM /ETC write up of TA 2784, Rich Brambley’s notes state: “Do not place [swap] space on other storage – do not place the vswap on VMFS. Follow best practices and keep the swap on the NFS storage.”

So, who’s right here? VMware or NetApp? I suppose the conservative thing is to keep the swap on VMFS until VMware tells us otherwise, with the only negatives being the increased operational complexity and reduced storage utilization – that’s what I’m doing. I can’t help but wonder if this is like VMware’s Linux timekeeping recommendations, though – conflicting information subject to change – and hopefully clarification – at a later date.

Update, 11/24/2008: Keeping ESX swap on NFS with the VMhome directory is considered the current best practice.

Suppose, for example, that you have a FAS2050; the maximum size FlexVol that you can dedupe is 1 TB. If the volume has ever been larger than 1 TB and then shrunk below that limit, it can’t be deduped, and, of course, you can’t grow a volume with A-SIS enabled beyond 1 TB. Fair enough, you say – but consider those limitations in the case of a volume where you aren’t sure how large it will eventually grow.

If you think your volume could eventually grow beyond 1 TB (deduped), and you’re getting a healthy 50% savings from dedupe you’ll actually need to undo A-SIS at 500GB. If you let your deduped data approach filling a 1TB volume, you will not be able to run “sis undo” – you’ll run out of space. TR-3505 has this to say about it:

Note that if sis undo starts processing and then there is not enough space to undeduplicate, it will stop, complain with a message about insufficient space, and leave the flexible volume dense. All data is still accessible, but some block sharing is still occurring. Use “df –s” to understand how much free space you really have and then either grow the flexible volume or delete data or Snapshot copies to provide the needed free space.

Bottom line: Either be absolutely sure you won’t ever need to grow your volume beyond the A-SIS limitations of your hardware platform, or run “sis undo” before the sum of the “used” and “saved” columns of “df -s” reaches the volume limit.

Postscript: If you were thinking – like I was – that ONTAP 7.3 would up the A-SIS limitations, apparently you need to think again.

Postscript 2: See also NOW KB35784, as pointed out by Dan C on Scott Lowe’s blog.