thinking sysadmin » iscsi

Interesting Linux VM Crash Pattern

Andy — Fri, 20 Nov 2009 19:09:16 +0000

I’ve just begun to pull together some interesting data on a series of Linux VM crashes I’ve seen. I don’t have a resolution yet, but some interesting patterns have emerged.

Crash Symptoms

A CentOS 4.x or 5.x guest will crash with a message similar to the following on its console:

CentOS 4.x:

[] .text.lock.scsi_error+0x19/0x34 [scsi_mod] [] mptscsih_io_done+0x5ee/0x608 [mptscsi] (…) [] common_interrupt+0x18/0x20 [] system_call+0x0/0x30

CentOS 5.x:

RIP [] list_del+0x48/0x71 RSP <0>Kernel Panic - not syncing: Fatal exception

A hard reset (i.e. pressing the reset button on the VM’s console) is required to reboot the guest.

Further Details

Five different VMs have encountered this issue, running at a mix of close-to-current CentOS 4.x and 5.x patch levels. Guest kernel versions when the crash occurred were 2.6.18-128.7.1.el5 and 2.6.18-128.1.10.el5 (5.x) and 2.6.9-89.0.9.ELsmp (4.x). Memory allocations on affected guests range from 512MB to 3072MB. Notably, all affected VMs are using SMP – each has 2 vCPUs – having been created before our in-house practices followed VMware guidelines and discouraged use of SMP on ESX guests when unnecessary. One VM was created via P2V; the rest were created de novo on virtual hardware.

All crashes have happened on a single node in an ESX 3.5 HA cluster composed of four Dell PowerEdge 1950s. ESX hosts have tracked the latest VMware patches closely. COS memory on the ESX host in question was increased from the default to 800MB prior to the three most recent crashes; in other words, the COS memory increase appears to have had no effect on the crashes. DRS is in use, set to “fully automated” and “apply recommendations with three or more stars” and no virtual machine rules have been created to control DRS host placement.

All guests are on the same NFS data store, served from a NetApp filer running ONTAP 7.2.x. One guest had its vmswap placed separately on an iSCSI data store; the rest have their swap stored on NFS with the VM. No log messages were seen on the filer during the event, although the a log message similar to the following has been seen several times on the ESX host:

vmkernel: 43:07:27:51.725 cpu2:2185)WARNING: NFS: 4590: Can't find call with serial number -2146566055

Curiously, all crashes have happened in the evening, in the 10 o’clock hour, after nightly backups have been completed. Backups are created using a combination of VMware and NetApp snapshots via a script similar to one detailed on vmwaretips.com. No substantial load or latency has been recorded on the NetApp during the crashes, and weeks have passed between events.

Speculation

Explanations I’m leaning towards, ranked by my judgment of their likelihood:

1) Hardware issue. Assuming a random distribution of VMs – recall that DRS is in use and no virtual machine rules are in place – the odds of all five crashes happening on one host out of four are slim: 1 in 1024. Unfortunately, by all measures we’ve used, including the VI Client’s “Health Status” and Dell OMSA, there are no hardware issues with the host.

Further, the distribution of VMs is not truly random. DRS migrations are infrequent in this cluster, and the largest determinant of guest location is migration following hosts being placed into maintenance mode for patching.

If it is a hardware issue, it’s subtle, and possibly only brought to the fore by the following issues.

2) Red Hat Enterprise Linux bug – which, by extension, is typically equivalent to a CentOS bug. In fact, this issue appears to have been raised with Red Hat already in bugs 197158 and 228108 – but, according the bug reports, the issue is resolved, and the patches have since been ported downstream to CentOS. However, perhaps the issue is not truly resolved – see comment 35 in 228108.

3) vSMP Bug. The majority of our Linux VMs are uniprocessor and appear so far to be immune to this issue; it is striking that the crash has only occurred on dual processor guests. I cannot articulate a mechanism for multiple vCPUs causing this crash, however.

4) NetApp issue. This appears to be a storage issue at some level, considering the mptscsi and NFS messages noted above, so performance of the NetApp filer would be a natural place for further investigation. However, we monitor the performance of our filer relatively closely, using the ONTAP SDK and Cacti, and nothing unusual was recorded during any crash. It seems unusual that all VMs reside on the same data store, but that data store shares an aggregate with multiple other unaffected data stores, and several LUNs are served from the same aggregate to non-ESX machines without complaint.

I have not yet opened a case with VMware on this issue – or Dell, or NetApp, for that matter – but if and when I do, I’ll update here to the extent possible.

Update 11/20/2009: Prompted by a helpful comment from nate below, I looked up and verified the NFS settings across the cluster. They are the same across all hosts, and are as follows:

NFS.IndirectSend 0 NFS.DiskFileLockUpdate 10 NFS.LockUpdateTimeout 5 NFS.LockRenewMaxFailureNumber 3 NFS.LockDisable 0 NFS.HeartbeatFrequency 12 NFS.HeartbeatTimeout 5 NFS.HeartbeatDelta 5 NFS.HeartbeatMaxFailures 10 NFS.MaxVolumes 8 NFS.SendBufferSize 264 NFS.ReceiveBufferSize 128 NFS.VolumeRemountFrequency 30 NFS.UDPRetransmitDelay 700

The only values that are changed from default are HeartbeatFrequency and HeartbeatMaxFailures, to match NetApp’s recommendations in TR-3428.

First Thoughts about Fishworks

Andy — Wed, 12 Nov 2008 00:20:37 +0000

With surprisingly little buzz (outside of sun.com) – must be that darned economy – Sun launched its new Fishworks product line yesterday: Three hardware products, several of them with flash drives, and an impressive looking user interface, which appears at first glace to surpass anything NetApp offers. Here’s a quick rundown of features from Mike Shapiro on blogs.sun.com:

NFS v3 and v4
CIFS
iSCSI
HTTP
WebDAV
FTP
RAID-Z (RAID-5 and RAID-6), Mirrored, and Striped disk configurations
Unlimited Read-only and Read-write Snapshots, with Snapshot Schedules
Built-in Data Compression
Remote Replication of data for Disaster Recovery
Active-Active Clustering (in the Sun Storage 7410) for High Availability
Thin Provisioning of iSCSI LUNs
Virus Scanning and Quarantine
NDMP Backup and Restore

A few comments: Looks like all of the usual ZFS features are there, with a few additions – in particular, I wasn’t aware that the virus scanning project existed, and I didn’t know that NDMP was far enough along to be included in a production release. Additionally, from looking at various Sun blogs, I believe that the remote replication feature is zfs send/recv, not AVS. Finally, from the nomenclature (“2008.11″), I’d guess that the software is based on the forthcoming release of OpenSolaris, not the recently released update to Solaris 10.

What’s missing? Off the top of my head:

Fibre Channel – COMSTAR is coming, presumably.
HSM – ADM is also presumably on its way in a future release.
HCL entries for various products like VMware, but again, I have to believe that Sun is working hard on this as well.

My first impression from the launch materials: Neat, but the price seems high. Looking at list prices for the models, and doing some quick calculations for RAID-Z2 configurations with at least one hot spare, the price per usable TB ranges from $3999 and $3933 for a 7210 with 250GB and 1TB drives, respectively, to $11,209 for a single head 7410. Compare this to the hardware that the 7210/250GB is based on, the X4540, where you pay $2513.71 per usable TB. Now, as far as I know, Sun isn’t offering flash drives with their non-Fishworks hardware, so it makes direct comparisons of the price of the Fishworks Special Sauce impossible for most of the rest of the line, but that may change later this fall.

Other thoughts: Why not use the UltraSPARC T2 instead of Opterons? I’d expect better performance from the UltraSPARCs, especially when using 10GbE. Is it a cost issue?

One final note: Sun is making their simulator (a VMware image) available for download – nice touch.

8/14/2008 Link Dump

Andy — Thu, 14 Aug 2008 21:44:23 +0000

Performance Report: Multiprotocol Performance Test of VMware® ESX 3.5 on NetApp Storage Systems: A complementary whitepaper to VMware’s own work comparing Fibre Channel, iSCSI and NFS as storage protocols for VMware ESX. (Seen at blog.scottlowe.org.)

Link Dump, 7/17/2008

Andy — Thu, 17 Jul 2008 19:50:44 +0000

Elektronkind: OpenSolaris 2008.11 – A Preview For The Storage Admin – A look at upcoming storage technologies in OpenSolaris 2008.11, including ZFS, iSCSI, NDMP, COMSTAR, AVS and SAM-QFS. These products really set OpenSolaris apart from Linux distributions, although I wonder how official this list is, and have some doubts about the status of some of the projects. For example, there doesn’t appear to be much activity on the SAM-QFS OpenSolaris project, although maybe I’m just looking in the wrong place. (Seen at c0t0d0s0.org.)
Ruling: SCO owes Novell $2.54 million from SCO-Sun SVRX deal – Interesting excerpt: “Judge Kimball also reviewed SCO’s agreement with Sun and found that some of the terms exceeded SCO’s licensing authority. Through the agreement, SCO lifted the confidentiality provisions of Sun’s 1994 SVRX deal with Novell even though SCO was not permitted to do so without Novell’s explicit consent. The judge concluded that lifting of the SVRX confidentiality provisions was not incidental to a UnixWare license and was consequently not permissible. This raises some intriguing legal questions about OpenSolaris, which includes SVRX code that we now know SCO clearly had no right to let Sun open.” I wonder if we’ll be hearing more about this in the coming months.
Interview: IT consumerization and the future of higher ed – Another interesting piece on Ars Technica from today, an interview with Oren Sreebny of the University of Washington, whose best bits obliquely refer to the challenges of miasma computing and information security. Quotes: “Lately we’ve been looking at Google and Microsoft offerings for commodity stuff, and one of the things we deal with in some of our research [departments] is government regulations about ‘exporting munitions.’ So one of the manifestations of those government regulations is that you cannot store your data outside the US if you’re working on some types of government-funded projects. Google has said, ‘We can’t guarantee that anybody’s stuff in particular won’t be in a datacenter that’s located outside the US, so don’t bring that stuff to us,’ which is exactly what I’d be saying if I was them. So we have to figure out, as we start to move in those directions, what we do about that.” Also: “[Separate identity principals for people who are working on sensitive data] is an interesting conversation because, in many ways we’ve spent the last decade trying to integrate people’s identity, and do single-sign-on, and not make them have lots of separate accounts in separate places. And in many ways it really goes against the grain to step back from that, but maybe it’s time to do that.”

VMware’s Comparison of Storage Protocol Performance

Andy — Sat, 09 Feb 2008 04:18:49 +0000

VMware has just released a paper entitled Comparison of Storage Protocol Performance (seen at Scale the Mind and blog.scottlowe.org); maybe this will help deflate some of the too-often repeated speculation that NFS is too slow for VMware ESX.

VMware’s findings match well with what I’ve seen. On some in-house application-specific benchmarking that I’ve done, I actually saw overall better performance with an NFS datastore than with a software iSCSI datastore on the same filer. I won’t get into details, because the benchmark was specific to us (and I’d probably need a lawyer to review the EULAs before “publishing” anything…), but NFS was equal to or slightly faster than iSCSI across the board on this specific set of tests in our specific environment. Given the management and deployment advantages of NFS, that’s huge.

Of course, I wouldn’t recommend basing your whole ESX/NetApp deployment strategy off of one unsubstantiated benchmark-related post on my blog; if you’re using ESX with NetApp storage, I would strongly recommend testing NFS datastores if you haven’t already, though.