thinking sysadmin

I’ve just begun to pull together some interesting data on a series of Linux VM crashes I’ve seen. I don’t have a resolution yet, but some interesting patterns have emerged.

Crash Symptoms

A CentOS 4.x or 5.x guest will crash with a message similar to the following on its console:

CentOS 4.x:

[<f883b299>] .text.lock.scsi_error+0x19/0x34 [scsi_mod]
[<f88c19ce>] mptscsih_io_done+0x5ee/0x608 [mptscsi] (…)
[<c02de564>] common_interrupt+0x18/0x20
[<c02ddb54>] system_call+0x0/0x30

CentOS 5.x:

RIP  [<ffffffff8014c562>] list_del+0x48/0x71 RSP <ffffffff80425d00> <0>Kernel Panic - not syncing: Fatal exception

A hard reset (i.e. pressing the reset button on the VM’s console) is required to reboot the guest.
Read the rest of this entry »

With surprisingly little buzz (outside of sun.com) – must be that darned economy – Sun launched its new Fishworks product line yesterday: Three hardware products, several of them with flash drives, and an impressive looking user interface, which appears at first glace to surpass anything NetApp offers. Here’s a quick rundown of features from Mike Shapiro on blogs.sun.com:

• NFS v3 and v4
• CIFS
• iSCSI
• HTTP
• WebDAV
• FTP
• RAID-Z (RAID-5 and RAID-6), Mirrored, and Striped disk configurations
• Unlimited Read-only and Read-write Snapshots, with Snapshot Schedules
• Built-in Data Compression
• Remote Replication of data for Disaster Recovery
• Active-Active Clustering (in the Sun Storage 7410) for High Availability
• Thin Provisioning of iSCSI LUNs
• Virus Scanning and Quarantine
• NDMP Backup and Restore

A few comments: Looks like all of the usual ZFS features are there, with a few additions – in particular, I wasn’t aware that the virus scanning project existed, and I didn’t know that NDMP was far enough along to be included in a production release. Additionally, from looking at various Sun blogs, I believe that the remote replication feature is zfs send/recv, not AVS. Finally, from the nomenclature (“2008.11″), I’d guess that the software is based on the forthcoming release of OpenSolaris, not the recently released update to Solaris 10.
Read the rest of this entry »

VMware has just released a paper entitled Comparison of Storage Protocol Performance (seen at Scale the Mind and blog.scottlowe.org); maybe this will help deflate some of the too-often repeated speculation that NFS is too slow for VMware ESX.
Read the rest of this entry »