thinking sysadmin

Update: NetApp has a KB article in NOW addressing this: Using SnapVault to Archive SnapManager for Exchange Backups Sets. Bottom line: You do not necessarily need ONTAP 7.3, Protection Manager and DataFabric Manager to send SnapManager for Exchange snapshots to a SnapVault secondary.

We recently acquired SnapManager for Exchange (SME) at my place of employment. We have an existing NetApp deployment consisting of two primary filers in a SnapVault arrangement with a third filer. The SME install is part of an upgrade from Exchange 2003 (on DAS) to 2007 (on Fibre Channel storage).

What we missed prior to purchasing SME: If you want to use SnapVault with SME, you need two additional pieces of software: Protection Manager and NetApp Management Console (part of DataFabric Manager, apparently). Here’s what p. 408 of the SnapManager® 5.0 for Microsoft® Exchange Installation and Administration Guide (NOW login required) says:

The following are the software dependencies for integrating SnapManager with
data set and SnapVault:

◆ Protection Manager 3.7 and later
◆ NetApp Management Console 3.7 and later
◆ SnapDrive for Windows 6.0 and later
◆ Data ONTAP 7.3 or later

Wish I’d known that sooner.

(This is the point where some random NetApp fanboy pops down to the comments and fires off something about how NetApp is the greatest storage company ever, and if I’d done appropriate due diligence, I wouldn’t have missed this requirement. My advice: Spare us, smart guy. I’m writing this post to make it easier for other NetApp customers to do their “due diligence”.)

There have been a few posts elsewhere discussing file-level recovery for Linux VMs on NetApp NFS datastores, but none that have dealt specifically with Linux LVM-encapsulated partitions.

Here’s our in-house procedure for recovery; note that we do not have FlexClone licensed on our filers.
Read the rest of this entry »

(William Gibson said that, I believe).

I see echoes of Toyota teaching its Toyota Production System in Google’s recent release of information about their data centers. Relatively straightforward concepts – the challenge is in adapting your existing systems to them.

(This post adds only a couple small details to work described at randys.org and cenolan.com – go there for background on this post and useful scripts for automated Duplicity backup to S3.)

First off, if you want to use Duplicity installed from FreeBSD Ports to backup to Amazon S3, be sure to also install the devel/py-boto and security/pinentry-curses ports.

If you attempt to run the backup script described at randys.org or cenolan.com from cron, you may run into an error similar to the following:
Read the rest of this entry »

I’ve written about running VMware ESX with VM swap on an NFS datastore previously – specifically whether or not it was supported/recommended:

• ESX Swap on NFS or Not?
• VMware about ESX swap on NFS: It’s okay

After writing the second post, I thought the issue was pretty much resolved: From multiple sources, the consensus seemed to be that running ESX with VM swap on NFS would be fine.  Imagine my surprise (and disappointment) at seeing the following VMware KB article 1008091, updated yesterday: An ESX virtual machine on NFS fails with swap errors. Further details are in the article itself, but VMware’s KB site is throwing intermittent errors for me at the moment, so I’ll provide the money quote:

The reliability of the virtual machine can be improved by relocating the swap file location to a non-NFS datastore. Either SAN or local storage datastores improve virtual machine stability.

In case you’re curious/adventurous/broke enough to try configuring your VMotion network on Fast Ethernet instead of Gigabit Ethernet, here’s what you can expect.

First, a warning from your VI client that you’re venturing into unsupported territory:

A friendly warning

And then, if you go ahead with the VMotion, a slight pause on the VM in question.  The following is output from running while true; do date; sleep 1; done on a Linux guest during the VMotion:

Tue Feb  3 13:23:17 PST 2009
Tue Feb  3 13:23:18 PST 2009
Tue Feb  3 13:23:19 PST 2009
Tue Feb  3 13:23:20 PST 2009
Tue Feb  3 13:23:21 PST 2009
Tue Feb  3 13:23:22 PST 2009
Tue Feb  3 13:24:12 PST 2009
Tue Feb  3 13:24:13 PST 2009
Tue Feb  3 13:24:14 PST 2009
Tue Feb  3 13:24:15 PST 2009
Tue Feb  3 13:24:16 PST 2009

Note the fifty second pause between 13:23:22 and 13:24:12? Ouch…

As a sysadmin that supports multiple R users, a post late last year on InsideHPC drew my attention – Parallel framework for statistical analysis package “R”.  The creators of the Simple Parallel R INTerface have “designed and built a prototype framework that allows the addition of parallelised functions to R to enable the easy exploitation of HPC systems.” (paper, source code)  In other words, a system that lets R users run on a cluster without learning parallel programming.
Read the rest of this entry »

Apparently, disk arrays are sensitive sorts that respond poorly when yelled at:

Makes me wonder how much engineering that I never thought about goes into designing disk shelves to keep drives insulated from vibrations. The Fishworks analytics interface is dazzling – wish I had that yesterday when I was looking at a possible Exchange I/O performance issue with perfmon…

I was checking out VMware’s new online search-able HCL and I noticed that the new Sun Unified Storage Systems were on the HCL. That was fast – and now I’m really curious as to how the systems with flash drives perform as storage for ESX.

Paul Manning, from VMware, in response to a question I asked in the VI:OPS forums:

The current best practice for NFS is to not seperate the VM swap space from the VMhome directory on a NFS datastore. The reason for the originial recommendation was just good old fashioned conservitiveness.

More at the forum post, including more on the reasoning for the old recommendation of separating swap when using NFS – thanks, Paul, you made my day.