thinking sysadmin

A picture is worth a thousand words, right?

That was easy...

Below is a quick walkthrough of my experience booting and installing the Fishworks VMware appliance; my thoughts follow.
Read the rest of this entry »

With surprisingly little buzz (outside of sun.com) – must be that darned economy – Sun launched its new Fishworks product line yesterday: Three hardware products, several of them with flash drives, and an impressive looking user interface, which appears at first glace to surpass anything NetApp offers. Here’s a quick rundown of features from Mike Shapiro on blogs.sun.com:

• NFS v3 and v4
• CIFS
• iSCSI
• HTTP
• WebDAV
• FTP
• RAID-Z (RAID-5 and RAID-6), Mirrored, and Striped disk configurations
• Unlimited Read-only and Read-write Snapshots, with Snapshot Schedules
• Built-in Data Compression
• Remote Replication of data for Disaster Recovery
• Active-Active Clustering (in the Sun Storage 7410) for High Availability
• Thin Provisioning of iSCSI LUNs
• Virus Scanning and Quarantine
• NDMP Backup and Restore

A few comments: Looks like all of the usual ZFS features are there, with a few additions – in particular, I wasn’t aware that the virus scanning project existed, and I didn’t know that NDMP was far enough along to be included in a production release. Additionally, from looking at various Sun blogs, I believe that the remote replication feature is zfs send/recv, not AVS. Finally, from the nomenclature (“2008.11″), I’d guess that the software is based on the forthcoming release of OpenSolaris, not the recently released update to Solaris 10.
Read the rest of this entry »

I’ve blogged before about the limits of NetApp’s A-SIS (Deduplication). In practical use, however, those limits can be even lower – here’s why:

Suppose, for example, that you have a FAS2050; the maximum size FlexVol that you can dedupe is 1 TB. If the volume has ever been larger than 1 TB and then shrunk below that limit, it can’t be deduped, and, of course, you can’t grow a volume with A-SIS enabled beyond 1 TB. Fair enough, you say – but consider those limitations in the case of a volume where you aren’t sure how large it will eventually grow.

If you think your volume could eventually grow beyond 1 TB (deduped), and you’re getting a healthy 50% savings from dedupe you’ll actually need to undo A-SIS at 500GB. If you let your deduped data approach filling a 1TB volume, you will not be able to run “sis undo” – you’ll run out of space. TR-3505 has this to say about it:

Note that if sis undo starts processing and then there is not enough space to undeduplicate, it will stop, complain with a message about insufficient space, and leave the flexible volume dense. All data is still accessible, but some block sharing is still occurring. Use “df –s” to understand how much free space you really have and then either grow the flexible volume or delete data or Snapshot copies to provide the needed free space.

Bottom line: Either be absolutely sure you won’t ever need to grow your volume beyond the A-SIS limitations of your hardware platform, or run “sis undo” before the sum of the “used” and “saved” columns of “df -s” reaches the volume limit.

Postscript: If you were thinking – like I was – that ONTAP 7.3 would up the A-SIS limitations, apparently you need to think again.

Postscript 2: See also NOW KB35784, as pointed out by Dan C on Scott Lowe’s blog.

Amazon’s much-awaited Elastic Block Store for EC2 is out this morning; I’m excited to give this a try. A couple downers from the announcement: The pricing is somewhat high – $0.10 per allocated GB per month plus $0.10 per 1 million I/O requests – and the reliability isn’t where I’d like it to be. Specifically, Amazon notes:

Volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives.

Because Amazon EBS servers are replicated within a single Availability Zone, mirroring data across multiple Amazon EBS volumes in the same Availability Zone will not significantly improve volume durability.

That last sentence makes it sound like there is a 0.1% – 0.5% chance of catastrophic data loss of many distinct EBS volumes in an availability zone. If that’s the case, that’s scary – off the top of my head, I’d say your run-of-the mill “Enterprise” SAN doesn’t have a one-in-two hundred risk of catastrophic failure per year.

More links, not all of which I’ve had a chance to fully digest yet:

• Feature Guide: Elastic Block Store
• Amazon EBS (Elastic Block Store) – Bring Us Your Data
• Amazon’s Elastic Block Store explained
• Why Amazon’s Elastic Block Store Matters
• Amazon EBS – Tool and Library Support
• Running MySQL on Amazon EC2 with Elastic Block Store

Update: For ONTAP 7.3.1 and later, this post is now out-of-date.

It’s been a while since I posted about my unhappiness with NetApp’s FAS2020. And while we’ve replaced our 2020 with a 2050, I still get a lot of traffic to those pages, and a few emailed questions from prospective or new 2020 owners asking about the so-called 8TB limit. Here’s a summary of what I know:
Read the rest of this entry »

Chuck’s got another, uh, thought-provoking blog post up, More Examples Of Why Server Vendors Just Don’t Get Storage, surely intended to ruffle a few feathers. And he does raise some really good points: Most server vendors need more of an SSD strategy than just making a flash drive an option (it’s how you use it, not that you have it!). And as big a fan as I am of ZFS and Sun’s storage options in general, to win in the “enterprise” (and not just, say, HPC) Sun needs to pull everything together into Solaris (from OpenSolaris) and make it less of a DIY operation.
Read the rest of this entry »

There’s been a lot of noise from the storage industry about flash recently – in particular, noise from EMC and Sun, both of whom recently announced storage products using flash, EMC in January and Sun earlier this month. Below are my thoughts on what EMC and Sun are doing, as well as what NetApp might do. Since I see a fair amount of visitors from all three companies here, if I’ve got something about your employer wrong, please correct me in the comments.
Read the rest of this entry »

• Chuck’s Blog: Vendor Differentiation Through Thought Leadership – (Yeah, that title made me gag, too.) Excerpt: “Sun came out loud and strong recently with characteristic unbridled enthusiasm. But, in classic Sun fashion, they missed the point entirely. It’s not just the flash drives (they’re just components, right?) it’s what you can do with them for customers. Seems that their target use case is ZFS running on a storage-oriented server. Not the most compelling use case, IMHO.” Maybe it’s just cynical old me, but I read that as “Sun has some pretty good technology with their server hardware and ZFS, Lustre and SAM-QFS; they might be able to put something together that could really hurt our margins, not just NetApp’s. Flash drives might help them do it. Uh-oh. Maybe we better hurry up and ship Maui.” The Sun/Intel PDF that Chuck links to is actually pretty interesting, although it would have been good to see some more numbers.

I just finished reading a paper presented at FAST ’08 from the University of Wisconsin, Madison (including the first and senior authors) and NetApp: Parity Lost and Parity Regained. The paper discusses the concept of parity pollution, where, in the words of the authors, “corrupt data in one block of a stripe spreads to other blocks through various parity calculations.”

With the NetApp employees as (middle) authors, the paper seems to have a slight orientation towards a NetApp perspective, but it does mention other filesystems, including ZFS both specifically and later by implication when discussing filesystems that use parental checksums, RAID and scrubbing to protect data. (Interestingly, the first specific mention of ZFS contains a gaffe where they refer to it using “RAID-4″ instead of RAID-Z.) The authors make an attempt to quantify the probability of loss or corruption – arriving at 0.486% probability per year for RAID-Scrub-Parent Checksum (implying ZFS) and 0.031% probability per year for RAID-Scrub-Block Checksum-Physical ID-Logical ID (WAFL) when using nearline drives in a 4 data disk, 1 parity disk RAID configuration and a read-write-scrub access pattern of 0.2-0.2-0.6.
Read the rest of this entry »

VMware has just released a paper entitled Comparison of Storage Protocol Performance (seen at Scale the Mind and blog.scottlowe.org); maybe this will help deflate some of the too-often repeated speculation that NFS is too slow for VMware ESX.
Read the rest of this entry »