thinking sysadmin

• We’re Never Content – Amazon announces a forthcoming CDN layered on top of S3 with “edge locations on three continents” – presumably North America, Europe and Asia – “in order to deliver your content from the most appropriate location.” Presumably Amazon is planning to use this in-house for their digital media sales, or possibly for static content on their website.
• Tape, Roman Chariots and Data Management – “But here’s where it gets insidious, we know look at the mess that tape has created, and instead of asking the question: ‘Is a data protection infrastructure predicated on creating whole copies on a regular basis flawed?’ We ask the question: ‘How can I make creating and storing full copies more efficient?’” An interesting read – nothing new – but somehow I don’t think that the solution the author would propose involves tape in an HSM scenario. Which is too bad, because an HSM environment using tape really can address the problems mentioned in the article, as well as other issues such as capacity and power.

• Timekeeping best practices for Linux – “This article presents best practices for Linux timekeeping. These recommendations include specifics on the particular kernel command line options to use for the Linux operating system of interest. There is also a description of the recommended settings and usage for NTP time sync, configuration of VMware Tools time synchronization, and Virtual Hardware Clock configuration, to achieve best timekeeping results.” Where has this document been since I started deploying VMware? Oh, wait, looks like it may have been written on August 19th… Still, thanks, VMware – exactly what I wanted!
• VI:OPS – A new VMware site: “We created VI:OPS to widen the discussion beyond pure, deep technical by adding five topics that VMware staff, partners and customers talk about all the time but where there is no online collaboration facility for these topics.” I found the above link through a post on this site.

• Your Usable Capacity May Vary – Chuck conducts a thought deployment comparing EMC, HP and NetApp usable space for a 120 disk Exchange deployment. And while he glosses over a couple perhaps non-minor issues (RAID-5 vs RAID-DP and whether EMC’s snapshots are adequately performant), he does hit one of NetApp’s weak spots dead on: Usable capacity, particularly on LUNs if you follow the 100% space reservation recommendation. (Being a NetApp admin these days, I can’t really comment on what he writes about HP – it’s been a long time since I’ve touched that StorageWorks stuff – and I can only repeat what I’ve heard others say about EMC.) More Chuck on this here.
• How to License Windows VMs in a Non Microsoft Virtual Environment: Why Windows Server 2008 Datacenter Edition may be the best choice. (Seen at blog.scottlowe.org.)
• Welcome – My friend, NetApp’s Vaughan Stewart: Chad Sakac highlights some flaws in NetApp’s TR-3697 (“Performance Report: Multiprotocol Performance Test of VMware® ESX 3.5 on NetApp Storage Systems”):
  

  What’s the scoop with:

  * 4K/8K IO size only
  * 2Gbps FC
  * You guys have “throughput/IOPs” shown only in relative, not in absolute.
  * 84 144GB drives with 16 VMs driving the IOMeter workloads with * 10GB of data each on them = 1.3% utilization (rounding up!).

• Less known Solaris Features: CacheFS – Joerg Moellenkamp at c0t0d0s0.org offers another installment of his excellent Less known Solaris Features series. Of note: “In the recent days there was some discussion about the declaration of the End-of-Feature status for CacheFS which will lead to the announcement of the removal of CacheFS. After a few days of discussion the ARC decided in favour of the removal.” While I’ve never personally used CacheFS – and see no use case for it on the horizon – I’m not thrilled to see it slated for removal as it does sound like it serves an important role. Perhaps ADM or SAM-QFS will become more general to support this style of HSM as well in the future.

• Part III: Since NFSv4 is Stateful It Must Be Less Robust, Right? – “This should conclude my series on this topic, but obviously it’s my blog and like any content provider with no self-respect, I am free to make as many sequels as I want in order to milk the topic for all it is worth. I am going to compare the impact on applications when the NFS client it is using recovers from an NFSv3 and NFSv4 server restart.” Good stuff.

• Performance Report: Multiprotocol Performance Test of VMware® ESX 3.5 on NetApp Storage Systems: A complementary whitepaper to VMware’s own work comparing Fibre Channel, iSCSI and NFS as storage protocols for VMware ESX. (Seen at blog.scottlowe.org.)

So I returned from a little five-day weekend to sunny Lake Chelan and the Columbia River to an RSS reader bursting at the seams with new posts. By far the best post was one later deleted:

• Amazon Elastic Block Store goes live! (Yeah, that link’s dead – like I said, it was later deleted.) The RightScale folks appear to have inadvertently published a draft (on 8/8, the day after I left town) of their blog post designed to coincide with the release of Amazon’s Elastic Block Store for EC2. They later deleted it, but Google Reader kindly cached the post for me. I won’t repeat anything in the post, nor would I bank on anything written there – would you gamble anything important on a retracted post about a not-yet-released product? I will add one comment: Will EBS attract attention of the lawsuit kind from NetApp? (I mean that comment only partially in jest – and you’d probably have to have seen the original post to know what I’m talking about.)

It’s summer, apparently everyone is on vacation, but, quietly:

• NetApp releases Data ONTAP 7.3 GA!!! Note that it’s a GA, not GD release; release notes are on NOW. (Seen at blog.scottlowe.org.)

• Amazon S3 Availability Event: July 20, 2008 – Amazon’s post-mortem on the 7/20 S3 outage. Excerpt: “We’ve now determined that message corruption was the cause of the server-to-server communication problems. More specifically, we found that there were a handful of messages on Sunday morning that had a single bit corrupted such that the message was still intelligible, but the system state information was incorrect.” (Seen first at Ars Technica.)
• VMware has released Update 2 for VMware Infrastructure 3 version 3.5 (I think that’s the Full Official Name That Only A Committee Could Love…). Scott Lowe has a good summary; release notes are here. Most notable among the updates is the ability to use VSS to quiesce Windows VMs prior to snapshotting.

• File Level Recovery from within a VMDK backup – Nick Triantos of NetApp covers file-level recovery from VMware (Windows) VMDK files again, in more depth than he has before. Nick notes that it can be a “a point-and-click process.”