thinking sysadmin

(In the spirit of Joerg Moellenkamp’s thought experiments:)

That virtualized Fishworks appliance got me thinking: What if you combined this with this? Yeah, managing Elastic Block Store devices would require some changes, but, if you needed a NAS for your EC2 instances…

Amazon’s much-awaited Elastic Block Store for EC2 is out this morning; I’m excited to give this a try. A couple downers from the announcement: The pricing is somewhat high – $0.10 per allocated GB per month plus $0.10 per 1 million I/O requests – and the reliability isn’t where I’d like it to be. Specifically, Amazon notes:

Volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives.

Because Amazon EBS servers are replicated within a single Availability Zone, mirroring data across multiple Amazon EBS volumes in the same Availability Zone will not significantly improve volume durability.

That last sentence makes it sound like there is a 0.1% – 0.5% chance of catastrophic data loss of many distinct EBS volumes in an availability zone. If that’s the case, that’s scary – off the top of my head, I’d say your run-of-the mill “Enterprise” SAN doesn’t have a one-in-two hundred risk of catastrophic failure per year.

More links, not all of which I’ve had a chance to fully digest yet:

• Feature Guide: Elastic Block Store
• Amazon EBS (Elastic Block Store) – Bring Us Your Data
• Amazon’s Elastic Block Store explained
• Why Amazon’s Elastic Block Store Matters
• Amazon EBS – Tool and Library Support
• Running MySQL on Amazon EC2 with Elastic Block Store

Since Sun and Amazon removed the limit on the number of OpenSolaris 2008.05 instances able to run on EC2, I’ve been curious – and a little bothered – by the fact that the 2008.05 AMI is 32-bit only. Curious because OpenSolaris shouldn’t have any issues running on a 64-bit EC2 instance (there are other 64-bit OpenSolaris AMIs available on EC2, after all), and a little bothered because there have been long-standing reports of trouble running Solaris on 32-bit architectures, which makes me hesitant to invest much effort in a 32-bit OpenSolaris EC2 environment.

Well, perhaps a 64-bit AMI is forthcoming – I think this is still a beta program – and perhaps Sun’s just trying to save us a buck or two, since the cheapest 64-bit EC2 instance is four times as expensive per hour as the cheapest 32-bit instance.

According to a blog post on blogs.sun.com, the capacity limit for OpenSolaris 2008.05 on EC2 has been removed.

The blog entry makes it sound like you no longer need to register with Sun to use OpenSolaris on EC2, but that doesn’t appear to be the case – I only see the AMI in my private instances, and the details on the image seem to confirm this.
Read the rest of this entry »

So I returned from a little five-day weekend to sunny Lake Chelan and the Columbia River to an RSS reader bursting at the seams with new posts. By far the best post was one later deleted:

• Amazon Elastic Block Store goes live! (Yeah, that link’s dead – like I said, it was later deleted.) The RightScale folks appear to have inadvertently published a draft (on 8/8, the day after I left town) of their blog post designed to coincide with the release of Amazon’s Elastic Block Store for EC2. They later deleted it, but Google Reader kindly cached the post for me. I won’t repeat anything in the post, nor would I bank on anything written there – would you gamble anything important on a retracted post about a not-yet-released product? I will add one comment: Will EBS attract attention of the lawsuit kind from NetApp? (I mean that comment only partially in jest – and you’d probably have to have seen the original post to know what I’m talking about.)

• The Hitz report – Robin Harris at StorageMojo on the Sun-NetApp lawsuit:
  

  NetApp’s biggest misperception is that WAFL is somehow central to the success they are enjoying today. That was true about 10 years ago. Guys, your average F500 CIO today could care less about WAFL.

  NetApp is growing because they offer a compelling value proposition of quality products, relevant services and worldwide support. WAFL certainly supports that, but as NetApp execs note much of their recent success is due to the integration software that NetApp now offers.

  WAFL is a small piece of the picture. Sun could copy it line for line and still not have a quarter of what NetApp offers.

  NetApp faces challenges. Storage commoditization threatens all vendors traditional 60% gross margins. The GX integration is problematic and the bottom line benefit uncertain. EMC’s move into cloud file services is a clever flanking strategy.

  An interesting opinion summed up nicely, I think.

• Saving and Restoring ZFS Snapshots to and from Amazon S3 – A ZFS to S3 workaround for the lack of persistent storage on EC2.

For grins, I tried a quick-and-dirty BFU of a SXCE 79 instance running on EC2 to the latest nightly build this morning. I roughly followed Ben Rockwood’s BFU instructions and didn’t do anything to resolve conflicts beyond running acr. On reboot, it looks like the system panicked – I presume the reason is probably somewhere in here. Console dump after the jump for the curious.
Read the rest of this entry »

• Eucalyptus – “EUCALYPTUS – Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems – is an open-source software infrastructure for implementing “cloud computing” on clusters. The current interface to EUCALYPTUS is compatible with Amazon’s EC2 interface, but the infrastructure is designed to support multiple client-side interfaces.” If VMware is perhaps working on responding to EC2, this could be thought of as EC2’s response to VMware ESX, in a way – although it’s coming from UCSB, not Amazon. Notable: This is a product that layers on top of a Rocks cluster installation. (Seen at High Scalability.)

I missed this: Apparently the OpenSolaris 2008.05 AMI on EC2 has reached a capacity limit. So, while I got Sun’s approval this morning for their OpenSolaris/SXCE.79 on EC2 beta, I don’t get to use OpenSolaris until they add more capacity; SXCE.79 is a nice consolation prize, though.

• SkyNet Lives! (aka EC2 @ SmugMug) – Blog post about how SmugMug uses (and doesn’t use) Amazon Web Services; I found the comment that EC2 Persistent Storage “isn’t performant enough” intriguing – I’ll be interested to see what its performance characteristics are once it’s available to the public. Excerpt from the post: “Let me be very clear here: I really don’t want to operate datacenters anymore despite the fact that we’re pretty good at it. It’s a necessary evil because we’re an Internet company, but our mission is to be the best photo sharing site. We’d rather spend our time giving our customers great service and writing great software rather than managing physical hardware. I’d rather have my awesome Ops team interacting with software remotely for 100% of their duties (and mostly just watching software like SkyNet do its thing). We’ll get there – I’m confident of that – we’re just not there yet.” (Seen at the Amazon Web Services Blog.)
• Rough Type: Microsoft to put “many millions” of servers in cloud
  – Nicholas Carr reports on Microsoft’s cloud plans. My thoughts: Does anyone see a latency problem with the following – think of the speed of light in glass, or the AWS Blog post mentioned above and its comments about locating services in the same cloud , or why SmugMug still runs some of its own servers: “We’re taking everything we do at the server level, and saying that we will have a service that mirrors that exactly. The simplest one of those is to say, okay, I can run Exchange on premise, or I can connect up to it as a service. But even at the BizTalk level, we’ll have BizTalk Services. For SQL, we’ll have SQL Server Data Services, and so you can connect up, build the database. It will be hosted in our cloud with the big, big data center, and geo-distributed automatically. This is kind of fascinating because it’s getting us to think about data centers at a scale that never existed before. Literally today we have, in our data center, many hundreds of thousands of servers, and in the future we’ll have many millions of those servers.”