thinking sysadmin

(This post adds only a couple small details to work described at randys.org and cenolan.com – go there for background on this post and useful scripts for automated Duplicity backup to S3.)

First off, if you want to use Duplicity installed from FreeBSD Ports to backup to Amazon S3, be sure to also install the devel/py-boto and security/pinentry-curses ports.

If you attempt to run the backup script described at randys.org or cenolan.com from cron, you may run into an error similar to the following:
Read the rest of this entry »

(In the spirit of Joerg Moellenkamp’s thought experiments:)

That virtualized Fishworks appliance got me thinking: What if you combined this with this? Yeah, managing Elastic Block Store devices would require some changes, but, if you needed a NAS for your EC2 instances…

• The Hitz report – Robin Harris at StorageMojo on the Sun-NetApp lawsuit:
  

  NetApp’s biggest misperception is that WAFL is somehow central to the success they are enjoying today. That was true about 10 years ago. Guys, your average F500 CIO today could care less about WAFL.

  NetApp is growing because they offer a compelling value proposition of quality products, relevant services and worldwide support. WAFL certainly supports that, but as NetApp execs note much of their recent success is due to the integration software that NetApp now offers.

  WAFL is a small piece of the picture. Sun could copy it line for line and still not have a quarter of what NetApp offers.

  NetApp faces challenges. Storage commoditization threatens all vendors traditional 60% gross margins. The GX integration is problematic and the bottom line benefit uncertain. EMC’s move into cloud file services is a clever flanking strategy.

  An interesting opinion summed up nicely, I think.

• Saving and Restoring ZFS Snapshots to and from Amazon S3 – A ZFS to S3 workaround for the lack of persistent storage on EC2.

• Daemonic Dispatches: Dissecting SimpleDB BoxUsage – Colin Percival takes a critical look at Amazon’s SimpleDB pricing scheme.

For grins, I tried a quick-and-dirty BFU of a SXCE 79 instance running on EC2 to the latest nightly build this morning. I roughly followed Ben Rockwood’s BFU instructions and didn’t do anything to resolve conflicts beyond running acr. On reboot, it looks like the system panicked – I presume the reason is probably somewhere in here. Console dump after the jump for the curious.
Read the rest of this entry »

• S3 data corruption: “We’ve isolated this issue to a single load balancer that was brought into service at 10:55pm PDT on Friday, 6/20. It was taken out of service at 11am PDT Sunday, 6/22. While it was in service it handled a small fraction of Amazon S3’s total requests in the US. Intermittently, under load, it was corrupting single bytes in the byte stream. When the requests reached Amazon S3, if the Content-MD5 header was specified, Amazon S3 returned an error indicating the object did not match the MD5 supplied. When no MD5 is specified, we are unable to determine if transmission errors occurred, and Amazon S3 must assume that the object has been correctly transmitted.” (Seen at Daemonic Dispatches.)

• The Register: Sun’s Niagara 3 will have 16-cores and 16 threads per core – “Each socket is chewing through an insane 256 threads.” I do love a good rumor.
• CloudStatus – Just-launched website for monitoring each Amazon Web Services component in more detail than Amazon gives you. Seen at O’Reilly Radar.

• Cloud Computing: Is the Cloud There Yet? – A Brief History: A mostly negative look at the prospects for cloud computing. Although the arguments it makes are fairly reasonable, the article appears to presume that cloud computing is an all-or-nothing proposal, not a tool for only some tasks. Also odd – “Many routine tasks which are not processor intensive and time critical are the most likely candidates to be migrated to cloud computing” seems wrong (think Sun’s network.com, or how SmugMug uses EC2). Further, the article seems to ignore that big pharma and financial institutions are the largest users of AWS. Still, all in all, a perspective worth considering. (Originally seen at InsideHPC.com.)
• What he said! – scalability.org: Joe Landman on “IT clusters. They are not HPC clusters by any stretch of the imagination. They don’t really work well. Some things sorta-kinda work. Lots of things don’t or cannot. You have some interesting failure modes.” I’ve seen exactly what he’s talking about from the biosciences arm of a large hardware/services company, where they sold something that was high margin, poorly configured, had a lousy interconnect, and, yes, was running RHEL. The article’s use of “IT” as a disparaging term was also interesting. (Also seen at InsideHPC.)
• EMC flashing CLARiiON? – Rumor that flash from EMC is slowly starting to march down-market.

I missed this: Apparently the OpenSolaris 2008.05 AMI on EC2 has reached a capacity limit. So, while I got Sun’s approval this morning for their OpenSolaris/SXCE.79 on EC2 beta, I don’t get to use OpenSolaris until they add more capacity; SXCE.79 is a nice consolation prize, though.

• SkyNet Lives! (aka EC2 @ SmugMug) – Blog post about how SmugMug uses (and doesn’t use) Amazon Web Services; I found the comment that EC2 Persistent Storage “isn’t performant enough” intriguing – I’ll be interested to see what its performance characteristics are once it’s available to the public. Excerpt from the post: “Let me be very clear here: I really don’t want to operate datacenters anymore despite the fact that we’re pretty good at it. It’s a necessary evil because we’re an Internet company, but our mission is to be the best photo sharing site. We’d rather spend our time giving our customers great service and writing great software rather than managing physical hardware. I’d rather have my awesome Ops team interacting with software remotely for 100% of their duties (and mostly just watching software like SkyNet do its thing). We’ll get there – I’m confident of that – we’re just not there yet.” (Seen at the Amazon Web Services Blog.)
• Rough Type: Microsoft to put “many millions” of servers in cloud
  – Nicholas Carr reports on Microsoft’s cloud plans. My thoughts: Does anyone see a latency problem with the following – think of the speed of light in glass, or the AWS Blog post mentioned above and its comments about locating services in the same cloud , or why SmugMug still runs some of its own servers: “We’re taking everything we do at the server level, and saying that we will have a service that mirrors that exactly. The simplest one of those is to say, okay, I can run Exchange on premise, or I can connect up to it as a service. But even at the BizTalk level, we’ll have BizTalk Services. For SQL, we’ll have SQL Server Data Services, and so you can connect up, build the database. It will be hosted in our cloud with the big, big data center, and geo-distributed automatically. This is kind of fascinating because it’s getting us to think about data centers at a scale that never existed before. Literally today we have, in our data center, many hundreds of thousands of servers, and in the future we’ll have many millions of those servers.”