thinking sysadmin

(This post adds only a couple small details to work described at randys.org and cenolan.com – go there for background on this post and useful scripts for automated Duplicity backup to S3.)

First off, if you want to use Duplicity installed from FreeBSD Ports to backup to Amazon S3, be sure to also install the devel/py-boto and security/pinentry-curses ports.

If you attempt to run the backup script described at randys.org or cenolan.com from cron, you may run into an error similar to the following:
Read the rest of this entry »

• We’re Never Content – Amazon announces a forthcoming CDN layered on top of S3 with “edge locations on three continents” – presumably North America, Europe and Asia – “in order to deliver your content from the most appropriate location.” Presumably Amazon is planning to use this in-house for their digital media sales, or possibly for static content on their website.
• Tape, Roman Chariots and Data Management – “But here’s where it gets insidious, we know look at the mess that tape has created, and instead of asking the question: ‘Is a data protection infrastructure predicated on creating whole copies on a regular basis flawed?’ We ask the question: ‘How can I make creating and storing full copies more efficient?’” An interesting read – nothing new – but somehow I don’t think that the solution the author would propose involves tape in an HSM scenario. Which is too bad, because an HSM environment using tape really can address the problems mentioned in the article, as well as other issues such as capacity and power.

• Amazon S3 Availability Event: July 20, 2008 – Amazon’s post-mortem on the 7/20 S3 outage. Excerpt: “We’ve now determined that message corruption was the cause of the server-to-server communication problems. More specifically, we found that there were a handful of messages on Sunday morning that had a single bit corrupted such that the message was still intelligible, but the system state information was incorrect.” (Seen first at Ars Technica.)
• VMware has released Update 2 for VMware Infrastructure 3 version 3.5 (I think that’s the Full Official Name That Only A Committee Could Love…). Scott Lowe has a good summary; release notes are here. Most notable among the updates is the ability to use VSS to quiesce Windows VMs prior to snapshotting.

• S3 data corruption: “We’ve isolated this issue to a single load balancer that was brought into service at 10:55pm PDT on Friday, 6/20. It was taken out of service at 11am PDT Sunday, 6/22. While it was in service it handled a small fraction of Amazon S3’s total requests in the US. Intermittently, under load, it was corrupting single bytes in the byte stream. When the requests reached Amazon S3, if the Content-MD5 header was specified, Amazon S3 returned an error indicating the object did not match the MD5 supplied. When no MD5 is specified, we are unable to determine if transmission errors occurred, and Amazon S3 must assume that the object has been correctly transmitted.” (Seen at Daemonic Dispatches.)

• SkyNet Lives! (aka EC2 @ SmugMug) – Blog post about how SmugMug uses (and doesn’t use) Amazon Web Services; I found the comment that EC2 Persistent Storage “isn’t performant enough” intriguing – I’ll be interested to see what its performance characteristics are once it’s available to the public. Excerpt from the post: “Let me be very clear here: I really don’t want to operate datacenters anymore despite the fact that we’re pretty good at it. It’s a necessary evil because we’re an Internet company, but our mission is to be the best photo sharing site. We’d rather spend our time giving our customers great service and writing great software rather than managing physical hardware. I’d rather have my awesome Ops team interacting with software remotely for 100% of their duties (and mostly just watching software like SkyNet do its thing). We’ll get there – I’m confident of that – we’re just not there yet.” (Seen at the Amazon Web Services Blog.)
• Rough Type: Microsoft to put “many millions” of servers in cloud
  – Nicholas Carr reports on Microsoft’s cloud plans. My thoughts: Does anyone see a latency problem with the following – think of the speed of light in glass, or the AWS Blog post mentioned above and its comments about locating services in the same cloud , or why SmugMug still runs some of its own servers: “We’re taking everything we do at the server level, and saying that we will have a service that mirrors that exactly. The simplest one of those is to say, okay, I can run Exchange on premise, or I can connect up to it as a service. But even at the BizTalk level, we’ll have BizTalk Services. For SQL, we’ll have SQL Server Data Services, and so you can connect up, build the database. It will be hosted in our cloud with the big, big data center, and geo-distributed automatically. This is kind of fascinating because it’s getting us to think about data centers at a scale that never existed before. Literally today we have, in our data center, many hundreds of thousands of servers, and in the future we’ll have many millions of those servers.”