thinking sysadmin » s3

Duplicity to Amazon S3 on FreeBSD: Building on the work of others

Andy — Mon, 02 Mar 2009 19:47:53 +0000

(This post adds only a couple small details to work described at randys.org and cenolan.com – go there for background on this post and useful scripts for automated Duplicity backup to S3.)

First off, if you want to use Duplicity installed from FreeBSD Ports to backup to Amazon S3, be sure to also install the devel/py-boto and security/pinentry-curses ports.

If you attempt to run the backup script described at randys.org or cenolan.com from cron, you may run into an error similar to the following:

2009-03-01_01:05:05: ... backing up filesystem
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in 
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 240,
in GPGWriteFile
bytes_to_go = data_size - get_current_size()
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 232,
in get_current_size
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory:
'/tmp/duplicity-gM4CN9-tempdir/mktemp-iZknw0-2'

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in 
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 232, in full_backup
sig_outfp = get_sig_fileobj("full-sig")
File "/usr/local/bin/duplicity", line 210, in get_sig_fileobj
fh = globals.backend.get_fileobj_write(sig_filename)
File "/usr/local/lib/python2.5/site-packages/duplicity/backend.py", line
354, in get_fileobj_write
fh = dup_temp.FileobjHooked(tdp.filtered_open("wb"))
File "/usr/local/lib/python2.5/site-packages/duplicity/path.py", line 716,
return gpg.GPGFile(1, self, gpg_profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in 
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 237,
in GPGWriteFile
file = GPGFile(True, path.Path(filename), profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

The solution to the above is simple – make sure the path includes /usr/local/bin, perhaps by including this at the start of the backup script:

export PATH=${PATH}:/usr/local/bin

Finally, when running an incremental backup, you may get this error:

Fatal Error: Neither remote nor local manifest is readable.

This can be solved by setting the HOME environment variable to /root assuming you’re running the backup as root (instead of the default /var/log for cron jobs):

export HOME=/root

Links, 9/18/2008

Andy — Thu, 18 Sep 2008 20:52:07 +0000

We’re Never Content – Amazon announces a forthcoming CDN layered on top of S3 with “edge locations on three continents” – presumably North America, Europe and Asia – “in order to deliver your content from the most appropriate location.” Presumably Amazon is planning to use this in-house for their digital media sales, or possibly for static content on their website.
Tape, Roman Chariots and Data Management – “But here’s where it gets insidious, we know look at the mess that tape has created, and instead of asking the question: ‘Is a data protection infrastructure predicated on creating whole copies on a regular basis flawed?’ We ask the question: ‘How can I make creating and storing full copies more efficient?’” An interesting read – nothing new – but somehow I don’t think that the solution the author would propose involves tape in an HSM scenario. Which is too bad, because an HSM environment using tape really can address the problems mentioned in the article, as well as other issues such as capacity and power.

Links 7/27/2008: S3 Outage Post-Mortem, Update 2 for VI 3 version 3.5

Andy — Sun, 27 Jul 2008 15:26:01 +0000

Amazon S3 Availability Event: July 20, 2008 – Amazon’s post-mortem on the 7/20 S3 outage. Excerpt: “We’ve now determined that message corruption was the cause of the server-to-server communication problems. More specifically, we found that there were a handful of messages on Sunday morning that had a single bit corrupted such that the message was still intelligible, but the system state information was incorrect.” (Seen first at Ars Technica.)
VMware has released Update 2 for VMware Infrastructure 3 version 3.5 (I think that’s the Full Official Name That Only A Committee Could Love…). Scott Lowe has a good summary; release notes are here. Most notable among the updates is the ability to use VSS to quiesce Windows VMs prior to snapshotting.

Linkage, 6/24/2008

Andy — Tue, 24 Jun 2008 23:19:37 +0000

S3 data corruption: “We’ve isolated this issue to a single load balancer that was brought into service at 10:55pm PDT on Friday, 6/20. It was taken out of service at 11am PDT Sunday, 6/22. While it was in service it handled a small fraction of Amazon S3’s total requests in the US. Intermittently, under load, it was corrupting single bytes in the byte stream. When the requests reached Amazon S3, if the Content-MD5 header was specified, Amazon S3 returned an error indicating the object did not match the MD5 supplied. When no MD5 is specified, we are unable to determine if transmission errors occurred, and Amazon S3 must assume that the object has been correctly transmitted.” (Seen at Daemonic Dispatches.)

thinking sysadmin » s3

Duplicity to Amazon S3 on FreeBSD: Building on the work of others

Links, 9/18/2008

Links 7/27/2008: S3 Outage Post-Mortem, Update 2 for VI 3 version 3.5

Linkage, 6/24/2008

Suggested Reading, 6/3/2008, Evening Edition