Duplicity to Amazon S3 on FreeBSD: Building on the work of others

(This post adds only a couple small details to work described at randys.org and cenolan.com – go there for background on this post and useful scripts for automated Duplicity backup to S3.)

First off, if you want to use Duplicity installed from FreeBSD Ports to backup to Amazon S3, be sure to also install the devel/py-boto and security/pinentry-curses ports.

If you attempt to run the backup script described at randys.org or cenolan.com from cron, you may run into an error similar to the following:

2009-03-01_01:05:05: ... backing up filesystem
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 240,
in GPGWriteFile
bytes_to_go = data_size - get_current_size()
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 232,
in get_current_size
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory:
'/tmp/duplicity-gM4CN9-tempdir/mktemp-iZknw0-2'

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 232, in full_backup
sig_outfp = get_sig_fileobj("full-sig")
File "/usr/local/bin/duplicity", line 210, in get_sig_fileobj
fh = globals.backend.get_fileobj_write(sig_filename)
File "/usr/local/lib/python2.5/site-packages/duplicity/backend.py", line
354, in get_fileobj_write
fh = dup_temp.FileobjHooked(tdp.filtered_open("wb"))
File "/usr/local/lib/python2.5/site-packages/duplicity/path.py", line 716,
return gpg.GPGFile(1, self, gpg_profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 237,
in GPGWriteFile
file = GPGFile(True, path.Path(filename), profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

The solution to the above is simple – make sure the path includes /usr/local/bin, perhaps by including this at the start of the backup script:

export PATH=${PATH}:/usr/local/bin

Finally, when running an incremental backup, you may get this error:

Fatal Error: Neither remote nor local manifest is readable.

This can be solved by setting the HOME environment variable to /root assuming you’re running the backup as root (instead of the default /var/log for cron jobs):

export HOME=/root
Advertisements

4 comments

  1. tash

    I’m getting that “Fatal Error: Neither remote nor local manifest is readable” message in my cron logs. Hopefully exporting the HOME variable does the trick. Thanks in advance if it does 🙂

  2. dave

    Thanks for the tip – this had me puzzled for a while!

    Also, why the security/pinentry-curses port? I don’t seem to require it…

  3. Andy

    @dave – It’s been a while, but if I recall correctly, there was a dependency problem in the ports build that manually building it first fixed. My guess would be that the dependency issue has probably been fixed now.

  4. Simon

    After looking through the duplicity code, I noticed this error was caused by gnupg package not being installed on my Cygwin system.

    Installing it fixed the problem.