Duplicity to Amazon S3 on FreeBSD: Building on the work of others
(This post adds only a couple small details to work described at randys.org and cenolan.com – go there for background on this post and useful scripts for automated Duplicity backup to S3.)
First off, if you want to use Duplicity installed from FreeBSD Ports to backup to Amazon S3, be sure to also install the devel/py-boto
and security/pinentry-curses
ports.
If you attempt to run the backup script described at randys.org or cenolan.com from cron, you may run into an error similar to the following:
2009-03-01_01:05:05: ... backing up filesystem Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this is probably a bug. Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this is probably a bug. Traceback (most recent call last): File "/usr/local/bin/duplicity", line 583, in <module> with_tempdir(main) File "/usr/local/bin/duplicity", line 577, in with_tempdir fn() File "/usr/local/bin/duplicity", line 558, in main full_backup(col_stats) File "/usr/local/bin/duplicity", line 234, in full_backup bytes_written = write_multivol("full", tarblock_iter, globals.backend) File "/usr/local/bin/duplicity", line 148, in write_multivol globals.gpg_profile, globals.volsize) File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 240, in GPGWriteFile bytes_to_go = data_size - get_current_size() File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 232, in get_current_size return os.stat(filename).st_size OSError: [Errno 2] No such file or directory: '/tmp/duplicity-gM4CN9-tempdir/mktemp-iZknw0-2' Traceback (most recent call last): File "/usr/local/bin/duplicity", line 583, in <module> with_tempdir(main) File "/usr/local/bin/duplicity", line 577, in with_tempdir fn() File "/usr/local/bin/duplicity", line 558, in main full_backup(col_stats) File "/usr/local/bin/duplicity", line 232, in full_backup sig_outfp = get_sig_fileobj("full-sig") File "/usr/local/bin/duplicity", line 210, in get_sig_fileobj fh = globals.backend.get_fileobj_write(sig_filename) File "/usr/local/lib/python2.5/site-packages/duplicity/backend.py", line 354, in get_fileobj_write fh = dup_temp.FileobjHooked(tdp.filtered_open("wb")) File "/usr/local/lib/python2.5/site-packages/duplicity/path.py", line 716, return gpg.GPGFile(1, self, gpg_profile) File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112, in __init__ 'logger': self.logger_fp}) File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357, in run create_fhs, attach_fhs) File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401, in _attach_fork_exec if process.pid == 0: self._as_child(process, gnupg_commands, args) File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442, in _as_child os.execvp( command[0], command ) File "/usr/local/lib/python2.5/os.py", line 354, in execvp _execvpe(file, args) File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe func(fullname, *argrest) OSError: [Errno 2] No such file or directory Traceback (most recent call last): File "/usr/local/bin/duplicity", line 583, in <module> with_tempdir(main) File "/usr/local/bin/duplicity", line 577, in with_tempdir fn() File "/usr/local/bin/duplicity", line 558, in main full_backup(col_stats) File "/usr/local/bin/duplicity", line 234, in full_backup bytes_written = write_multivol("full", tarblock_iter, globals.backend) File "/usr/local/bin/duplicity", line 148, in write_multivol globals.gpg_profile, globals.volsize) File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 237, in GPGWriteFile file = GPGFile(True, path.Path(filename), profile) File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112, in __init__ 'logger': self.logger_fp}) File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357, in run create_fhs, attach_fhs) File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401, in _attach_fork_exec if process.pid == 0: self._as_child(process, gnupg_commands, args) File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442, in _as_child os.execvp( command[0], command ) File "/usr/local/lib/python2.5/os.py", line 354, in execvp _execvpe(file, args) File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe func(fullname, *argrest) OSError: [Errno 2] No such file or directory
The solution to the above is simple – make sure the path includes /usr/local/bin
, perhaps by including this at the start of the backup script:
export PATH=${PATH}:/usr/local/bin
Finally, when running an incremental backup, you may get this error:
Fatal Error: Neither remote nor local manifest is readable.
This can be solved by setting the HOME
environment variable to /root
assuming you’re running the backup as root (instead of the default /var/log
for cron jobs):
export HOME=/root
I’m getting that “Fatal Error: Neither remote nor local manifest is readable” message in my cron logs. Hopefully exporting the HOME variable does the trick. Thanks in advance if it does 🙂
Thanks for the tip – this had me puzzled for a while!
Also, why the security/pinentry-curses port? I don’t seem to require it…
@dave – It’s been a while, but if I recall correctly, there was a dependency problem in the ports build that manually building it first fixed. My guess would be that the dependency issue has probably been fixed now.
After looking through the duplicity code, I noticed this error was caused by gnupg package not being installed on my Cygwin system.
Installing it fixed the problem.