Here’s our in-house procedure for recovery; note that we do not have FlexClone licensed on our filers.

Prerequisites

• An existing VMware ESX infrastructure, connected to a NetApp filer NFS datastore; SnapRestore speeds the recovery process but is not mandatory – see discussion below.
• A backup script or system which coordinates VMware snapshots with NetApp snapshots – perhaps something along the lines of Rick Scherer’s script.
• A dedicated Linux restore VM, at a similar version level to the rest of your Linux VM infrastructure. This VM should have LVM support, but should not have any volume groups (VGs) or logical volumes (LVs) configured – volume group and logical volume names on the VMDK you are restoring from must not conflict with VGs and LVs already in use on the restore system; the simplest way to guarantee this is to simply not have any VGs or LVs.

Restore Procedure

• Restore the VMDK file from the appropriate snapshot to a new location in the datastore. With SnapRestore, this can be done as follows (one line in the filer CLI, restoring from snapshot sv_daily.0 to a new file – again, be extremely careful not to overwrite the current version of the VMDK in your datastore, consider restoring to an entirely different directory in the FlexVol):

  snap restore -t file -s sv_daily.0
-r /vol/vmware04_sis/system.example.com/system.example.com-restore.vmdk
/vol/vmware04_sis/system.example.com/system.example.com.vmdk

  Follow the prompts, verifying the restore path is correct and is not the path to your existing VMDK. Do the same for the flat VMDK file (again, one line, and, as before, use caution to make sure you do not clobber an existing file):

  snap restore -t file -s sv_daily.0
-r /vol/vmware04_sis/system.example.com/system.example.com-restore-flat.vmdk
/vol/vmware04_sis/system.example.com/system.example.com-flat.vmdk

  Without SnapRestore, you can simply mount the NFS export of the datastore on a Linux machine and use “cp” to copy the files out of the snapshot. For flat VMDK files, expect this copy to run for a substantial amount of time compared the nearly-instant recovery SnapRestore offers.

• Manually edit the line below “# Extent description” in the recovered .vmdk file to match the path to the recovered flat VMDK. In this case, it would look something like this:
  # Extent description
RW 20971520 VMFS "system.example.com-restore-flat.vmdk"
• Attach the recovered VMDK to your powered-off restore host. Boot the restore host.
• Once your restore host is up, use “pvscan”, “vgscan” and “lvscan” (each without arguments) as root to examine available LVM components. Then, use the “lvchange” command to activate the necessary volume group (in this case, “VolGroup00″):
  # lvchange -ay VolGroup00
• Mount the appropriate logical volume – for example, LogVol00 in VolGroup00:
  mount -o ro /dev/VolGroup00/LogVol00 /mnt
  Restore files by copying them out of /mnt.

Cleanup

• Shut down the Linux restore host.
• Remove the recovery VMDK – the files restored with SnapRestore or by “cp” above – from the restore host in the VMware Infrastructure Client.
• Delete the recovery .vmdk and -flat.vmdk files in the NFS datastore. Don’t screw up here: Be sure to delete the recovery files only, not the working VMDK.

First off, if you want to use Duplicity installed from FreeBSD Ports to backup to Amazon S3, be sure to also install the devel/py-boto and security/pinentry-curses ports.

If you attempt to run the backup script described at randys.org or cenolan.com from cron, you may run into an error similar to the following:

2009-03-01_01:05:05: ... backing up filesystem
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 240,
in GPGWriteFile
bytes_to_go = data_size - get_current_size()
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 232,
in get_current_size
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory:
'/tmp/duplicity-gM4CN9-tempdir/mktemp-iZknw0-2'

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 232, in full_backup
sig_outfp = get_sig_fileobj("full-sig")
File "/usr/local/bin/duplicity", line 210, in get_sig_fileobj
fh = globals.backend.get_fileobj_write(sig_filename)
File "/usr/local/lib/python2.5/site-packages/duplicity/backend.py", line
354, in get_fileobj_write
fh = dup_temp.FileobjHooked(tdp.filtered_open("wb"))
File "/usr/local/lib/python2.5/site-packages/duplicity/path.py", line 716,
return gpg.GPGFile(1, self, gpg_profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 237,
in GPGWriteFile
file = GPGFile(True, path.Path(filename), profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

The solution to the above is simple – make sure the path includes /usr/local/bin, perhaps by including this at the start of the backup script:

export PATH=${PATH}:/usr/local/bin

Finally, when running an incremental backup, you may get this error:

Fatal Error: Neither remote nor local manifest is readable.

This can be solved by setting the HOME environment variable to /root assuming you’re running the backup as root (instead of the default /var/log for cron jobs):

export HOME=/root