> rpm -q slocate
slocate-2.7-13.el4.8.i386

or

> rpm -q mlocate
mlocate-0.15-1.el5.2.x86_64

If so, multiple RHEL VMs plus mlocate or slocate may be adding up to an array-crushing 4:02am shared storage load and latency spike for you. Before being addressed, this spike was bad enough at my place of employment (when combined with a NetApp Sunday-morning disk scrub) to cause a Windows VM to crash with I/O errors. Ouch.

Details and ideas for resolution:

By default, a line in /etc/crontab runs the scripts within /etc/cron.daily at 4:02am each morning:

02 4 * * * root run-parts /etc/cron.daily

One of those scripts – mlocate.cron or slocate.cron, depending on your OS version – launches updatedb; as the man page says, “updatedb creates or updates a database used by locate(1).” (The “locate” binary is a filesystem search tool, see “man locate” for more information.) Updatedb refreshes its database by walking the filesystem, generating a fair amount of I/O on a single system. Imagine upwards of thirty of these running in parallel through VMDKs on one shared storage system carrying out internal maintenance at the same time, and you’re pretty much picturing the problem my employer had.

I see three options for addressing this issue:

1) Uninstall mlocate or slocate. If you don’t currently use “locate” and you’re not interested in learning to use a tool that will likely make you more effective at your job (again, see “man locate”), this is probably the best option. (Yeah, I know, people that fit this bill generally don’t read blogs more technical than this one, so I could probably have skipped it here. Consider it an option for completeness, or if you really need to strip down an install.)

2) Disable the scheduled job by removing mlocate.cron or slocate.cron from /etc/cron.daily. This keeps locate available for your use, but requires that you update locate’s database ad-hoc and interactively by running the following as root:

# updatedb

This will take a few minutes to return, depending on the size of your file systems.

I don’t recommend this option either; at least it doesn’t fit the way I work. I often find myself using locate in high-pressure situations in which I need to quickly get a file location on a system. Waiting minutes for updatedb to return is extra painful when every second counts.

3) Stagger when updatedb runs by inserting a random delay into the script.. This is my preferred alternative; locate’s database is kept current automatically, and your storage doesn’t have to bear a sudden spike in load. I implemented this by adding the lines in bold (lines 2-7 if your browser doesn’t display the bold text clearly):

#!/bin/sh
# sleep up to two hours before launching job:
value=$RANDOM
while [ $value -gt 7200 ] ; do
value=$RANDOM
done
sleep $value
nodevs=$(< /proc/filesystems awk '$1 == "nodev" { print $2 }')
renice +19 -p $$ >/dev/null 2>&1
/usr/bin/updatedb -f "$nodevs"


The added code inserts a pseudo-random sleep delay of up to two hours before updatedb runs, with the key being the built-in Bash function $RANDOM. In our environment, this removed a 2000 IOPS spike at 4:02am, and eliminated a corresponding jump in filer latency. Obviously, adjust the delay period as appropriate for your environment. Additionally, be sure to add this change to your configuration management or installation management tools so that all of your RHEL and RHEL-derived VMs get the updated script.

Using $RANDOM to avoid this variant of the thundering herd problem also works nicely for a range of similar problems; I believe I first saw it at Moundalexis.com.

(This problem may apply to other Linux distributions being run as VMs, and FreeBSD does something equivalent – weekly – with /etc/periodic/weekly/310.locate. A similar solution can be applied to these environments, if necessary.)

Here’s our in-house procedure for recovery; note that we do not have FlexClone licensed on our filers.

Prerequisites

• An existing VMware ESX infrastructure, connected to a NetApp filer NFS datastore; SnapRestore speeds the recovery process but is not mandatory – see discussion below.
• A backup script or system which coordinates VMware snapshots with NetApp snapshots – perhaps something along the lines of Rick Scherer’s script.
• A dedicated Linux restore VM, at a similar version level to the rest of your Linux VM infrastructure. This VM should have LVM support, but should not have any volume groups (VGs) or logical volumes (LVs) configured – volume group and logical volume names on the VMDK you are restoring from must not conflict with VGs and LVs already in use on the restore system; the simplest way to guarantee this is to simply not have any VGs or LVs.

Restore Procedure

• Restore the VMDK file from the appropriate snapshot to a new location in the datastore. With SnapRestore, this can be done as follows (one line in the filer CLI, restoring from snapshot sv_daily.0 to a new file – again, be extremely careful not to overwrite the current version of the VMDK in your datastore, consider restoring to an entirely different directory in the FlexVol):

  snap restore -t file -s sv_daily.0
-r /vol/vmware04_sis/system.example.com/system.example.com-restore.vmdk
/vol/vmware04_sis/system.example.com/system.example.com.vmdk

  Follow the prompts, verifying the restore path is correct and is not the path to your existing VMDK. Do the same for the flat VMDK file (again, one line, and, as before, use caution to make sure you do not clobber an existing file):

  snap restore -t file -s sv_daily.0
-r /vol/vmware04_sis/system.example.com/system.example.com-restore-flat.vmdk
/vol/vmware04_sis/system.example.com/system.example.com-flat.vmdk

  Without SnapRestore, you can simply mount the NFS export of the datastore on a Linux machine and use “cp” to copy the files out of the snapshot. For flat VMDK files, expect this copy to run for a substantial amount of time compared the nearly-instant recovery SnapRestore offers.

• Manually edit the line below “# Extent description” in the recovered .vmdk file to match the path to the recovered flat VMDK. In this case, it would look something like this:
  # Extent description
RW 20971520 VMFS "system.example.com-restore-flat.vmdk"
• Attach the recovered VMDK to your powered-off restore host. Boot the restore host.
• Once your restore host is up, use “pvscan”, “vgscan” and “lvscan” (each without arguments) as root to examine available LVM components. Then, use the “lvchange” command to activate the necessary volume group (in this case, “VolGroup00″):
  # lvchange -ay VolGroup00
• Mount the appropriate logical volume – for example, LogVol00 in VolGroup00:
  mount -o ro /dev/VolGroup00/LogVol00 /mnt
  Restore files by copying them out of /mnt.

Cleanup

• Shut down the Linux restore host.
• Remove the recovery VMDK – the files restored with SnapRestore or by “cp” above – from the restore host in the VMware Infrastructure Client.
• Delete the recovery .vmdk and -flat.vmdk files in the NFS datastore. Don’t screw up here: Be sure to delete the recovery files only, not the working VMDK.

• Wow, the VI client is really buggy and slow. It crashes for me at least once a day. We haven’t upgraded from 3.0.2 to 3.5 yet, hopefully some of this is fixed in the new version.
• There also seems to be an IP storage bug on reboot where ESX server doesn’t reconnect to iSCSI or NFS storage – possibly something to do with a transient name resolution or default route problem in ESX. I haven’t gotten any good suggestions out of VMware support about this, but the case is still open.
• VMware does a good job of pricing ESX so it’s expensive but affordable for non-profits.

All in all, I have to admit, I’m pretty pleased with ESX so far, but I also know that the honeymoon isn’t over – we only have one virtual machine in quasi-production so far.