ESX Swap on NFS or Not?
Scott Lowe recently linked to a VMware KB article entitled Storing swap files on VMFS when running virtual machines from NFS. The article (from 3/31/2008) is perhaps the latest word from VMware in the frustrating back-and-forth on whether placing an ESX VM’s swap on NFS is acceptable or not.
The KB entry itself isn’t directly about (not) placing VM swap on NFS – it details how to specify the location of swap in a VM’s config file. Regarding the question of swap location, the article (weakly) recommends against placing swap on NFS:
This articles describes changing the default location for your virtual machines [sic] swap files. You may need to use this if you are running virtual machines from NFS storage.
Note: VMware recommends storing your swap on a VMFS3 volume, when running virtual machines on NFS storage.
(Emphasis mine.) The article provides none of the reasoning behind this recommendation, and a search of the VMware Knowledge Base for “nfs swap” only returns this article. There are quite a few mentions of not putting swap on NFS in the VMware Communities forums, citing performance reasons, but, as far as I could find, no support for the performance issue assertion.
Personally, I’m not buying the performance claims without substantiation – NFS and iSCSI to the same filer should be nearly equally performant. I could imagine a difference in locking behavior, but I’m having trouble imagining a scenario where lock contention with a VM swap file would become an issue.
On the other side of the issue, I’ve been hearing from a few folks – mostly NetApp employees or users – that placing swap on NFS is okay. On the viops site, Steve Chambers writes in a preview of VMworld TA 2784 (which I did not attend) that “there is no need to place swap space on non IP storage.” IP storage could mean iSCSI here, of course, but in the VM /ETC write up of TA 2784, Rich Brambley’s notes state: “Do not place [swap] space on other storage – do not place the vswap on VMFS. Follow best practices and keep the swap on the NFS storage.”
So, who’s right here? VMware or NetApp? I suppose the conservative thing is to keep the swap on VMFS until VMware tells us otherwise, with the only negatives being the increased operational complexity and reduced storage utilization – that’s what I’m doing. I can’t help but wonder if this is like VMware’s Linux timekeeping recommendations, though – conflicting information subject to change – and hopefully clarification – at a later date.
Update, 11/24/2008: Keeping ESX swap on NFS with the VMhome directory is considered the current best practice.
Hey Andy, your reference to what I said in http://viops.vmware.com/home/docs/DOC-1157 – that actually came from a gentleman called Paul Manning who is our storage expert. I will seek clarification from him.
had an ESX PSOD, support looked at the log and told me it’s because ESX could not retrieve data from VM swap file on NFS–our NFS storage went offline for a few minutes during a reboot (it’s a lab environment).
and said if the swap file is on iSCSI, then ESX would hang, but not PSOD.
that’s probably why VMWare recommends putting swap file on vmfs.