Gotcha #3: Don't Open a Single File on More than 8 Hosts (unless it is on NFS)
This gotcha applies to any file that would be opened by more than 8 hosts, and here are some examples:
- ISO, custom or otherwise (for example a driver ISO to install a custom VIB file)
- Golden Master VDMK for a linked clone scenario (this is why you can't have more than 8 hosts accessing a linked clone for Horizon View)
Note: however, that up to 32 ESXi 5.1+ hosts can open a single file on NFS datastores.
The ISO situation actually comes up a LOT, because quite often administrators forgot to disconnect (and disassociate) their ISO images from their VMs after they do their installs (here's how to find which VMs are currently accessing ISO images).
So why does it happen?
In short, the file lock holder runs out of space. Each file has 512 bytes allocated as part of its metadata to maintain a "lock" on the file to prevent other hosts from accessing/modifying it inappropriately (ever had to release a file lock on a failed ESXi host?—see VMware KB 10051).
So why is NFS unaffected by this? Because there are two types of locking: mandatory, in which the ESXi kernel forces the file lock at all times, and advisory, which is more flexible. The NFS protocol itself does not support mandatory locking, only advisory locking as shown below.
Note: You must have vSphere 5.1 or higher installed to take advantage of the NFS advisory lock mechanism; this feature is not supported on 4.1 and earlier.
This is, incidentally, also a good use case for why you might choose to make NFS datastore for large-scale ISO content repositories.