Best Practice #4: Enable Storage I/O Control
In official technical terms, VMware vSphere® Storage I/O Control is used to
provide I/O prioritization for virtual machines running on a group of VMware vSphere® hosts that have access to a shared storage pool. . . . It increases administrator productivity by reducing active performance management. Storage I/O Control can trigger device-latency monitoring that hosts observe when communicating with that datastore. When latency exceeds a set threshold, the feature engages to relieve congestion. Each virtual machine that accesses that datastore is then allocated I/O resources in proportion to their shares. — VMware
For vSphere 5.1, SIOC defaults to a 30ms average latency threshold before limiting IO on the noisy neighbor. For some shops, 20ms is the standard and you will want to adjust this.
But with all the differing performance characteristics of disks and disk tiers these days, it is much more difficult to deterministically set a threshold. As such, the process has been dynamicized (I just made that up) in 5.5:
The latency threshold is set to the value determined by the I/O injector (a part of Storage I/O Control). When the I/O injector calculates the peak throughput, it then finds the 90 percent throughput value and measures the latency at that point to determine the threshold. vSphere administrators can change this set throughput value to another percent value or they can continue to input a millisecond value. — VMware
As part of this new feature, vSphere 5.5 also introduced a new metric to provide a more guestOS-like measurement of this auto-calculated threshold, and it's called VM Observed Latency.
In essence, you could say that SIOC does some fancy calculations based on a normalized average of latency, IOPS, and IO request size, so that it intelligently understands the variety of workloads and IO operations—not all IO ops are created equal, after all.
A common misconception about SIOC is that it’s compute cluster based. The process of determining the datastore-wide average latency really reveals the key denominator – hosts connected to the datastore – (sic). All hosts connected to the datastore write to the IORMSTATS.SF file, regardless of cluster membership. Other than enabling SIOC, vCenter is not necessary for normal operations. Each connected host reads the IORMSTATS.SF file each 4 seconds and locally computes the datastore-wide average to use for managing the I/O stream. Therefor cluster membership is irrelevant.
— Frank Denneman
For block storage you would need to use vSphere version 4.1 or later. For NAS storage, you would need to use vSphere version 5.0 or later. — Debunking the Myths of Storage I/O Control