I am going to follow-up with the conference by posting a couple of #ProTips that, according to participant feedback, were the most interesting to them from my talk. Here's the list:
- Disabling Delayed TCPIP Acknowledgements for Better iSCSI Performance
- Removing Removable Devices to Conserve CPU Cycles for Better Performance
- Single-Root I/O Virtualization (SR-IOV) on Cisco UCS for Better Performance Through VM-FEX
You can also read more about my vSphere 5.5 Performance Best Practices series that I did last April/May.
Disable Delayed ACK for Better iSCSI Performance
As always, you should consult your vendor best practices for iSCSI documentation in conjunction with VMware KB 1002598 in order to determine if this practice is ideal for your environment. I know from the Equallogic Best Practices guide that this does, in fact, apply to all their arrays, and the VMware KB mentions LeftHand in the vmkernel log errors. But according to the A “Multivendor Post” on using iSCSI with VMware vSphere—which was written by all the major players in the iSCSI market including NetApp—this issue can apply to just about any iSCSI storage environment, because they all rely on standard TCP protocols.
Delayed ACK is specified in RFC 813: Window and Acknowledgement Strategy in TCP. The "TCP Window" is the strategy for handling TCP congestion, or as we commonly call it, Flow Control. The "TCP Acknowledgement" is the strategy for maintaining data integrity while preserving CPU cycles.
The acknowledgement mechanism is at the heart of TCP. Very simply, when data arrives at the recipient, the protocol requires that it send back an acknowledgement of this data. . . . The protocol contains only a general assertion that data should be acknowledged promptly, but gives no more specific indication as to how quickly an acknowledgement must be sent, or how much data should be acknowledged in each separate acknowledgement. — RFC 813
Why does this create a delay? Because iSCSI storage arrays want to make sure that their data has been received! Imagine that! What happens in simplistic terms is that the array will throttle the network performance so that ACKs will be received in a timely way.
Below you can see how the TCP flow occurs when you turn off Delayed ACK for iSCSI:
Note: For vSphere 3.x/4.x installations, consult VMware KB 1002598 for the proper method to disable Delayed ACK.
To disable Delayed ACK: (the easiest way is to turn off by Discovery Address, since all array LUNs from the same vendor typically follow the same best practices)
- Log in to the vSphere Client and select the host.
- Navigate to Configuration | Storage Adapters.
- Click the iSCSI vmhba that you want to modify and then select Properties.
- Select the discovery address, and click Dynamic Discovery and then Server Address.
- Click Settings | Advanced.
- Deselect Inherit From parent.
- Deselect DelayedAck.
- Click OK about a dozen times, and then reboot the host—yes, this requires a reboot.