Best Practice #4: ESXTOP (Network)
If you missed my first post on ESXTOP (CPU), it includes how to get started if you a new to the utility. I have also already covered ESXTOP (Memory).
What to Check
Physical NIC utilization may be too high.Physical NIC utilization may be too high.
%DRPTX— is the world Transmission Drop rate. This metric monitors the physical NICs for dropped outbound packets. If you are having trouble with a particular VM, you can use this to check the utilization for that particular uplink, regardless of whether it is on a standard vSwitch or a distributed vSwitch—that's one of my favorite features!
If you have a %DRPTX greater than zero, it means that the uplink is the dropping the packets probably for one of three reasons, in my experience: 1) NIC is over-utilized, and you need more physical NICs—maybe just reallocating from another portgroup, or maybe you need to buy one; 2) NIC teaming is misconfigured, say, for example, you are using Route Based on IP Hash but you are not using a port channel (big no-no).
Route based on IP Hash load balancing requires that the physical switch ports be combined into an EtherChannel (sometimes called an aggregation bond, port-channel, or trunk). This ensures that the same hashing algorithm is used for traffic returning in the opposite direction. — VMware KB 2006129
%DRPRX— is the world Reception Drop rate. This metric monitors the physical NICs for dropped inbound packets. If you are having trouble with a particular VM, and your %DRPRX is greater than zero, it is likely due to the same three causes above, plus two more important considerations.
1) CPU allocation. If the VM is dropping packets, it may simply be that the CPU is overworked and can't process the packets fast enough, which results in #2—the buffer filling up. Add another processor or increase the CPU limit to accommodate the needed cycles.
2) Buffer size. Every NIC driver has a set buffer size in the VMkernel. If the buffer fills up, the likely result is dropped packets. And these packets are not actually dropped at the kernel, but between the kernel and the guest OS:
Esxtop might show receive packets dropped at the virtual switch if the virtual machine’s network driver runs out of receive (RX) buffer memory the packets will be treated in a FIFO (First in first out basis) which the network can be degraded. Even though esxtop shows the packets as dropped at the virtual switch, they are actually dropped between the virtual switch and the guest operating system driver.
— VMware KB 1010071
Note: This is just a small ESXi host with a couple of VMs on it, a few vSMP and a few configured for only a single vCPU.
Well, that's it! Happy troubleshooting, and stay tuned for the next installment where we will look at ESXTOP and Disk metrics.