Best Practice #3: ESXTOp (Memory)
If you missed my first post on ESXTOP, it includes how to get started if you a new to the utility.
Once you are in the Memory monitoring section, press F to customize the fields that are being shown. Choose to show MCTL and NUMA, which I will explain in a moment, by scrolling down and highlighting them (you can use the o key to change the order if you like) and then press the space bar to check each line you want shown (or uncheck, as the case may be).
What to Check
Check for memory overcommitment on host.
Check for memory overcommitment on host.
Check for VMs with large memory entitlements.
%SW— is the world Swap rate. %SWR/s is the current swap rate to/from the vswp file in terms of the percent of pages per second. %SWCUR is the historical swap rate, ranging from last power on (from power off state—not just a soft reboot) until now. %SWWAIT indicates whether or not the host is waiting to swap memory pages. Basically, if any of these %SW— metrics is greater than zero, you have memory overcommitment issues, and you will see a corresponding sluggishness in some or all of your VMs, depending upon their configuration.
%MCTLSZ is the world Memory Balloon rate. If you are experienced in ESXi, you will notice that the name of this metric is the name of the ballon driver: mctl, which is the VMM memory controller. As its name suggests, this metric indicates the balloon rate of memory pages per second reclaimed from the guest OS in order to supply active memory to other VMs. Any metric over the threshold of zero indicates memory overcommitment—which may not be affecting performance, and may, in fact, be by design, depending on the host configuration and the business requirements—and should be monitored carefully.
N%L is the world NUMA locality percentage rate. I have already written about the importance of NUMA locality, so I won't cover that again (or read the link to refresh yourself). This metric provides the percentage of NUMA locality for each world. Ideally, this would be 100%, however, because each world runs process through the CPU scheduler and other kernel worlds, it is likely that you will only see 98–99% on VMs.
A couple of things about this, however. 1) Depending upon the virtual machine and the hardware, you may have no choice at the moment but to space NUMA nodes; that's okay, just note it so that you can plan properly for future purchases. 2) Different workloads are more or less tolerant to microsecond latency within the memory paging system. So you might have a VM that is performing just fine (i.e., no one is complaining about it) and it's locality might be <80%; again just be aware of it. 3) Large SMP virtual machines that have to span physical processors will therefore also span physical NUMA nodes, and because of that the VMM will attempt to keep memory local to that CPU on the local node. So you may see a very low number on NUMA locality, but in reality that is a good thing because the VMM is keeping the memory pages resident to the processor scheduling them, which in turn provides more optimum performance (i.e., the pages are not traversing the processor interconnect).
You can check which nodes are actively being accessed by each VM under GST_ND#, where the # is the number of the local NUMA node. In the case below, there are only two NUMA nodes as there are only two physical sockets.
Note: This is just a small ESXi host with a couple of VMs on it, a few vSMP and a few configured for only a single vCPU.
Finally, our N%L NUMA Locality is also good; everything is about 98%, which is awesome! And we are also see which processes are running on which NUMA nodes, which is also good. But how is it that the first VM shows 100% locality but has values on both GST_ND0 and GST_ND1? The answer is above: the VMM is making sure the memory pages are remaining local to each physical processor, and this particular VM has 4 vCPUs spanning two sockets. So the VMM is doing its job to properly align page writes to the pCPU that is writing them.
Well, that's it! Happy troubleshooting, and stay tuned for the next installment where we will look at ESXTOP and Network metrics.