Continuing my ongoing recap of my recent vSphere 5.5 technical deep-dive, I now shift to Best Practices. This is installment eighteen in this series. To view all the posts in this series, click Geeknicks in the categories list.
Best Practice #3: ESXTOp (Memory)
A very powerful troubleshooting tool is included straight of the box (so to speak) with ESXi: ESXTOP. There is a ton of features to it, so we won't cover them all here; rather I will refer you to a great post by Duncan Epping, ESXTOP master.
If you missed my first post on ESXTOP, it includes how to get started if you a new to the utility.
If you missed my first post on ESXTOP, it includes how to get started if you a new to the utility.
Navigating ESXTOP into Memory Mode
When you start ESXTOP, to enter the Memory section of the utility type
m
I know, revolutionary. C gets you to CPU, M to Memory, D to Disk, and so forth. I will cover the rest in subsequent posts.
Once you are in the Memory monitoring section, press F to customize the fields that are being shown. Choose to show MCTL and NUMA, which I will explain in a moment, by scrolling down and highlighting them (you can use the o key to change the order if you like) and then press the space bar to check each line you want shown (or uncheck, as the case may be).
Once you are in the Memory monitoring section, press F to customize the fields that are being shown. Choose to show MCTL and NUMA, which I will explain in a moment, by scrolling down and highlighting them (you can use the o key to change the order if you like) and then press the space bar to check each line you want shown (or uncheck, as the case may be).
Using ESXTOP to Monitor Memory Utilization
Most of my metric explanations are from Interpreting ESXTOP Statistics, though sometimes in my own terms. I have also relied on Duncan Epping's ESXTOP Forum for many of the metrics measurements (as I will do in future posts), though in some places I choose different thresholds for one reason or another.
| Metric %SW-- %MCTLSZ N%L | Threshold >0 >0 <85–95 | What to Check Check for memory overcommitment on host. Check for memory overcommitment on host. Check for VMs with large memory entitlements. |
vSphere 5.5 | ESXTOP in Memory Mode
Now, for a deeper explanation, and I will include a couple of more metrics here in terms of explanation.
%SW— is the world Swap rate. %SWR/s is the current swap rate to/from the vswp file in terms of the percent of pages per second. %SWCUR is the historical swap rate, ranging from last power on (from power off state—not just a soft reboot) until now. %SWWAIT indicates whether or not the host is waiting to swap memory pages. Basically, if any of these %SW— metrics is greater than zero, you have memory overcommitment issues, and you will see a corresponding sluggishness in some or all of your VMs, depending upon their configuration.
%MCTLSZ is the world Memory Balloon rate. If you are experienced in ESXi, you will notice that the name of this metric is the name of the ballon driver: mctl, which is the VMM memory controller. As its name suggests, this metric indicates the balloon rate of memory pages per second reclaimed from the guest OS in order to supply active memory to other VMs. Any metric over the threshold of zero indicates memory overcommitment—which may not be affecting performance, and may, in fact, be by design, depending on the host configuration and the business requirements—and should be monitored carefully.
N%L is the world NUMA locality percentage rate. I have already written about the importance of NUMA locality, so I won't cover that again (or read the link to refresh yourself). This metric provides the percentage of NUMA locality for each world. Ideally, this would be 100%, however, because each world runs process through the CPU scheduler and other kernel worlds, it is likely that you will only see 98–99% on VMs.
A couple of things about this, however. 1) Depending upon the virtual machine and the hardware, you may have no choice at the moment but to space NUMA nodes; that's okay, just note it so that you can plan properly for future purchases. 2) Different workloads are more or less tolerant to microsecond latency within the memory paging system. So you might have a VM that is performing just fine (i.e., no one is complaining about it) and it's locality might be <80%; again just be aware of it. 3) Large SMP virtual machines that have to span physical processors will therefore also span physical NUMA nodes, and because of that the VMM will attempt to keep memory local to that CPU on the local node. So you may see a very low number on NUMA locality, but in reality that is a good thing because the VMM is keeping the memory pages resident to the processor scheduling them, which in turn provides more optimum performance (i.e., the pages are not traversing the processor interconnect).
You can check which nodes are actively being accessed by each VM under GST_ND#, where the # is the number of the local NUMA node. In the case below, there are only two NUMA nodes as there are only two physical sockets.
%SW— is the world Swap rate. %SWR/s is the current swap rate to/from the vswp file in terms of the percent of pages per second. %SWCUR is the historical swap rate, ranging from last power on (from power off state—not just a soft reboot) until now. %SWWAIT indicates whether or not the host is waiting to swap memory pages. Basically, if any of these %SW— metrics is greater than zero, you have memory overcommitment issues, and you will see a corresponding sluggishness in some or all of your VMs, depending upon their configuration.
%MCTLSZ is the world Memory Balloon rate. If you are experienced in ESXi, you will notice that the name of this metric is the name of the ballon driver: mctl, which is the VMM memory controller. As its name suggests, this metric indicates the balloon rate of memory pages per second reclaimed from the guest OS in order to supply active memory to other VMs. Any metric over the threshold of zero indicates memory overcommitment—which may not be affecting performance, and may, in fact, be by design, depending on the host configuration and the business requirements—and should be monitored carefully.
N%L is the world NUMA locality percentage rate. I have already written about the importance of NUMA locality, so I won't cover that again (or read the link to refresh yourself). This metric provides the percentage of NUMA locality for each world. Ideally, this would be 100%, however, because each world runs process through the CPU scheduler and other kernel worlds, it is likely that you will only see 98–99% on VMs.
A couple of things about this, however. 1) Depending upon the virtual machine and the hardware, you may have no choice at the moment but to space NUMA nodes; that's okay, just note it so that you can plan properly for future purchases. 2) Different workloads are more or less tolerant to microsecond latency within the memory paging system. So you might have a VM that is performing just fine (i.e., no one is complaining about it) and it's locality might be <80%; again just be aware of it. 3) Large SMP virtual machines that have to span physical processors will therefore also span physical NUMA nodes, and because of that the VMM will attempt to keep memory local to that CPU on the local node. So you may see a very low number on NUMA locality, but in reality that is a good thing because the VMM is keeping the memory pages resident to the processor scheduling them, which in turn provides more optimum performance (i.e., the pages are not traversing the processor interconnect).
You can check which nodes are actively being accessed by each VM under GST_ND#, where the # is the number of the local NUMA node. In the case below, there are only two NUMA nodes as there are only two physical sockets.
vSphere 5.5 | ESXTOP Showing Guest NUMA Nodes
%ZIP is the world Compressed Memory rate. Anything greater than zero indicates the host is compressing memory, likely due to overcommitment. Again, this may be by design, but is something of which you should be aware. Likewise %UNZIP is the rate at which the world is accessing that compressed memory.
Interpreting ESXTOP Memory Metrics
So, we just explained the metrics. Now what? Well, let's use the same example above (which I will repost below so you don't have to keep scrolling) and make some observations.
Note: This is just a small ESXi host with a couple of VMs on it, a few vSMP and a few configured for only a single vCPU.
Note: This is just a small ESXi host with a couple of VMs on it, a few vSMP and a few configured for only a single vCPU.
First, %SW— is all zeroes across the board; that's great! We know our few VMs are not swapping any memory out to the VSWP file. Likewise, %MCTLSZ looks good. We can see from the MCTL? that the balloon driver is enabled on some of the VMs but not on all. Regardless, none of the VMs with the balloon driver enabled are actively ballooning memory. So that's good.
Finally, our N%L NUMA Locality is also good; everything is about 98%, which is awesome! And we are also see which processes are running on which NUMA nodes, which is also good. But how is it that the first VM shows 100% locality but has values on both GST_ND0 and GST_ND1? The answer is above: the VMM is making sure the memory pages are remaining local to each physical processor, and this particular VM has 4 vCPUs spanning two sockets. So the VMM is doing its job to properly align page writes to the pCPU that is writing them.
Well, that's it! Happy troubleshooting, and stay tuned for the next installment where we will look at ESXTOP and Network metrics.
Finally, our N%L NUMA Locality is also good; everything is about 98%, which is awesome! And we are also see which processes are running on which NUMA nodes, which is also good. But how is it that the first VM shows 100% locality but has values on both GST_ND0 and GST_ND1? The answer is above: the VMM is making sure the memory pages are remaining local to each physical processor, and this particular VM has 4 vCPUs spanning two sockets. So the VMM is doing its job to properly align page writes to the pCPU that is writing them.
Well, that's it! Happy troubleshooting, and stay tuned for the next installment where we will look at ESXTOP and Network metrics.
RSS Feed
