This is part four of my ongoing recap of my recent vSphere 5.5 technical deep-dive. To view all the posts in this series, click Geeknicks in the categories list.

Backing up vCenter is a crucial element to all installations. However, few people take the time to do it. Here are a couple of resources to help you. 

• Backing Up 4.x/5.0 vCenter ADAM instances
• Backing Up 5.1 vCenter SSO instances
• Backing Up vCenter SQL Database instances

Why is it so important to backup vCenter? First, because it is the heart of your virtual data center. Take, for example, virtual Distributed Switches (vDS). While traffic will continue to flow in the event of a vCenter outage (when setup using "Static Binding"), you won't be able to make any changes. That alone should cause you to backup, and backup often.

Additionally, if there is a (unlikely) probably during the vCenter installation, that irreparably changes the database or corrupts it...how will you get everything back? The only solution, if you are without backup, is to rebuild, and that is a lot of time and work. 

Save yourself some headache, and back that thing up! 

This is part three of my ongoing recap of the technical deep-dive  into vSphere 5.5 my company, Adcap Network Systems, recently held. To view all the posts in this series, click Geeknicks in the categories list.

This is actually not a vSphere 5.5 gotcha in particular, but a vSphere gotcha in general. I first came across it doing large rollout for the DoD, where we would be setting up 20 or 30 Exchange hosts at a time, and as soon as we would mount the ISO to the ninth host, all the hosts would effectively come to a halt during their installation—and a failed Exchange installation is, well, messy to clean up.

This gotcha applies to any file that would be opened by more than 8 hosts, and here are some examples:

• ISO, custom or otherwise (for example a driver ISO to install a custom VIB file)
• Golden Master VDMK for a linked clone scenario (this is why you can't have more than 8 hosts accessing a linked clone for Horizon View)
  Note: however, that up to 32 ESXi 5.1+ hosts can open a single file on NFS datastores.

The ISO situation actually comes up a LOT, because quite often administrators forgot to disconnect (and disassociate) their ISO images from their VMs after they do their installs (here's how to find which VMs are currently accessing ISO images).

So why does it happen?

In short, the file lock holder runs out of space. Each file has 512 bytes allocated as part of its metadata to maintain a "lock" on the file to prevent other hosts from accessing/modifying it inappropriately (ever had to release a file lock on a failed ESXi host?—see VMware KB 10051).

Because the lock space allocated per file is 512 bytes, and the Universally Unique Identifier (UUID) is 64 bytes, well, it is a case of simple math: 512 / 64 = 8. That's how many hosts can fit within this.

So why is NFS unaffected by this? Because there are two types of locking: mandatory, in which the ESXi kernel forces the file lock at all times, and advisory, which is more flexible. The NFS protocol itself does not support mandatory locking, only advisory locking as shown below. 

In these situations, the NFS protocol will interact with the advisory lock and say, "Hey, I need access to this file, can you free up a lock for me?" The locking mechanisms in the kernel will work to free up a lock, commit the write, and and the process will repeat with other hosts, quite rapidly, as you might imagine. 

Note: You must have vSphere 5.1 or higher installed to take advantage of the NFS advisory lock mechanism; this feature is not supported on 4.1 and earlier.

This is, incidentally, also a good use case for why you might choose to make NFS datastore for large-scale ISO content repositories. 

Recently, Adcap Network Systems hosted a technical deep-dive called Geeknicks into vSphere 5.5 (yours truly was the presenter). For those unable to attend, and because those who attended, statistically, only retained about 5–10%, I am covering much of the material here in a new series of posts. To view all the posts in this series, click Geeknicks in the categories list.

vSphere 5.1 introduced Single Sign-On, or SSO. With that, a user's Active Directory (AD) credentials, when properly configured, can be used to sign on with vCenter and administrator accordingly. 

The operative phrase here being, "when properly configured." There are a lot of headaches that people have had with SSO. In fact, when I recently did a poll about vSphere 5.5's best new features, one person commented, "I wish you had an entry for SSO that actually worked" (LinkedIn profile and group permissions required to view comment).

A lot of people have waited to upgrade their environment, until a "version 2" of SSO had come out, kind of like the way that many folks will wait to upgrade major version of Microsoft software until the first Service Pack is released—they don't want to be early adopters.

vSphere 5.5 is the "version 2" so-to-speak of SSO, and having used both, I can say it is much better. For instance, if you install vCenter 5.5 while logged in via a domain account, the installer is smart enough to know you want SSO installed and configured for that domain, so it does it for you.

The gotcha comes in two forms: installing vCenter 5.5 for the first time (you have a new environment), and rebuilding vCenter 5.5 from scratch instead of upgrading. Here's what happens. 

1. You install vCenter, and configure SSO.
2. You login, and everything looks great.
3. You add your hosts and such.
4. You grant access to the rest of your team—doh! They login, but they can't see anything! Everything shows up with 0 inventory! What in the world??!! Yeah. That's about right.

The problem is that while SSO enables a "single sign-on" there is still two places you have to set Role-Based Authentication Control (RBAC) permissions: SSO configuration, and Inventory configuration. Most people remember the first, but for some reason forget the second. So in essence, what happens is admins set it up, and grant others the right to login to the system, but they have no rights to actually see anything. 

The solution is to navigate to the top-level vCenter inventory object, or press the web client shortcut (^ +⌥+2 – vCenter Home View) and select Manage | Permissions. From there, add your groups (preferably, as a best practice) and users according to their business requirements. Wallah! Now you have a fully populated inventory. 

Recently, Adcap Network Systems hosted a technical deep-dive called #Geeknicks into vSphere 5.5 (yours truly was the presenter). For those unable to attend, and because those who attended, statistically, only retained about 5–10%, I am covering much of the material here in a new series of posts. To view all the posts in this series, click Geeknicks in the categories list.

With the release of vSphere 5.5 and ESXi 5.5, there was a corresponding revision upgrade of virtual machine hardware to version 10. Here is a summary of VM hardware versions and their ESXi or ESX compatibility (for the full list, see VMware KB 1003746):

     10     5.5
       9     5.1
       8     5.0
       7     4.x
       4     3.x

So, should you upgrade? Here's the first gotcha. And it's a big enough gotcha that there is VMware KB 2061336 to address it!  If you upgrade, you will incur two potentially problematic issues:

1. The VM will only be editable in the vSphere Web Client; the legacy (desktop) client will allow you to view but not edit the hardware. 
2. This is especially problematic if you don't have a good vCenter implementation (which you should) or have some standalone hosts...you won't be able to manage them the way in which you are accustomed.
3. If you have a standalone host, you will have to use the vCLI or Power CLI to manage the VMs on that host; the only supported GUI is the vSphere 5.5 Web Client.

Tomorrow: Gotcha.02 | SSO and Inventory Permissions

There are so many awesome things about the vCenter Server Appliance (vCSA). Here's a brief list to start:

• Manageable via SSH
• Works with SSO and AutoDeploy
• Scales up to 1,000 hosts and 10,000 virtual machines (with Oracle DB) per instance
• Up and running in minutes, not hours

And there's more . . . the vCenter Simulator, or vCSim (at least that's how I abbreviate it).

William Lam first put me onto this with his blog, and there are several other great write-ups about it as well. To get yourself started:

Boom! You just deployed 15,000 virtual machines on 1000 hosts!