Simple IT made complex

Update: The vCPU benchmark can be found here: https://dryerr.wordpress.com/2010/04/02/vcpu-performance-degradation/

Having recently made a simple vCPU benchmark, I thought about posting the results here. But for those numbers, you really need to understand how vCPU works on the ESX host, or more important, how multiple vCPU works on the ESX host.

First, lets look at how a normal CPU works in a physical environment
The term CPU cycle is the key here. A CPU cycle happens with every Hz, so a 2GHz CPU can do up to 2 billion CPU cycles each second. If you then add another CPU to the mix, you can now do up to 4 billion CPU cycles per second.
So far, pretty straightforward, but, with 2 CPUs, each cycle now happens twice, once on CPU0 and once on CPU1. This doesn’t mean that both CPUs will do some work, the cycle might do something demanding for CPU0, but be empty for CPU1, leaving a free CPU for more work.

So what’s the interesting part here? Both CPUs must run a cycle at the same time. This is probably done to avoid synchronization problems or something similiar.

Now, in a virtual environment
Lets say you have a VM, this VM has 1 vCPU. With every cycle, the vCPU asks the hypervisor for some real CPU to do the work. The hypervisor then tries to reserve a physical CPU to take care of the cycle.
Now your VM has 2 vCPU, what happens then, is that the VM asks the hypervisor to do 2 CPU cycles, and the hypervisor then tries to reserve 2 physical CPUs to do the cycles. This is all very good and well, but it presents a problem. It’s harder to reserve 2 physical CPUs than 1 (oh really?).

For fun, imagine you have an ESX server with 8 physical CPUs available. You also have 5 VMs with 1 vCPU each. In this case, every VM shouldn’t really fight for CPU, as there is always one available. Now add another 2 VMs with 2 vCPU each, a total of 7 VMs, and 9 vCPUs. All the VMs with 2 vCPU will have a harder time reserving CPU time, because the 1 vCPU VMs can jump in at any free CPU. Imagine all the VMs is actively using a lot of CPU. This requires 9 physical CPUs, but we only have 8, so the ESX does what it’s best at, queuing the work and making our virtual environment doing it’s thing. However, the VMs with 2 vCPU will have a hard time reserving CPU time, because everytime a CPU frees up, a single vCPU jumps in and grabs it. I’m pretty sure ESX has a great way to reduce this kind of grabbing, but it does matter. A lot.

If you look on the performance charts on the VM, you have a graph called “CPU Ready”. I find the title a bit confusing, because what it means is, that’s the amount of time the vCPU was ready to do work, but couldn’t get time on a physical CPU. You can’t avoid some CPU ready delay, not when you’re doing virtual environments.