Number of Guest CPU's recommendation

This is for discussing general topics about how to use VirtualBox.
Locked
The Raven
Posts: 82
Joined: 18. Aug 2016, 20:43

Number of Guest CPU's recommendation

Post by The Raven »

Fairly new to VirtualBox but not to Hypervisors.

I've been trying to gain more insight into VirtualBox by reading through various
forum entries. There seems to be a consensus that a guest should not have the
same number of virtual processors assigned to it as there are real processors.
An example quote is :
you have a dual core host and you assigned both cores to the VM, leaving 0 cores for the host


I'm a little mystified by this, for the following reasons :
In the above example of a two-way real processor, what is the difference between running a guest
with two virtual cpus (thus potentially using both real processors) or running two guests, both defined
with a single virtual processor, which I would assume could both be dispatched simultaneously and
therefore also use both real cpus ?

The argument seems to be that the virtual cpus will somehow block out the host. However, the host will
only ever dispatch the guests virtual cpu when it's ready to do so. It's the host driving the guest.

Even so, I can see that there may be moments when both real cpus are executing guest virtual cpus and
the host - whether it's VirtualBox or the true Host (say Linux) - need to execute. Firstly, I would assume
these instances do not occur often and it's far more likely that the guests are regularly leaving VT-X operation
by a VM exit. Even if they are not, I thought that's what the VT-X pre-emption timer was for, so that the
Host (VirtualBox) can regain control after a specified time interval. Also, Host I/O interrupts are likely
to be happening almost all the time, thus resulting a VM exit on the processor fielding the interrupt.
If all else fails, I would expect the true host (Linux say), would always regain control via a timer-pop as it's
based on a pre-emptive time-slice dispatcher. It would then dispatch tasks based on priority/readiness.

The biggest problem I've seen with Hypervisors is not the Host being blocked but a multi-processor guest
having one or more of its (virtual) processors not being dispatched frequently enough. Thus usually results
in spin-lock type problems in the guest.

My previous experience of Hypervisors involved bare-metal types, so I'm assuming a hosted type such as VirtualBox
has different scheduling/dispatching considerations ?
Any insight to the operation of VirtualBox that relates to the recommendation of the number of virtual cpus
in relation to the number of real cpus would be most welcome.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Number of Guest CPU's recommendation

Post by socratis »

What's going on today? It has to be the virtual vs. actual day!
Take a look at today's viewtopic.php?f=9&t=79731 ignoring the AMD specific stuff. And yes, you're right on most of your points, as far as I understand it myself, but I'm not as knowledgeable.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
The Raven
Posts: 82
Joined: 18. Aug 2016, 20:43

Re: Number of Guest CPU's recommendation

Post by The Raven »

No I hadn't seen that thread, thanks.

I should have stated in my initial post that I was ignoring the (ongoing) confusion
regarding AMD cores etc. My points were purely related to real cpus (not hyperthreaded
or anything else) and the number of virtual cpus defined to any particular guest.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Number of Guest CPU's recommendation

Post by mpack »

Yes, assigning all N cores to one VM is just as bad as running N VMs. I'm not sure what led you to believe that the latter is more acceptable - it isn't. But, the former arises more often in these forums.

You are not prevented from assigning all of your cores to a guest. However that means that the host may not be able to service events in a timely fashion. And bear in mind that the host OS is ALWAYS executing code, unless its asleep. If the clock is running then it must execute code, so it's not like sometimes the host can manage without a CPU.

If the host has to share a CPU with the VM then this must degrade the responsiveness of the host and the VM. In some cases these additional latencies will cause glitches of various kinds.
The Raven
Posts: 82
Joined: 18. Aug 2016, 20:43

Re: Number of Guest CPU's recommendation

Post by The Raven »

Thank you for your reply mpack.
However that means that the host may not be able to service events in a timely fashion.

For the reasons I quoted in my original post, I felt that ultimately the Host is always in control
and only dispatches guest vcpus when ready. The interrupts within VT-X and/or Host timer pops will
will always ensure the Host (VirtualBox) will regain control and thus be able to do its required work.
bear in mind that the host OS is ALWAYS executing code, unless its asleep. If the clock is running then it must execute code

Why must it execute code ? That would imply that it doesn't use the concept of a "no work wait" and constantly spins on loop,
even when truly idle ? I know that in reality the Host O/S is always doing something as it's managing the
whole system but you get my drift.

I'm not doubting that you have to be careful with virtual to real cpu allocation but in all my experience
of (mainframe) Hypervisors, the problems are experienced in the Guest, due to non-dispatch of vcpus,
not in the Host.
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Number of Guest CPU's recommendation

Post by scottgus1 »

Does seem to be the day for discussing this question.

In my experience, sans techincal details concerning how the data gets run through the CPU*, I have run more virtual CPUs than there were cores on the host:
Two two-cpu guests and a one-cpu guest on a 4-core I7 with hyperthreading on. The going advice on the forum is that hyperthreading doesn't count so the I7 only has 4 cores, even though Windows says there's 8.

So I had 5 cores for the guests simultaneiously on a four-core host. Ran like a top. No problems. Why? Because the guests weren't all number crunching full bore all at once. Every so often a process would raise CPU on a guest to 50%, or a multi-threaded process would go to 100%, or two processes go to 50% each, totalling 100%. But it wan't all guests and the host all going to 100% all at once, so I was OK.

Being conservative or liberal with the processor settings comes down to what you intend to do with the guests. Stick to the conservative if your guests are running super number crunching. Running multiple desktops office users will be emailing and word processiing on may allow more guest CPUs than there are in the host. Just be aware that if they all go full throttle down the trench to keep the fighters off their tail, something somewhere will start to suffer.

* one thought I remember from the manual leads me to believe that the number of cores set in a guest aren't taken away from the host, as if the host could no longer use the physical cores while the guest is using them. I forget what it's called, but there's a process where if the guest calls for a CPU instruction that the host CPU doesn't have Virtualbox will exchange that instruction for something the host can do. Sounds to me like the host is always in control and able to route instructions as needed as long as the host doesn't get swamped.
The Raven
Posts: 82
Joined: 18. Aug 2016, 20:43

Re: Number of Guest CPU's recommendation

Post by The Raven »

My question was really prompted after I read the VT-X documentation from Intel.
I am most familiar with a virtualisation mechanism called Start Interpretive Execution (SIE)
on IBM mainframes which was first introduced in 1981. This was used by both their bare-metal hypervisor
and subsequently by their hardware partitioning (PR/SM or LPAR).
When I started reading the VT-X documentation I could see that it worked in a very similar manner
manner to SIE.

So I could see that a VM (whether bare metal or hosted) can dispatch guests under the control
of VT-X. At that point the real CPU is running the Guest instructions. The real cpu will
typically exit this mode when an instruction is about to executed that needs Host resolution,
an I/O interrupt is encountered or the VT-X pre-emption timer expires. So, the Host is always able to
regain control.
Lets take a simple example, of 2-way CPU machine with two guests each with one vcpu.
The first guest becomes ready to run and the Hypervisor dispatches it under VT-X. So we have one real
processor busy and one idle (well the Host dispatcher task is running on it looking for work). Before the
first real cpu relinquishes control the other guest is ready to run and is dispatched under VT-X.
We now have both real processors running guest instruction streams under VT-X control. Is that a
problem ? it certainly shouldn't be. For all the reasons previously mentioned, control will always
come back to the Host. Say, in an extreme case, both guests instruction streams started looping ?
Either the VT-X pre-emption timer would expire or as soon as you caused an I/O interrupt (move the
mouse perhaps), one of real cpus would leave VT-X mode and the interrupt would be handled,
thus ultimately passing control back to the host.

I'm not surprised by the experience of scottgus1, that's what I would expect. In fact, in theory, you
should be able to heavily load all the active guests. The guest performance will degrade but the Host
is always the highest priority work in the system.

It was also stated that :
If the host has to share a CPU with the VM then this must degrade the responsiveness of the host and the VM

Let's forget about Hypervisors for a moment and take a normal OS running on a uni-processor.
Once the Host dispatcher has passed to control to your "hello world" program, the Host has
(albeit temporarily) lost control of the real cpu, it's executing the user program code. How else would you test your
wonderful new code otherwise ? However, the one real CPU is obviously being "shared" between
the Host and your program and there's no problem there is there ? The Host will only dispatch user
programs (tasks) when it's ready to do so and by implication has no other higher priority system work
to run. In addition, the Host regains control either through an I/O interrupt or the dispatcher time-slice
expiring.
With that in mind, if we replace user programs (tasks) with virtual cpus, then the same argument applies.
There is no problem for the Host in "sharing" the real CPU because it only passes control to the
user task (in this case a virtual cpu) when it's ready to do so.

The whole notion of an event-driven, time slicing, dispatching scheduling mechanism within an OS is premised
on the fact that it constantly loses and regains control of the real processor.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Number of Guest CPU's recommendation

Post by mpack »

There is a difference between normal time slicing and the big and near total context switches needed for a VM. It's my understanding that one of the things VT-x does for you is make those big context switches work much more efficiently. But, VT-x can only help you switch between actual cores, not threads, with the added complication that the host OS doesn't know anything about it because it doesn't manage the VT-x resource. And while the transition is more efficient, it is not entirely without cost, so a ring0 interrupt in the host OS is going to have extra latency.
The Raven
Posts: 82
Joined: 18. Aug 2016, 20:43

Re: Number of Guest CPU's recommendation

Post by The Raven »

VT-x can only help you switch between actual cores, not threads, with the added complication that the host OS doesn't know anything about it because it doesn't manage the VT-x resource


Any hardware based virtualisation solution (SIE or VT-X) doesn't switch between cores. It's a mode of operation that the processor is
placed into to run guests. There is no such thing as a VT-X "resource". Resources are things like processors, storage, disks etc.
It's the role of the hypervisor to manage those resources efficiently such that as much time as possible is spent executing guest instruction streams.

The question of interrupt latency for the host is somewhat moot. Certainly a hypervisor fielding an interrupt when no guests are
active will result in the fastest resolution of the interrupt. When that same processor is in VT-X mode and takes the interrupt,
then of course, you incur the cost of the VM exit processing before control is returned to host. However, that's what running
guests is all about. If you wanted a hypervisor to run at peak efficiency it would never run any guests at all :)
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Number of Guest CPU's recommendation

Post by mpack »

The Raven wrote:Resources are things like processors, storage, disks etc.
Sez who?

A resource is anything which can be consumed or borrowed from a finite supply.
The Raven
Posts: 82
Joined: 18. Aug 2016, 20:43

Re: Number of Guest CPU's recommendation

Post by The Raven »

Sez who?


Seriously ? Almost anyone with a basic grasp of how an OS works. Where do think the
term "System Resource Manager" came from ?

I'm not sure why you feel the need to quote the dictionary definition of the word "resource" ?
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Number of Guest CPU's recommendation

Post by mpack »

It was you who corrected my usage of the word "resource", so I assumed you had good justification. Apparently it was just BS.

I assure you that I have a good grasp of the English language and do not need you to invent jargon for me.

I have no idea where the term "System Resource Manager" came from. I wasn't even aware that it was a universal term.
The Raven
Posts: 82
Joined: 18. Aug 2016, 20:43

Re: Number of Guest CPU's recommendation

Post by The Raven »

With all due respect, I merely attempted to point out that VT-X was not a resource
in the way that hardware items like processors and storage are.
If I did not make that clear, then I apologise.
Apparently it was just BS.

It's a pity that you feel the need to use such phrases but still, that's up to you.

I opened this thread in all good faith in order to try and get a sensible, technical
discussion. Unfortunately that now seems at an end.
Ramshankar
Oracle Corporation
Posts: 793
Joined: 7. Jan 2008, 16:17

Re: Number of Guest CPU's recommendation

Post by Ramshankar »

Forewarning: I did not read all the posts in this thread but I will simply attempt to answer this question:

Why is it a bad idea to allocate as many VCPUs as there are physical CPUs?

You cannot have the best of both worlds. Most modern Intel CPUs have VT-x preemption timers, which VirtualBox has been using for years now. This lets us run chunks of guest code and still get interrupted to run host code depending on how we program the preemption timer. However, the question is not whether we can or cannot interrupt guest code, we normally can. The problem is that there are tasks that require to be run in reasonable frequency & amount of time both on the host *and* the guest. If you starve the host or guest of interrupts or introduce latency because there simply isn't enough compute power available, you will be creating bottlenecks.

Getting in and out of executing guest code via VT-x is still quite an expensive operation. We call it a world-switch or world round-trip, (i.e. VM-entry, execute guest, VM-exit). This is done in ring-0 (kernel) on the host, sometimes (especially on Windows hosts) we are forced to return all the way to ring-3 sometimes in order to satisfy DPC (Deferred Procedure Call) latency. Overall, you're going to have strange latencies introduced in unexpected places if you "overcommit". It is totally possible to run 4 VCPU VM on a 4 CPU host (I do it on my own my Linux dev box sometimes) but it is not something you should be doing if you care about reasonable performance; in extreme cases of overcommitment you may encounter program misbehavior (like when disk requests times out) which the programs are never designed to handle. In the not so severe case you may end up with some strange timeouts but not fatal errors.

Hope that helps.
Oracle Corp.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Number of Guest CPU's recommendation

Post by socratis »

Ramshankar wrote:Hope that helps.
Indeed it does. Thank you for the thorough explanation. This is going on my Copy/Paste VirtualBox-answers clipboard.

+1 on mpack's comment as well. It would be good to have a core dev's perspective on the subject.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Locked