Over Provisioning on Linux Hosts

Discussions related to using VirtualBox on Linux hosts.
Post Reply
AndrewPW
Posts: 10
Joined: 10. May 2015, 20:23

Over Provisioning on Linux Hosts

Post by AndrewPW »

Before I start digging too much I have a quick question about over provisioning

I have 8 vms (during a migration) on a single host with 4 cores (8 threads) and 16 GB of ram on the physical host.

Recently some or all the vm have been aborting with no clear errors in the logs, except
VMMDev: vmmDevHeartbeatFlatlinedTimer: Guest seems to be unresponsive. Last heartbeat received 4 seconds ago

Reading the forums it looks like over provisioning on virtual box is a bad idea, i.e if you have 4 cores , it's one for the host and then a max of 3 vms with one core each. There appears to be strong recommendation not to over provision allowing the vms to flex as required?

If this true? Is this a know issue wth VirtualBox , or should i be digging a lot more into he logs


Many Thnks
Andrew
erdeslawe
Volunteer
Posts: 241
Joined: 8. Jul 2015, 10:23

Re: Over Provisioning on Linux Hosts

Post by erdeslawe »

I wouldn't call it a "VirtualBox Issue" that a Computer needs at least 1 CPU and sufficient memory to function normally and each Virtual Machine is a single computer running on top of the Host Computer which itself needs to retain sufficient processing power and memory because, at the very least, is has the parent Operating System + the Virtualisation software to run. Hardware needs are also affected by the software and hardware to be run on both the Host and Guest(s), including the Operating System, e.g. Windows 10 really needs at least 2 Cores to function satisfactorily and if you then run multiple applications then ....
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Over Provisioning on Linux Hosts

Post by socratis »

It's called mathematics. Remember that simple rule:
  • Total CPUs for your guests + 1 Host CPUs.
  • Total RAM+vRAM+overhead for your guests Host available RAM + some.
Sure you can over-commit: For testing purposes. If you know what you're doing. If you know what your VMs are doing. But do NOT wonder if it crashes.
 Edit: Fixed the math of the "simple rule", it was "simply wrong". 
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
AndrewPW
Posts: 10
Joined: 10. May 2015, 20:23

Re: Over Provisioning on Linux Hosts

Post by AndrewPW »

It looks to me like virtual box has limitations as virtualisation was designed to run large numbers of VMs on single servers, not limit them to one per core. Any vm host which crashes when more CPU are allocated then physically available , instead of time slicing is not worth using, and this may be the root of my issues. The same would apply to memory (but in a slightly more limited way) , as long as the maximum available memory's not all used at the same time its ok. You just need to be careful of performance issue and crashes if the vm's are not balanced.

Any other opensource products recommenced.

I have extensive experience of VMware in corporate environments, but this is for home use.
erdeslawe
Volunteer
Posts: 241
Joined: 8. Jul 2015, 10:23

Re: Over Provisioning on Linux Hosts

Post by erdeslawe »

It's your hardware that defines the limitations of the capability to run multiple VMs effectively, rather than the virtualisation software. If you have a four core processor and 8Gb RAM that is one limitation, it you have an 8 core processor and 64Gb RAM that is a different limitation; and VirtualBox works with whatever limitations exist (as do the alternatives).

You can't deploy resources that you don't have and expect everything to behave flawlessly! Can a physical machine with a single core processor and 4Gb memory be expected to run any software that specifically requires a multi-core processor and 16Gb of memory to work?
Martin
Volunteer
Posts: 2560
Joined: 30. May 2007, 18:05
Primary OS: Fedora other
VBox Version: PUEL
Guest OSses: XP, Win7, Win10, Linux, OS/2

Re: Over Provisioning on Linux Hosts

Post by Martin »

Only available RAM limits you directly when having multiple active VMs "at the same time".
You just need to take into consideration that virtualization is essentially a time-sharing implementation for the usage of CPU and I/O activity.
Virtualbox just switches very fast between runnning the VMs. To do this effectively it should always have a physical CPU core available, so no single VM should have (more than) all physical cores assigend.
On a quad core system you could run for example 5-10 VMs with a single or two vCPU - if theses VMs don't create high CPU or I/O usage.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Over Provisioning on Linux Hosts

Post by mpack »

AndrewPW wrote:It looks to me like virtual box has limitations as virtualisation was designed to run large numbers of VMs on single servers
That use of VMs (run many virtual servers on a single physical server) was designed to make full (100%) use of available CPU and network capacity, on the assumption that most servers would spend all of their time doing nothing except waiting for data. That's EXCEEDINGLY FAR from saying that VMs in general are exempt from the laws of arithmetic - and I don't care what VM platform you pick, the laws of arithmetic do not change. None of them can let you use more than 100% of CPU.

So sure, if you had a scenario where your VMs are each expected to use ~0% CPU almost all of the time, then certainly you can have many of these running at the same time. I believe VirtualBox's limit is something like 128 VMs running at once.

As an aside, I'd also object to the term "over provisioning". It's not possible to "over provide" CPU. You allocate CPU.
AndrewPW
Posts: 10
Joined: 10. May 2015, 20:23

Re: Over Provisioning on Linux Hosts

Post by AndrewPW »

Martin wrote:Only available RAM limits you directly when having multiple active VMs "at the same time".
You just need to take into consideration that virtualization is essentially a time-sharing implementation for the usage of CPU and I/O activity.
Virtualbox just switches very fast between runnning the VMs. To do this effectively it should always have a physical CPU core available, so no single VM should have (more than) all physical cores assigend.
On a quad core system you could run for example 5-10 VMs with a single or two vCPU - if theses VMs don't create high CPU or I/O usage.
Agreed, i have a number of vms all which are less than the physical capacity and virtual box is becoming unstable. Think is time for KVM
AndrewPW
Posts: 10
Joined: 10. May 2015, 20:23

Re: Over Provisioning on Linux Hosts

Post by AndrewPW »

erdeslawe wrote:It's your hardware that defines the limitations of the capability to run multiple VMs effectively, rather than the virtualisation software. If you have a four core processor and 8Gb RAM that is one limitation, it you have an 8 core processor and 64Gb RAM that is a different limitation; and VirtualBox works with whatever limitations exist (as do the alternatives).

You can't deploy resources that you don't have and expect everything to behave flawlessly! Can a physical machine with a single core processor and 4Gb memory be expected to run any software that specifically requires a multi-core processor and 16Gb of memory to work?
sort of i should be able to deploy large numbers of vms which individually are less then the physical, however in totality are greater, and let the vm engine slice the cpu times. The effect of over provisioning similar loads should be poor performance not instability
Martin
Volunteer
Posts: 2560
Joined: 30. May 2007, 18:05
Primary OS: Fedora other
VBox Version: PUEL
Guest OSses: XP, Win7, Win10, Linux, OS/2

Re: Over Provisioning on Linux Hosts

Post by Martin »

I've only seen instability (of guests, not the host) when there was not enough RAM or too heavy disk i/o.
But every environment is different.
If you have a repeatable problem scenario we could maybe give you some hints after looking at your log files and system configuration.
AndrewPW
Posts: 10
Joined: 10. May 2015, 20:23

Re: Over Provisioning on Linux Hosts

Post by AndrewPW »

Thanks Martin

I've switched to KVM, and no stability issues at all even when under heavy load (some expected performance issues given the load), with the same provisioning. It was the guests which were not stable the host was perfect. Looks like its good bye for Virtualbox for me.
Last edited by socratis on 25. Feb 2018, 19:59, edited 1 time in total.
Reason: Removed unnecessary verbatim quote of the whole previous message.
Post Reply