Experience running VirtualBox inside KVM based guest?

Discussions related to using VirtualBox on other (unsupported) hosts like FreeBSD.
Post Reply
t11n
Posts: 3
Joined: 7. Feb 2018, 14:52

Experience running VirtualBox inside KVM based guest?

Post by t11n »

Hi there,

I am in the situation to evaluate virtual box running inside a KVM Debian-Guest, that in turn runs on a Debian-Hypervisor.

The Debian-Hypervisor is one of my company's big machines, that runs several unrelated hosts. The Guest is a Debian system, that acts as the Jenkins host and VirtualBox is eventually used to create the build- and testing-environment :shock:

This setup is generally working, but we hit performance issues when running unit tests. One test suite completing in 20 minutes within VirtualBox on my laptop, takes hours to complete within the jenkins setup. Well, actually it does not fully complete there, but test-php processes are going wild after some hours (supposingly due to performance issues, since the test suites run fine on all local development machines running the same VM as on the build server).

The Jenkins Host, which is the KVM guest has all virtualisation CPU flags passed through (on an Intel-Host i.e. vmx, ept, vpid) and even /sys/module/kvm_intel/parameters/nested states nested paging is enabled with an 'Y'. For the testing phase, we've got 8 cores and plenty of RAM. CPU and RAM do not hit any limits.

Long story short: The setup is quite untypical and I am looking for reasons for our operations department to move the Jenkins host to a bare metal machine, where it should be easily able to run the virtual box setup.

So question is: Does anyone have experience running virtual box inside a KVM- or any other Hypervisor based guest? What were your performance results like? Can you please tell me why you use or do not use an approach like this? Does it make any sense at all to run a build server in a virtualised environment? If not, what about setting up Jenkins in the cloud? After all it is virtualised there, too.

Looking forward to your opinions.

Thanks!
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Experience running VirtualBox inside KVM based guest?

Post by socratis »

I would try to avoid nested virtualization. It's not supported AFAIK, either if VirtualBox is at the 1st or the 2nd level, i.e. if it runs on bare metal and you virtualize a guest inside a guest using VirtualBox (in both guest and guest-guest) or you're running VirtualBox inside a guest on another hypervisor...

Do you see the performance issues if you run VirtualBox on bare metal?
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
t11n
Posts: 3
Joined: 7. Feb 2018, 14:52

Re: Experience running VirtualBox inside KVM based guest?

Post by t11n »

socratis wrote:I would try to avoid nested virtualization. It's not supported AFAIK, either if VirtualBox is at the 1st or the 2nd level, i.e. if it runs on bare metal and you virtualize a guest inside a guest using VirtualBox (in both guest and guest-guest) or you're running VirtualBox inside a guest on another hypervisor...
Yes, we'll try to avoid it. Initially we thought it is a good idea, since we have a big server virtualised, that could have hosted jenkins, too. The only thing I'm still wondering about is the cloud. In the cloud it seems to be possible to run virtual boxes. It is common practice and cloud servers are all virtualised themselves. Maybe this is because they use Type 1 Hypervisors?
socratis wrote:Do you see the performance issues if you run VirtualBox on bare metal?
No, unit tests execution time fairly compares to bare metal. I do not have any problems running the virtual box VM on my laptop.
michaln
Oracle Corporation
Posts: 2973
Joined: 19. Dec 2007, 15:45
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Any and all
Contact:

Re: Experience running VirtualBox inside KVM based guest?

Post by michaln »

t11n wrote:This setup is generally working, but we hit performance issues when running unit tests.
Welcome to nested virtualization!

The performance of a nested VM will be worse than that of a non-nested VM, that is a given. The question is if it's going to be more like 95% or closer to 5% of non-nested performance. That depends mostly on the hardware used and on the top-level hypervisor (how well it can take advantage of the hardware features). Without knowing the exact CPU model of the system running the nested VM, it's hard to say more. Generally things which are slow in virtualization are going to be even slower in nested virtualization, possibly an order of magnitude slower.
t11n
Posts: 3
Joined: 7. Feb 2018, 14:52

Re: Experience running VirtualBox inside KVM based guest?

Post by t11n »

I would say it is ~60% of the performance compared to the non-nested virtualisation. Anyway, I am and always was convinced we need a bare metal machine to run the build server on. We just gave it a try with KVM. Well, does not seem to work.

The only question still rising is how do people use jenkins with vagrant and virtual box in the cloud? Cloud servers are always virtualised, so we have nested virtualisation there, too. That seems to work just fine. Does it make such a big difference for nested virtualisation if the first level host is a type 1 or type 2 hypervisor?
Last edited by socratis on 8. Feb 2018, 12:35, edited 1 time in total.
Reason: Removed unnecessary verbatim quote of the whole previous message.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Experience running VirtualBox inside KVM based guest?

Post by socratis »

t11n wrote:how do people use jenkins with vagrant and virtual box in the cloud?
You could/should ask the Jenkins or the Vagrant people. Most people around here use neither of these two. You might have better luck in any of those two other forums.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Post Reply