Experience running VirtualBox inside KVM based guest?

[t11n]

Hi there,

I am in the situation to evaluate virtual box running inside a KVM Debian-Guest, that in turn runs on a Debian-Hypervisor.

The Debian-Hypervisor is one of my company's big machines, that runs several unrelated hosts. The Guest is a Debian system, that acts as the Jenkins host and VirtualBox is eventually used to create the build- and testing-environment

This setup is generally working, but we hit performance issues when running unit tests. One test suite completing in 20 minutes within VirtualBox on my laptop, takes hours to complete within the jenkins setup. Well, actually it does not fully complete there, but test-php processes are going wild after some hours (supposingly due to performance issues, since the test suites run fine on all local development machines running the same VM as on the build server).

The Jenkins Host, which is the KVM guest has all virtualisation CPU flags passed through (on an Intel-Host i.e. vmx, ept, vpid) and even /sys/module/kvm_intel/parameters/nested states nested paging is enabled with an 'Y'. For the testing phase, we've got 8 cores and plenty of RAM. CPU and RAM do not hit any limits.

Long story short: The setup is quite untypical and I am looking for reasons for our operations department to move the Jenkins host to a bare metal machine, where it should be easily able to run the virtual box setup.

So question is: Does anyone have experience running virtual box inside a KVM- or any other Hypervisor based guest? What were your performance results like? Can you please tell me why you use or do not use an approach like this? Does it make any sense at all to run a build server in a virtualised environment? If not, what about setting up Jenkins in the cloud? After all it is virtualised there, too.

Looking forward to your opinions.

Thanks!

[socratis]

I would try to avoid nested virtualization. It's not supported AFAIK, either if VirtualBox is at the 1st or the 2nd level, i.e. if it runs on bare metal and you virtualize a guest inside a guest using VirtualBox (in both guest and guest-guest) or you're running VirtualBox inside a guest on another hypervisor...

Do you see the performance issues if you run VirtualBox on bare metal?

[t11n]

I would try to avoid nested virtualization. It's not supported AFAIK, either if VirtualBox is at the 1st or the 2nd level, i.e. if it runs on bare metal and you virtualize a guest inside a guest using VirtualBox (in both guest and guest-guest) or you're running VirtualBox inside a guest on another hypervisor...

Yes, we'll try to avoid it. Initially we thought it is a good idea, since we have a big server virtualised, that could have hosted jenkins, too. The only thing I'm still wondering about is the cloud. In the cloud it seems to be possible to run virtual boxes. It is common practice and cloud servers are all virtualised themselves. Maybe this is because they use Type 1 Hypervisors?

Do you see the performance issues if you run VirtualBox on bare metal?

No, unit tests execution time fairly compares to bare metal. I do not have any problems running the virtual box VM on my laptop.

[michaln]

This setup is generally working, but we hit performance issues when running unit tests.

Welcome to nested virtualization!

The performance of a nested VM will be worse than that of a non-nested VM, that is a given. The question is if it's going to be more like 95% or closer to 5% of non-nested performance. That depends mostly on the hardware used and on the top-level hypervisor (how well it can take advantage of the hardware features). Without knowing the exact CPU model of the system running the nested VM, it's hard to say more. Generally things which are slow in virtualization are going to be even slower in nested virtualization, possibly an order of magnitude slower.

[t11n]

I would say it is ~60% of the performance compared to the non-nested virtualisation. Anyway, I am and always was convinced we need a bare metal machine to run the build server on. We just gave it a try with KVM. Well, does not seem to work.

The only question still rising is how do people use jenkins with vagrant and virtual box in the cloud? Cloud servers are always virtualised, so we have nested virtualisation there, too. That seems to work just fine. Does it make such a big difference for nested virtualisation if the first level host is a type 1 or type 2 hypervisor?

[socratis]

how do people use jenkins with vagrant and virtual box in the cloud?

You could/should ask the Jenkins or the Vagrant people. Most people around here use neither of these two. You might have better luck in any of those two other forums.