Nested virtualization trouble

Discussions related to using VirtualBox on Windows hosts.
Post Reply
cirrusblau
Posts: 4
Joined: 10. Jan 2022, 20:08

Nested virtualization trouble

Post by cirrusblau »

I've been running an Ubuntu vbox kvm vm on a physical windows system that works great.

Problem is, now I need to run another Ubuntu VM within that VM and it's not going nearly as well.

I've enabled nested virtualization on the windows host and grepping for vmx shows the CPU entries in the first Ubuntu VM, but performance of the second vm (Ubuntu VM is the host) is abysmal and makes me think it's not properly virtualized. It's so bad that while I can start to boot of the installation iso, it doesn't reach the installation prompts in a reasonable time.

What have I missed? Any ideas on how to troubleshoot?
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Nested virtualization trouble

Post by scottgus1 »

Post a vbox.log from the first VM on the Windows host. It will be able to check some needed capabilities.

Start the original Ubuntu VM from full normal shutdown, not save-state. Log into the Ubuntu OS, let it run for a minute, then shut down the VM.

Right-click the VM in the main Virtualbox window's VM list, choose Show Log. Save the far left tab's log, zip it, and post the zip file, using the forum's Upload Attachment tab.
cirrusblau
Posts: 4
Joined: 10. Jan 2022, 20:08

Re: Nested virtualization trouble

Post by cirrusblau »

Thanks for the help on this. Here are the logs

I shutdown cleanly, started the Ubuntu guest on the Windows host, logged in, let it sit approx 1 minute until it seemed all startup activity was done, then shutdown cleanly.

Is there anything in particular I should look for in the logs?
Attachments
VBox - Copy.zip
(44.99 KiB) Downloaded 6 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Nested virtualization trouble

Post by fth0 »

For a test, provide only 4 vCPUs and 16 GB RAM to the first level VM, and use the VMSVGA Graphics Controller. Does it make a difference?

Are you using VirtualBox for the second level VM? If yes, please provide VBox.log files for both VMs from the same run.
cirrusblau
Posts: 4
Joined: 10. Jan 2022, 20:08

Re: Nested virtualization trouble

Post by cirrusblau »

Here are the logs for both win bare metal running ubuntu and that ubuntu vm trying to start the nested ubuntu guest, also in vbox.

As requested, I ran only 4 vCPU and 16000 MB ram with the vmsvga controller.

There's no discernible change in performance of the vm or the nested vm; the nested vm trying to start from the iso to install is still very slow. I let the nested vm run for 3-5 minutes before killing it. The host vm was shutdown properly.

I did leave the nested vm boot going earlier this morning, it does eventually proceed to the next screen (that screen is not terribly responsive either) but the host CPU is pegged at 100% and it takes much longer than when doing the same on a vm on bare metal. It certainly seems that the nested virtualization isn't properly setup, but is there a way to see in the logs with certainty whether this is the case? I just want to be sure I haven't made an incorrect assumption here.
Attachments
win+linux.zip
(73.49 KiB) Downloaded 4 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Nested virtualization trouble

Post by fth0 »

I can see the slowness in the second level log file, but no immediate cause for it.

In the first level log file, I noticed that you're using a mixture of two VirtualBox Guest Additions (6.1.26_Ubuntu, 5.2.42_Ubuntu). I'd suggest to uninstall them and install the Oracle VirtualBox Guest Additions version 6.1.30 instead.
cirrusblau
Posts: 4
Joined: 10. Jan 2022, 20:08

Re: Nested virtualization trouble

Post by cirrusblau »

Thanks for catching that, I guess I didn't go through and update guest additions after last vbox update. It should be on the correct version now.

For what it's worth, while I've mostly reverted to my old settings (back to the old virtual graphics controller, approx 22 GB RAM but staying on 4 vCPU), the nested vm is still the same performance as previously. I don't think you were expecting any change but I figured I'd give it a try just in case.

What are you seeing in the logs that confirms the slowness? The logs are long and I'm not familiar with what to look for.

How can we confirm if the nested vm is properly getting access to the CPU? Are there any checks/diagnostics or other troubleshooting I can go through? Is there a way to force vbox to use the vCPU? Or confirm it is currently using it?

I'm not sure if the below is of any use but here it is just in case

Code: Select all

$ grep -c vmx /proc/cpuinfo 
4
$ lscpu | grep Virtualization
Virtualization:      VT-x
Virtualization type: full
$ grep -m1 vmx /proc/cpuinfo 
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti tpr_shadow flexpriority fsgsbase avx2 invpcid rdseed clflushopt md_clear flush_l1d
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Nested virtualization trouble

Post by fth0 »

cirrusblau wrote:For what it's worth, while I've mostly reverted to my old settings (back to the old virtual graphics controller, approx 22 GB RAM but staying on 4 vCPU), the nested vm is still the same performance as previously. I don't think you were expecting any change but I figured I'd give it a try just in case.
We rarely encounter a mixture of VirtualBox versions involving 6.1.x and 5.2.y, so there is not much practical knowledge of possible consequences, and it's not supported anyway. Therefore, it would be pointless to investigate further without correcting this first (as you've done).
cirrusblau wrote:What are you seeing in the logs that confirms the slowness?
The VirtualBox Guest Additions 6.0.0 kernel driver being started only after 3 minutes indicates a slow start of the second level guest. And the /TM/CPU statistics showing that more than 50% of cNsTotal is in the cNsOther category.
cirrusblau wrote:How can we confirm if the nested vm is properly getting access to the CPU?
The host CPU is the only host hardware that the guest has more or less direct access to, but the CPU capabilities are filtered. This can be seen in much detail in the CPUID dump section of the VBox.log file.
Post Reply