Nested virtualization, VT-x/AMD-V in the guest: Success stories (and failures)

[socratis]

Since 6.0.0 was released on 2018-12-18, a huge new feature became a reality; nested virtualization! Please share your successful (and the not so) stories...

Another important aspect is that virtualization is now available in the guest; AMD-V is enabled for the moment, with VT-x to follow.

From the 6.0.0 changelog:

This space is intentionally left blank.

And from the 6.0.0 User Manual:

Nested Virtualization

Oracle VM VirtualBox supports nested virtualization on host systems that run AMD CPUs. This feature enables the passthrough of hardware virtualization functions to the guest VM. That means that you can install a hypervisor, such as Oracle VM VirtualBox or KVM, on an Oracle VM VirtualBox guest. You can then create and run VMs within the guest VM.

You can enable the nested virtualization feature in one of the following ways:

• From the VirtualBox Manager, select the Enable Nested VT-x/AMD-V check box on the System tab. To disable the feature, deselect the check box.
• Use the --nested-hw-virt option of the VBoxManage modifyvm command to enable or disable nested virtualization. See chapter 8.8, VBoxManage modifyvm, page 133.

Initially this is for AMD CPUs only.
All Intel CPU posts *will* be deleted/split.

Sorry people, but you need to pay attention!

Edit: Minimum information required: Host kind, version and build. » VirtualBox version running on the host. » Guest kind, version and build. » VirtualBox version running on the guest. » Nested guest kind, version and build.

[Henk717]

As i posted in the 6.0 release post so far i have not had much luck with this feature when trying to use both Hyper-V and Virtualbox itself in a Windows Server 2016 guest.
I am making use of a Ryzen 1700X (AM4) system which is capable of doing nested virtualization and has successfully run nested VM's using other paid software.

Scenario:
For my upcoming microsoft exams i would like to emulate a Hyper-V failover cluster, to do this i need 2 nested VM's both running an instance of Hyper-V and a guest VM.

Result:
Hyper-V can not be installed due to a missing features: Second level address translation (SLAT).
This prevents me from further usage of Hyper-V inside the guest as it can not be installed using normal means.

To rule out its just Hyper-V i attempted to install Virtualbox 6 inside of the same VM (Which at this point did not have Hyper-V since the installation can not be executed) and once i run a 64-bit VM the guest OS crashes and starts consuming 100% of the CPU resources i allocated.

It seems that before this feature becomes usable for Windows Guests Second level address translation needs to be added to this feature.
I have yet to test with products like the Citrix Xenserver, Proxmox and VMware ESX.

[Yoda]

I tested this with my Ryzen 7 desktop. Running Fedora 29 as desktop,
with Fedora 29 in a Vbox guest, and running RHEL 8 beta in a guest
under that seems to work pretty good Mouse seemed to be not
working good - but that is most likely just problems with RHEL beta 8
itself (it looks like they removed the vbox additions from the kernel)..

More tests to come

[Henk717]

Proxmox 5.3-5 Linux Distro (8 cores, 8GB of ram) running WIndows 10 64-bit trough KVM.

Hardware used: Ryzen 1700X, MSI 370X Pro Carbon
Software used: Windows 10 build 1607, Virtualbox 6.0

Result: Virtual machine boots and runs well, although a lot slower then running it natively on Virtualbox.

[socratis]

although a lot slower then running it natively on Virtualbox

Do you have any numbers that go along with that "feeling"? In terms of disk/network I/O, CPU usage, task completion times, other?

[Ramshankar]

The reduced performance is to be expected. Nested paging for Nested virtualization is a feature that is not yet implemented.

Having an AMD-Ryzen CPU (or newer) is an advantage because VirtualBox supports Virtualized-VMSAVE/VMLOAD and Virtual-GIF (Global Interrupt Flag) features of these CPUs. This results in incurring fewer #VMEXITs, thus increasing performance by lowering the overhead while switching between executing the guest and nested-guest.

[socratis]

VirtualBox supports ... Virtual-GIF

I thought that this was patent encumbered, that's why Virtual-PNG was invented!

(sorry, couldn't resist)

[Henk717]

although a lot slower then running it natively on Virtualbox

Do you have any numbers that go along with that "feeling"? In terms of disk/network I/O, CPU usage, task completion times, other?

I no longer have the VM and currently don't have the time to rebuild and measure.
But think in spending a minute or two to get past the stage where the Windows PE ramdisk loads into ram rather then a few seconds.

The significant speed difference can be easily explained by the lack of nested paging since i was loading an entire bootable Windows PE shell into ram for quick testing which is a very ram intensive operation. I expect the performance to be roughly the same once this feature is added, and it will likely also solve the compatibility issue with Hyper-V guests.

[Tom B]

Unfortunately I could not get this working on a linux host.

CPU: AMD Threadripper 1950x
Host OS: Arch Linux
Guest OS: Windows 10 (64bit)

Windows 10 freezes at the loading screen when Nested Vritualization is enabled. I couldn't even get far enough to try running virtualbox inside virtualized windows 10.

[socratis]

It would be good if posters added a complete ZIPPED VBox.log from a startup / success|fail / shutdown sequence, mainly so that we see what a successful vs a failed log looks like...

[marius_1956]

Grat new!
Any plan to implement nested virtualization for Intel CPUs?
marius

[socratis]

@marius_1956
Yes, but no timeframe as of today...

[Friis]

I am getting an error saying:

The instruction at 0x0000000055F0E7D6 referrenced memory at
0x0000000000000008. The memory Could not be read.

unraid running windows 10 with virtualbox attempting to run windows 7 (all 64 bit)

[socratis]

@Friis
Did you read the 1st post about Intel vs. AMD CPUs?

We need to see a complete VBox.log, from a complete VM run:

• Start the VM from cold-boot (not from a paused or saved state) / Observe problem / Shutdown the VM (force close it if you have to).
• With the VM completely shut down (not paused or saved), right-click on the VM in the VirtualBox Manager and select "Show Log".
• Save only the first "VBox.log", ZIP it and attach it to your response. See the "Upload attachment" tab below the reply form.

[Friis]

I updated Virtualbox to 6.0.2 and felt like I didn't get as far as I used to. I made a log and downgraded to 6.0.0 to see if there was a difference. there was.
I have put logs for 6.0.2 and 6.0.0 in the zip.

I am using AMD Threadripper. in Unraid I have run activate nested virtualization script. Setting for the VM Hyper-V is set to Yes. (so settings in that end should be fine)