Nested virtualization/AMD-V in the guest: Success stories (and failures)

This is for discussing general topics about how to use VirtualBox.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by socratis »

Friis wrote:I updated Virtualbox to 6.0.2 and felt like I didn't get as far as I used to.
Not sure what I'm seeing here, and not sure why both the logs. You included the Win10 host logs, in which I see no errors...
Friis wrote:in Unraid I have run activate nested virtualization script
Not sure what that is as well...
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Friis
Posts: 6
Joined: 16. Jan 2019, 12:09

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by Friis »

socratis wrote:Not sure what I'm seeing here, and not sure why both the logs. You included the Win10 host logs, in which I see no errors...
Then I don't know what logs needs to be added. VBoxHardening.log? windows log?
socratis wrote:Not sure what that is as well...
Unraid is my host OS, Windows 10 is my VM, windows 7 is my nested VM.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by socratis »

@Friis
The way that I understand nested virtualization is VirtualBox-inside-VirtualBox. Is there a VirtualBox for Unraid OS? What's Unraid based on? And are you trying to run?
  • Unraid (version?) as host
    • » VirtualBox (version?)
      • » Win10 (flavor/version?) as guest
        • » VirtualBox (version?)
          • » Win7 (flavor/version?) as nested guest
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Friis
Posts: 6
Joined: 16. Jan 2019, 12:09

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by Friis »

Unraid, uses KVM virtualization.
  • Unraid (Version 6.6.6) host
    • Windows 10 (pro 1803) as guest
      • VirtualBox (6.0.0 - works best)
        • Windows 7 (Enterprise? I think, been some time since I have been in it) as nested guest
Unraid support GPU passthrough (Type 1 virtualization I think)

Virtualbox 6.0.0 I get to the point where windows asks me if I would like to start normally or run repair. after click run normally, the VM freezes. (6.0.2 I don't get that option)
Unraid has Hyper-V activated for Windows 10 VM, which also seem to be what is the best paravirtualization option in Virtualbox.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by socratis »

Pending an answer from the developers, I (personally) am going to be looking initially, only in the following scenarios:
socratis wrote:The way that I understand nested virtualization is VirtualBox-inside-VirtualBox.
I wouldn't know where/what to start looking for in a KVM log. :o
And I don't plan to either, because I can't run KVM...
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Tom B
Posts: 6
Joined: 23. Oct 2015, 14:36

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by Tom B »

Virtualbox: 6.0.2
Host: Arch Linux
Guest: Windows 10 with Nested virtualization enabled
CPU: AMD Threadripper 1950x

Crashes during windows boot.

Log file:
Attachments
Windows 10-2019-01-20-13-45-07.zip
(23.18 KiB) Downloaded 62 times
0xs4ni
Posts: 1
Joined: 20. Mar 2019, 23:03

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by 0xs4ni »

>Host: Ubuntu Bionic 18.04.2
>Host VBox: 6.0.4
-> Guest: Ubuntu Bionic 18.04.1
-> Guest Vbox: 6.0.4
--> Nested Guest: Boot2Docker 18.09.3

I was going through the Docker tutorial for swarms while following along in a VM then ran into an error code about the bios needing to support "VT-x/AMD-V" despite the box already being checked and found the Other Thread, realized my host vbox didn't have the update. After purging the previous installation and its related files and reinstalling+acquiring the expansion pack I was able to get it running as expected.

Not sure yet how Boot2Docker is configured by default but it took a sec to setup, these aren't things relating to virtualbox itself and more-so how the setup of a nested vm works through Docker that simply utilizes Virtualbox in the process. Happy to see this implemented, thank you for the hard work!

Initially I just removed the previous installation, but upon receiving an error code relating to drivers I circled back with

$ apt-get purge virtualbox-5.2
$ dpkg -P virtualbox-5.2
$ apt-get autoremove
$ reboot

then install the Expansion Pack, no more errors, nested virtualization achieved.

The host is using an AMD A12 processor.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by socratis »

0xs4ni wrote:then install the Expansion Pack, no more errors, nested virtualization achieved.
Thanks for sharing your success story! 8)

Would you mind doing a couple of things?
  1. Start the 1st VM,
  2. start the 2nd VM,
  3. shutdown the 2nd VM,
  4. grab the VBox.log of the 2nd VM,
  5. shutdown the 1st VM,
  6. grab the VBox.log of the 1st VM,
  7. ZIP both logs and attach them here
I'd like to see how a log within a log looks like... ;)
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
NeilZhang
Posts: 1
Joined: 1. Jul 2019, 08:47

Re: Nested virtualization, VT-x/AMD-V in the guest: Success stories (and failures)

Post by NeilZhang »

First thanks a lot for such wonderful feature!
My issue is about ESXi 6.7.
I installed ESXi 6.7 on VirtualBox 6.0.8 (amd64), then created a 64bit Linux guest on the ESXi.
I could created it without any issue so I thought the Nested virtualization did work and I also confirmed this point through below ESXi command.
[root@vCenter:~] esxcfg-info -w | grep "HV Support"
      |----HV Support...............................................3
While when I tried to start the guest I would get error 'This host does not support "AMD RVI" hardware assisted MMU virtualization' and failed.

I did similar things with VMware workstation and ESXi then also created a 64bit LInux guest on the ESXi, and it worked well.

So it seems VirtualBox could not expose AMD-RVI to the ESXi guest, right?
Last edited by socratis on 2. Jul 2019, 06:23, edited 1 time in total.
Reason: Enclosed the information in [quote][pre] tags for better readability
Bidyut
Posts: 10
Joined: 19. Aug 2018, 15:52

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by Bidyut »

As stated above, "virtual box supports nested virtualization on host systems with AMD CPUs". Could you please help me understand the below scenarios with regards to the same ?
  1. VMWare ESXi installed on bare metal hardware (host machine) with Intel CPUs. Multiple VMs with windows OS are created. Now virtual box is installed inside ESXi VM and then VM with windows OS is created inside virtual box. Therefore Virtual Box VM is nested within ESXi VM. Is this supported with latest version of virtual box specifically when using VBoxHeadless ? If not supported today, is there any future plan to support this configuration.
  2. The nested virtualization described in point (1) was working in the older version of virtual box when we were using 5.1.30. With latest version 6.0.4, we noticed that nested virtualization is disabled. Is there any way we could enable it in latest version or it may not work due to CPU limitation mentioned above ? Is there any option for an extended technical support ? Please let me know.
  3. In addition, we are looking for solution for this case too. VMWare Workstation Player is installed on host machine with Intel CPU. Virtual Box VM is present inside VMWare VM.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by socratis »

@'Bidyut'
If VMware can pass to its guests the necessary CPU properties (like VT-x), then you could potentially run VirtualBox inside an ESXi VM. But as you can tell from the previous sentence, the burden is on VMware. And we don't really use VMware around here... ;)
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Leowu
Posts: 2
Joined: 4. Dec 2019, 13:17

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by Leowu »

Someone uses AMD from PowerColor?
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by socratis »

According to Wikipedia, and their website, PowerColor makes GPUs, not CPUs. The question isn't relevant I'm afraid, we're talking about AMD CPUs here...
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Bidyut
Posts: 10
Joined: 19. Aug 2018, 15:52

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by Bidyut »

I was wondering how virtual box VM inside ESXi VM (ESXi v6.5) was working fine when virtual box version was 5.1.30. But with the latest version 6.0.4 of virtual box and without any changes in ESXi VM, why now virtual box VM is not running / working ?
multiOS
Volunteer
Posts: 797
Joined: 14. Sep 2019, 16:51
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: WIN11,10, 7, Linux (various)
Location: United Kingdom

Re: Nested virtualization/AMD-V in the guest: Success stories (and failures)

Post by multiOS »

@Bidyut,

Wouldn't your question be best posted to the the developers of ESXi (VMWare)? They're the one's who have ultimate control of what will/won't work on their platform and they are also the one's who need to provide information to other developers if any changes are required.
Locked