VirtualBox 6.0 and Hyper-V

[jruhe]

Again: Turning off hyper-v is no option for most of us.

jruhe

[socratis]

Again: Turning off hyper-v is no option for most of us.

Good. Then start complaining to Microsoft for being selfish. Let's see how far that gets you...

[Gerhart]

Hello!

I tried launch Virtual Box 6.0.14 on Windows 10 19H1 with October 2019 Updates and get error:

Call to WHvSetupPartition failed: ERROR_SUCCESS (Last=0xc000000d/87) (VERR_NEM_VM_CREATE_FAILED).
Код ошибки: E_FAIL (0x80004005)
Компонент: ConsoleWrap
Интерфейс: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}

I found post about g_pfnVidMessageSlotMap and tried to collect some additional information.
When

Code: Select all   Expand viewCollapse view

if (g_pfnVidMessageSlotMap(hPartitionDevice, &MappedMsgSlot, iCpu))

is executed, vid.dll!VidMessageSlotMap calls vid.sys!VidMessageSlotMap from kernel mode.
vid.sys!VidMessageSlotMap function has check

Code: Select all   Expand viewCollapse view

 mov     rcx, 200000000h
 test    [rdi+10h], rcx
 short loc_FFFFF80441B83F27

where rdi points to Prtn object. Prtn object in my environment contains:

Code: Select all   Expand viewCollapse view

WINDBG>dq @rdi
ffffc804`e99d2000  00000000`6e747250 00000000`00000000
ffffc804`e99d2010  00000002`00200244 00000000`00000000

It is means, that

Code: Select all   Expand viewCollapse view

test    [rdi+10h], rcx

check is true, then vid.sys!VidMessageSlotMap returns error 0C000000Dh.

If i change Prtn object windbg command ed @rdi+14 0, VirtualBox continues to run and goes to loop cycle in boot stage without error.

Loop thread stack:

Code: Select all   Expand viewCollapse view

        ffffd881`69a0bdc0 fffff804`3ec86ff4 0xfffff804`3e920003
        ffffd881`69a0bdc8 fffff804`3ec8701a nt!HvcallpExtendedFastHypercall+0x54
        ffffd881`69a0bdd8 fffff804`3ee007e7 nt!HvcallpExtendedFastHypercallWithOutput+0x1a
        ffffd881`69a0bdf0 fffff804`41bd9cfa nt!HvcallFastExtended+0x27
        ffffd881`69a0be30 fffff804`41bdb635 winhvr!WinHvpDispatchVp+0x92
        ffffd881`69a0bec0 fffff804`41bdb0ec winhvr!WinHvpVpDispatchLoop+0x225
        ffffd881`69a0bfa0 fffff804`41b56823 winhvr!WinHvRunVpDispatchLoop+0x4c
        ffffd881`69a0bfe0 fffff804`41b564a7 Vid!VidXSchedulerpVpRun+0x73
        ffffd881`69a0c010 fffff804`41b89036 Vid!VidXSchedulerVpRun+0x1f
        ffffd881`69a0c050 fffff804`41b531e9 Vid!VidVpRun+0x11a
        ffffd881`69a0c080 fffff804`41b53fa3 Vid!VidIoControlFastPartition+0x10f1
        ffffd881`69a0c410 fffff804`41b52a94 Vid!VidExoFastIoControlPartition+0x21f
        ffffd881`69a0c4f0 fffff804`418d2ba6 Vid!VidIoControlFast+0xb54
        ffffd881`69a0c5a0 fffff804`43e3d7c9 VBoxDrv+0x12ba6
        ffffd881`69a0c640 fffff804`43e3dcb2 VMMR0!REMFlushTBs+0x7749
        ffffd881`69a0c6e0 fffff804`43d77ce3 VMMR0!REMFlushTBs+0x7c32
        ffffd881`69a0c7a0 fffff804`418c1771 VMMR0!VMMR0EntryFast+0x193
        ffffd881`69a0c830 fffff804`418d4ec3 VBoxDrv+0x1771
        ffffd881`69a0c860 fffff804`3f0a4d0b VBoxDrv+0x14ec3
        ffffd881`69a0c900 fffff804`3f0a45d6 nt!IopXxxControlFile+0x71b
        ffffd881`69a0ca20 fffff804`3ec8eb18 nt!NtDeviceIoControlFile+0x56
        ffffd881`69a0ca90 00007ffc`c9b5c1a4 nt!KiSystemServiceCopyEnd+0x28 (TrapFrame @ ffffd881`69a0cb00)

I mean, no chance to get successful vid.sys!VidMessageSlotMap and g_pfnVidMessageSlotMap results, when Prtn object has 2 value in 0x14 offset.

[Andrew Brehm]

Well then...

I guess that's why no ones posting a success as it doesn't work without the "workaround" >:)

Let me do it: SUCCESS!
[...]
-> Hyper-V -> Hyper-V-Platform -> Hyper-V-Hypervisor "disabled"

What success? You disabled Hyper-V and VirtualBox works. I think the problem here was getting VirtualBox to run while Hyper-V is enabled.

[Gerhart]

VirtualBox 6.1 beta 2 works with Hyper-V. It doesn't use VidMessageSlotMap function by default.

[klaus]

Gerhart, sorry to destroy your illusion, but the NEM code in 6.0.14 and 6.1.0_BETA2 is for all practical purposes identical.

Also, VidMessageSlotMap would be called (except for probing the ioctl numbers, which is done with a dummy handle parameter) after the call to WHvSetupPartition which errors out so mysteriously (if the returned result code is actually ERROR_SUCCESS then it shouldn't error out, but the code should continue...).

We're talking about https://www.virtualbox.org/browser/vbox ... .cpp#L1386 which is quite straightforward. You need to scroll down a few screenfuls to get to the VidMessageSlotMap calls.

[Gerhart]

Klaus, no illusion )
I simply attach WinDBG to host computer, make breakpoints on Vid.sys!VidMessageSlotMap and Vid.sys!VidCreatePartition, and then start VM (Windows 10 19H1) in Virtual Box 6.1 Beta 2.
Vid.sys!VidCreatePartition was triggered, but Vid.sys!VidMessageSlotMap - no (i waited for guest OS login screen).

If you see 6.0.14 source code of NEMR3Native-win.cpp (https://download.virtualbox.org/virtual ... 14.tar.bz2):

Code: Select all   Expand viewCollapse view

#if defined(NEM_WIN_WITH_RING0_RUNLOOP) || defined(NEM_WIN_USE_OUR_OWN_RUN_API)
        {
            VID_MAPPED_MESSAGE_SLOT MappedMsgSlot = { NULL, UINT32_MAX, UINT32_MAX };
            if (g_pfnVidMessageSlotMap(hPartitionDevice, &MappedMsgSlot, iCpu))
            {
                AssertLogRelMsg(MappedMsgSlot.iCpu == iCpu && MappedMsgSlot.uParentAdvisory == UINT32_MAX,
                                ("%#x %#x (iCpu=%#x)\n", MappedMsgSlot.iCpu, MappedMsgSlot.uParentAdvisory, iCpu));
                pVCpu->nem.s.pvMsgSlotMapping = MappedMsgSlot.pMsgBlock;
            }
            else
            {
                NTSTATUS const rcNtLast  = RTNtLastStatusValue();
                DWORD const    dwErrLast = RTNtLastErrorValue();
                return VMSetError(pVM, VERR_NEM_VM_CREATE_FAILED, RT_SRC_POS,
                                  "Call to WHvSetupPartition failed: %Rhrc (Last=%#x/%u)", hrc, rcNtLast, dwErrLast);
            }
        }
#endif

you can see error message: "Call to WHvSetupPartition failed" in function g_pfnVidMessageSlotMap handler. It was incorrect message and we see it in VBox log. It is haven't any mysterious things (btw, in Virtual Box 6.1 sources that message was fixed)
More then g_pfnVidMessageSlotMap has conditional compilation directives

Code: Select all   Expand viewCollapse view

#if defined(NEM_WIN_WITH_RING0_RUNLOOP) || defined(NEM_WIN_USE_OUR_OWN_RUN_API)

It is not so hard to change compilation options and exclude that part of code.

I compared WHvSetupPartition execution in Virtual Box 6.0.14 and Qemu with -accel whpx options and don't see any differences. In both cases function returned handler to partition object.
Error in Virtual Box 6.0.14 was generated not because of WHvSetupPartition results (if you were uploading pdb files for Vbox windows builds, then the answer would be more accurate).

But Virtual Box 6.1 Beta 2 is working with Hyper-V. It's good.

Screen.png (115.24 KiB) Viewed 628 times

[klaus]

Doh. You're right. The misleading/useless error messages (they look like copy/paste bugs to me) fooled me. And you seem to be right, 6.1.0_BETA2 brought a change due to tweaking the conditional compilation, but to me this change appears to be accidental (and will decrease the already bad performance a lot more). If it were intentional it should've been backported to 6.0.x, which didn't happen.

Either way, I already heard that even BETA2 won't work with the next Windows 10 version, at least Insider Previews appear to be not compatible with the NEM code in BETA2. So there's still some trouble with changes from Microsoft's side...

[Gerhart]

Yes, even VMware Workstation is not compatible with Hyper-V still...
But Microsoft employees think, that Windows Hypervisor Platform API is good, and lack of compatibility with Hyper-V primarily problem of virtualization software vendor https://twitter.com/aall86/status/1180753546804772864.
It would be great, if somebody from Virtual Box developer team will reports argued bottlenecks of Hypervisor Platform API to Microsoft (at least through twitter).

[Andrew Brehm]

So ultimately...

Am I correct in my understanding that VirtualBox can run 32 bit (x86) VMs while running under Hyper-V but not 64 bit (amd64) VMs?

I understand that VirtualBox always had that ability using binary translation akin to how VMware used to virtualise x86 before virtualisation extensions of the architecture allowed for hardware support for virtualisation. The change with 6.0 is then that VirtualBox can now use the Hyper-V hypervisor to run 32 bit (x86) VMs (instead of using binary translation) but still cannot run 64 bit (amd64) VMs.

The one screenshot I saw here in the forum suggests as much since the guest shown running in VirtualBox was apparently Windows 98 (a system as x86 and non-amd64 as possible) while anyone trying to start a 64 bit (amd64) VM in VirtualBox under Hyper-V saw an error message.

This would ultimately be good news, since 32 bit (x86) VMs are exactly the ones that Hyper-V does not support well (32 bit Windows) or at all (OS/2).

I will try this out tonight.

[klaus]

No, VirtualBox 6.0 running in a Windows 10 Hyper-V root partition can execute both 32-bit and 64-bit VMs, as long as you have a Windows 10 version which is compatible with VirtualBox's code supporting Hyper-V. Currently this is Windows 10 1803 and 1809.

To my knowledge some change present in Windows 10 1903 and later broke the code in VirtualBox again (and while 6.1.0_BETA2 is said to work, this is actually a bug since it disables the code which brings a tiny bit of performance).

If you talk about running VirtualBox inside a Hyper-V VM, then please state this clearly, as it's a completely different topic.

[Andrew Brehm]

No, VirtualBox 6.0 running in a Windows 10 Hyper-V root partition can execute both 32-bit and 64-bit VMs, as long as you have a Windows 10 version which is compatible with VirtualBox's code supporting Hyper-V. Currently this is Windows 10 1803 and 1809.

That explains it. I tried it with a current verson of Windows 10.

That means waiting for a new version of VirtualBox, I suppose.

[socratis]

That means waiting for a new version of VirtualBox, I suppose.

Or a new Win10 version, it's them that they broke it after all, VMware won't even manage to start...

[jmar83_the2nd]

That's very interesting - thank you for your informations!! :-)

[Andrew Brehm]

After fighting with Hyper-V's inability to clone VMs and its other erratic behaviour I have disabled it again and returned to using just VirtualBox.