Host crash when running nested VM

[GLucas]

Here's the Guru Meditation:

00:07:16.169834 !!R0-Assertion Failed!!
00:07:16.169835 Expression: <NULL>
00:07:16.169835 Location : D:\tinderbox\win-7.0\src\VBox\VMM\VMMAll\PGMAllPool.cpp(5128) void __cdecl pgmPoolTrackDeref(struct PGMPOOL *,struct PGMPOOLPAGE *)
00:07:16.169862 enmKind=34 GCPhys=00000000acd5b000
00:07:16.169868 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
00:07:16.169881 Console: Machine state changed to 'Stuck'
00:07:16.170124 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
00:07:16.170125 !!
00:07:16.170125 !! VCPU2: Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)
00:07:16.170134 !!
00:07:16.170138 !!R0-Assertion Failed!!

Please take a look here for a similar situation, which unfortunately we never heard back on: viewtopic.php?t=103993

Hello!
So I've tried to disable lazy allocation of RAM and to disable large pages as suggested in the post you linked me and again, it didnt work.
I still have the same error but it's now VCPU1, no VCPU2 anymore.

Code: Select all

00:04:23.065306 !!R0-Assertion Failed!!
00:04:23.065307 Expression: <NULL>
00:04:23.065307 Location  : D:\tinderbox\win-7.0\src\VBox\VMM\VMMAll\PGMAllPool.cpp(5128) void __cdecl pgmPoolTrackDeref(struct PGMPOOL *,struct PGMPOOLPAGE *)
00:04:23.065330 enmKind=34 GCPhys=0000000055de0000
00:04:23.065335 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
00:04:23.065349 Console: Machine state changed to 'Stuck'
00:04:23.065555 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
00:04:23.065556 !!
00:04:23.065556 !!         VCPU1: Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)
00:04:23.065575 !!
00:04:23.065579 !!R0-Assertion Failed!!

[fth0]

Thanks for reporting back that disabling large pages doesn't help!

Regarding the vCPUs, does it always affect the same vCPU when you run the VM multiple times without changing the configuration?

Unfortunately, ATM I don't have any useful suggestion what to try next.

[GLucas]

Thanks for reporting back that disabling large pages doesn't help!

Regarding the vCPUs, does it always affect the same vCPU when you run the VM multiple times without changing the configuration?

Unfortunately, ATM I don't have any useful suggestion what to try next.

I didnt change anything regarding the configuration but in the log, as you can see, it first says about VCPU2 and in my last post, it was VCPU1.

If you have any ideas coming up, let me know

[fth0]

I didnt change anything regarding the configuration but in the log, as you can see, it first says about VCPU2 and in my last post, it was VCPU1.

Your previous post made me "see" that the affected vCPU changed when trying disabling large pages or lazy allocation. I didn't expect the issue being bound to a specific vCPU at all. Thanks for clarifying!

[dmigowski]

I have exactly the same problem!

I am on Windows 10, run a Debian 12 guest in Virtual Box, and use nested virtualization to create a Debian 11 KVM machine.

Here is the Guru Mediation:

Code: Select all

00:16:09.153058 
00:16:09.153058 !!R0-Assertion Failed!!
00:16:09.153059 Expression: <NULL>
00:16:09.153059 Location  : D:\tinderboxa\win-7.0\src\VBox\VMM\VMMAll\PGMAllPool.cpp(5128) void __cdecl pgmPoolTrackDeref(struct PGMPOOL *,struct PGMPOOLPAGE *)
00:16:09.153072 enmKind=34 GCPhys=0000000108107000
00:16:09.153075 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
00:16:09.153082 Console: Machine state changed to 'Stuck'
00:16:09.153199 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
00:16:09.153200 !!
00:16:09.153200 !!         VCPU6: Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)
00:16:09.153205 !!
00:16:09.153207 !!R0-Assertion Failed!!

Also the same problem with my CPU, or the VmcsShadowing, it seems, or is the (1) meaning it is actually enabled? Then why the 0?:

00:00:04.372613 Invpcid - Enable INVPCID = 1 (1)
00:00:04.372613 VmFuncs - Enable VM Functions = 0 (1)
00:00:04.372614 VmcsShadowing - VMCS shadowing = 0 (1)
00:00:04.372614 RdseedExiting - RDSEED exiting = 0 (1)
00:00:04.372614 PML - Page-Modification Log (PML) = 0 (0)

Attached are my logs in case that helps. Just tell, if I can help with more information, these crashs start to be a bit annoying.

[scottgus1]

The guru meditated over something apparently related to RAM, if I read correctly:

00:16:09.153059 Location : D:\tinderboxa\win-7.0\src\VBox\VMM\VMMAll\PGMAllPool.cpp(5128) void __cdecl pgmPoolTrackDeref(struct PGMPOOL *,struct PGMPOOLPAGE *)

The state of RAM at the start was:

00:00:03.880808 Host RAM: 32502MB (31.7GB) total, 14470MB (14.1GB) available
00:00:03.971657 RamSize <integer> = 0x0000000200000000 (8 589 934 592, 8.0 GiB)
00:00:03.971777 VRamSize <integer> = 0x0000000001100000 (17 825 792, 17.0 MiB)

Though you had 6 more GB than you needed at the start of the VM, it is possible that something else took up what was requested by the VM before Virtualbox allocated the RAM. Virtualbox uses "lazy" allocation by default, meaning the VM's RAM is only taken as the VM OS wants it, no all at once in the beginning. Linux takes its RAM slowly, Windows takes it all at once at the beginning.

Try rebooting the host then starting the VM first before starting other RAM-hungry apps.

You could also try turning off lazy allocation for this VM:

VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1

*****************************

VmcsShadowing, it seems, or is the (1) meaning it is actually enabled? Then why the 0?:

The 0 is the indicator. Your CPU doesn't have VmcsShadowing. There's no enabling it: the CPU has to come with it already working. The only way to enable it is to swap the CPU for one that has it already.

I don't know if lack of VmcsShadowing would guru-meditate the VM or will it just mean nested virtualization will be slow. But eaiter way, you won't like nested on this host.

[fth0]

00:00:04.372604 *********************** VT-x features ***********************
00:00:04.372605 Nested hardware virtualization - VMX features
00:00:04.372605   Mnemonic - Description                                  = guest (host)
00:00:04.372605   VMX - Virtual-Machine Extensions                        = 1 (1)
00:00:04.372611   Ept - Extended Page Tables                              = 1 (1)
00:00:04.372612   UnrestrictedGuest - Unrestricted guest                  = 1 (1)
00:00:04.372614   VmcsShadowing - VMCS shadowing                          = 0 (1)

Your host CPU supports VMCS Shadowing, so you're on the good side for nested virtualization. The (virtual) guest CPUs do not need it (unless you'd want efficient double nesting, which means a VM3 in a VM2 in a VM1 on a host H0). I don't have any good news regarding the crash, though.

The guru meditated over something apparently related to RAM, if I read correctly

In a wide general sense, yes, but we're on a deeper level here: The Page Manager (PGM) is responsible for shadowing the memory paging of the guest OS and the nested guest OS, and the internal structures got corrupted for some unknown reason.

[scottgus1]

Your host CPU supports VMCS Shadowing

Oops, looks like I've been reading that wrong! Sorry!

[dmigowski]

You could also try turning off lazy allocation for this VM:

VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1

I did, but still get the same crash. Just checked the log and it also tell me:

Code: Select all

00:00:04.111541 Extradata overrides:
00:00:04.111546   VBoxInternal/RamPreAlloc="1"

So I surely didn't forget to actually apply that setting.

[dmigowski]

HA! I can now reliably reproduce that bug. It occurs mainly when large amounts of files are written in the child vm, like during the installing of the guest OK (crashes 40% of the time), or when you do a copy of the whole filesystem like with this command, which has a 95% crash rate for me (it copied the whole root partition into a child partition on the same disk):

Code: Select all

(cd / && tar cf - --exclude='proc/*' --exclude='sys/*' --exclude='dev/*' --exclude='home/ikoffice' \
                  --exclude='mnt/*' --exclude='run/*' --exclude='data/*' --exclude='usr/local/*' . ) |\
(cd /mnt/rescue && tar xvfp -)

I could upload the Oracle Virtual Box somewhere and write a small documentation how to reproduce the bug. If any Oracle Gurus like to reproduce this I would be willing to do the work. It needs about 40 gigs disk space (14G 7zipped download).

Bonus points if you also have a Windows 10 installed on a 13th Gen Intel(R) Core(TM) i9-13900HX CPU.

But the other dude runs on 11th Gen Intel(R) Core(TM) i7-1165G7 CPU, so any late gen Intel should suffice to reproduce it.

[dmigowski]

WORKAROUND: I downgraded Virtual Box from 7.0.12 to 7.0.2, because Virtual Box 7.0.4 stated they have made some improvements with nested paging. Now the VM doesn't crash anymore.

[scottgus1]

I could ... write a small documentation how to reproduce the bug.

A Bugtracker ticket is the place to write the reproduction scenario, though the large upload may not be possible. Zipped logs from a successful run in 7.0.2 and a failing run in 7.0.12 would help. That you've pinned it to a change since 7.0.2 is good detective work, and can help the devs find the errors.

[dmigowski]

This bug is now ticket #21805 in the Bug Tracker: https://www.virtualbox.org/ticket/21805

[fth0]

Thanks for the tests and the ticket updates!

I downgraded Virtual Box from 7.0.12 to 7.0.2, because Virtual Box 7.0.4 stated they have made some improvements with nested paging.

From monitoring the VirtualBox source code changes, I know that nested virtualization and nested paging have been worked on between multiple VirtualBox 7.0.x release updates (which is not reflected in the Changlog). I don't know if it would help the VirtualBox devs, but could you also try (some of) the other release updates?

[dmigowski]

OK, I had time to check the other versions. It is exactly

7.0.4 <- HERE BE BUGS

that introduced the bug. Happy hunting! 7.0.2 works fine for me. 7.0.6 also crashes very soon. Didn't test 7.0.8, but 7.0.10 is OPs version, and 7.0.12 was my crashing version.