VirtualBox 7.0.12 changing the VM state to 'GURU_MEDITATION' and stops

[bugdayci]

VirtualBox VM 7.0.12

After a couple of hours working, the machine gets into 'GURU_MEDITATION' state and then stops.

It was working real slow in VirtualBox 6. Hence updated to 7 and saw good performance increase, but having crashes after the update:

Can not put a link to the log file and the log file is 2.5 MB so removing some flooding stuff like "Write Guest Paging Access Handler" from the log file and attaching it like that.

Code: Select all

02:10:33.896167 !!R0-Assertion Failed!!
02:10:33.896167 Expression: RT_SUCCESS_NP(rc)
02:10:33.896167 Location  : D:\tinderboxa\win-7.0\src\VBox\VMM\VMMAll\PGMAllPool.cpp(2697) int __cdecl pgmPoolMonitorInsert(struct PGMPOOL *,struct PGMPOOLPAGE *)
02:10:33.896417 PGMHandlerPhysicalRegisterEx 000000035656d000 failed with -80
02:10:33.896437 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
02:10:33.896469 Console: Machine state changed to 'Stuck'
02:10:33.896917 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
02:10:33.896918 !!
02:10:33.896919 !!         VCPU1: Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)
02:10:33.896932 !!
02:10:33.896937 !!R0-Assertion Failed!!
02:10:33.896937 Expression: RT_SUCCESS_NP(rc)
02:10:33.896937 Location  : D:\tinderboxa\win-7.0\src\VBox\VMM\VMMAll\PGMAllPool.cpp(2697) int __cdecl pgmPoolMonitorInsert(struct PGMPOOL *,struct PGMPOOLPAGE *)
02:10:33.896938 PGMHandlerPhysicalRegisterEx 000000035656d000 failed with -80
02:10:33.896941 !! ACTIVE TRAP=0e ERRCD=6 CR2=00002717c4000038 PC=0000000000b01ad9 Type=0 cbInstr=ff fIcebp=false (Guest!)
02:10:33.896951 !!
02:10:33.896952 !! AssertJmpBuf:

[bugdayci]

Checked the source code and the error seems like "Out of Resources":

Code: Select all

 ./include/iprt/err.mac:
  133: %define VERR_OUT_OF_RESOURCES    (-80)

Since the method called is "PGMHandlerPhysicalRegisterEx", I believe this happens when VBox tries to get some more RAM and gives out of resources error.

Now our machine has 64GB of ram, where I reserve 16GB for the VM, so there seems to be an error on that part.

I checked some other posts, and set the RAM to be pre-allocated via the following extra data:

Code: Select all

VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1

I checked the config via "getextradata" and it was set and also I can see from the logs that it is set to 1.

The VM stopped again. with same error code:

Code: Select all

00:37:45.111735 
00:37:45.111739 !!R0-Assertion Failed!!
00:37:45.111739 Expression: RT_SUCCESS_NP(rc)
00:37:45.111740 Location  : D:\tinderboxa\win-7.0\src\VBox\VMM\VMMAll\PGMAllPool.cpp(2697) int __cdecl pgmPoolMonitorInsert(struct PGMPOOL *,struct PGMPOOLPAGE *)
00:37:45.112006 PGMHandlerPhysicalRegisterEx 000000036bc17000 failed with -80
00:37:45.112031 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
00:37:45.112059 Console: Machine state changed to 'Stuck'
00:37:45.112525 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
00:37:45.112526 !!
00:37:45.112526 !!         VCPU2: Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)
00:37:45.112538 !!
00:37:45.112542 !!R0-Assertion Failed!!
00:37:45.112542 Expression: RT_SUCCESS_NP(rc)
00:37:45.112542 Location  : D:\tinderboxa\win-7.0\src\VBox\VMM\VMMAll\PGMAllPool.cpp(2697) int __cdecl pgmPoolMonitorInsert(struct PGMPOOL *,struct PGMPOOLPAGE *)
00:37:45.112543 PGMHandlerPhysicalRegisterEx 000000036bc17000 failed with -80
00:37:45.112545 !! ACTIVE TRAP=0e ERRCD=4 CR2=00005576b92c063c PC=00005576b85aa44f Type=0 cbInstr=ff fIcebp=false (Guest!)
00:37:45.112554 !!
00:37:45.112554 !! AssertJmpBuf:
00:37:45.112555 !!

I updated the Guest Additions and also decreased the Ram size to 8GB (less error means less probability of happening?). Will monitor it with this configuration.

Any other ideas I can try?

[scottgus1]

This has come up before, recently. Unfortunately I can't remember what was causing it, but it may not have had to do with actual RAM. You could try web-searching:
VERR_VMM_RING0_ASSERTION site:forums.virtualbox.org
A post within maybe the last 6 months discussed it.

[bugdayci]

Thank you @scottgus1 for the reply.

I actually checked before posting it and in general the error with -1701, had been given:

Code: Select all

PGMHandlerPhysicalRegisterEx ..... failed with -1701

include/VBox/err.h:
#define VERR_MM_HYPER_NO_MEMORY                 (-1701)

Where as I am getting -80 error (VERR_OUT_OF_RESOURCES). That's the reason why I created this post.

1. When I checked the code there is a comment:

Code: Select all

static int pgmPoolMonitorInsert(PPGMPOOL pPool, PPGMPOOLPAGE pPage)
    int rc;
    PPGMPOOLPAGE pPageHead = pgmPoolMonitorGetPageByGCPhys(pPool, pPage);
    if (pPageHead)
    {
......
    }
    else
    {
        if (PGMPOOL_PAGE_IS_NESTED(pPage))
            Log7Func(("Started monitoring GCPhysPage=%RGp HCPhys=%RHp enmKind=%s\n", pPage->GCPhys, pPage->Core.Key, pgmPoolPoolKindToStr(pPage->enmKind)));

        Assert(pPage->iMonitoredNext == NIL_PGMPOOL_IDX); Assert(pPage->iMonitoredPrev == NIL_PGMPOOL_IDX);
        PVMCC pVM = pPool->CTX_SUFF(pVM);
        const RTGCPHYS GCPhysPage = pPage->GCPhys & ~(RTGCPHYS)PAGE_OFFSET_MASK;
        rc = PGMHandlerPhysicalRegister(pVM, GCPhysPage, GCPhysPage + PAGE_OFFSET_MASK, pPool->hAccessHandlerType,
                                        pPage - &pPool->aPages[0], NIL_RTR3PTR /*pszDesc*/);
        /** @todo we should probably deal with out-of-memory conditions here, but for now increasing
         * the heap size should suffice. */
        AssertFatalMsgRC(rc, ("PGMHandlerPhysicalRegisterEx %RGp failed with %Rrc\n", GCPhysPage, rc));
        PVMCPU pVCpu = VMMGetCpu(pVM);
        AssertFatalMsg(!(pVCpu->pgm.s.fSyncFlags & PGM_SYNC_CLEAR_PGM_POOL) || VMCPU_FF_IS_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3), ("fSyncFlags=%x syncff=%d\n", pVCpu->pgm.s.fSyncFlags, VMCPU_FF_IS_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3)));
    }

Here it is commented that:
"@todo we should probably deal with out-of-memory conditions here, but for now increasing the heap size should suffice."

Which process' heap size should I increase? It is not clear. Should I give an extra option to VBoxManage?

2. What are the effects of the following options?

VBoxManage modifyvm "tm1" --largepages off
VBoxManage setextradata VM_NAME "VBoxInternal/MM/CanUseLargerHeap" 1

Have seen these in the posts but was not sure if these are related to my problem.

Thanks

[fth0]

Since the method called is "PGMHandlerPhysicalRegisterEx", I believe this happens when VBox tries to get some more RAM and gives out of resources error.

In principle, yes, but we're on a deeper level here: The Page Manager (PGM) is responsible for shadowing the memory paging of the guest OS, and one of the internal structures was (unexpectedly) already full for some unknown reason. So the VERR_OUT_OF_RESOURCES does not directly relate to the RAM provided to the VM, but rather to some type of bookkeeping error (e.g. a race condition).

This type of issue has been reported by other users from time to time (*), but I don't know of any solution yet. Your statements about "After a couple of hours working", "working real slow in VirtualBox 6" and the "flooding" part could indicate that a lot of small memory allocations could play a role, so perhaps you could restart the guest OS after some runtime as a workaround (or as a test).

Other than that, I'd suggest to create a ticket in the Bugtracker. Note that there's already ticket #21805 for another variant of the Guru Meditation, but I don't know if it's really related or not.

(*) @scottgus1: Regarding the "Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)": I don't know for sure if the two "Locations" pgmPoolMonitorInsert and pgmPoolTracDerefGCPhys indicate the same type of issue or not, because I haven't seen both locations being reported by the same user yet. And #21805 links to (the 2nd page of) the forum thread you were probably thinking about.

[embedde_dude]

This made me laugh, I haven't seen a guru meditation message since the Commodore Amiga back in the late 80s.