Page 1 of 1

Guru Meditation error

Posted: 26. Sep 2021, 02:16
by sakeebhoss
Running into this crash consistently, happened 3 times in a row. Not really sure how to proceed so I would really appreciate some advice!

Version of VirtualBox: 6.1.26 r145957 (Qt5.6.2)
Using Guest Additions: No
Host OS: Windows Server 2019, 64-bit, 1.5 TB RAM
Guest OS: Red Hat Enterprise Linux 8.4, 64-bit, 300 GB RAM

Description of what I was doing: I was running a web application server on the VM while running a Postman-like tool on the host machine, simply sending requests to the server in the VM.

Re: Guru Meditation error

Posted: 26. Sep 2021, 03:23
by scottgus1
Have to say, looks like an awesome server! 96 cores 'hyperthtreaded' (dual AMDs), 1.5TB RAM! Must have hit the war chest a little bit! :D

Despite having "only" :shock: 300 GB out of that 1.5TB RAM set aside for the RedHat VM, the guru meditated over insufficient RAM:
01:34:44.800442 PGMR3PhysAllocateLargePage: allocating large pages takes too long (last attempt 103 ms; nr of timeouts 11); DISABLE
02:27:05.319955
02:27:05.319957 !!R0-Assertion Failed!!
02:27:05.319963 Expression: cRefs == PGMPOOL_TD_CREFS_PHYSEXT
02:27:05.319964 Location : F:\tinderbox\win-6.1\src\VBox\VMM\VMMAll\PGMAllPool.cpp(4132) void __cdecl pgmPoolTrackPhysExtDerefGCPhys(struct PGMPOOL *,struct PGMPOOLPAGE *,union PGMPAGE *,unsigned short)
02:27:05.320021 cRefs=0 pPhysPage=A-:RAM:01bd88200000:001d000:-0000 pPage=ffffa76d9f21d9c0:{.idx=495}
02:27:05.320038 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
02:27:05.320069 Console: Machine state changed to 'GuruMeditation'
02:27:05.320399 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
02:27:05.320401 !!
02:27:05.320402 !! VCPU0: Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)
I surmise this is because Virtualbox has "lazy" RAM allocation for its VMs: it pulls RAM from the host as the VM calls for it. However, if the VM eventually calls for all of it, RAM has to be available for the VM. Since Virtualbox does not by default allocate all of the RAM at once, it is possible for other processes or VMs to take up RAM and leave the VM asking for RAM it expects but that the host cannot now allocate.

Adding to this possibility is that Linux does not touch all RAM at boot, so Linux VMs allow lazy allocation to work. Windows VMs touch all RAM at boot so all VM RAM gets allocated at boot or a RAM error occurs at boot, before you can get into usage of the VM.

A command can be used to turn off lazy allocation for a VM:

cd "C:\Program Files\Oracle\Virtualbox"
VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1


Run it for each VM that you want to avoid lazy allocation.

Re: Guru Meditation error

Posted: 26. Sep 2021, 10:47
by fth0
The following advice may seem counterintuitive at first, but please try if disabling large pages helps, by executing the following command while the VM is not running:

Code: Select all

cd C:\Program Files\Oracle\VirtualBox
VBoxManage modifyvm "tm1" --largepages off
PS: If following scottgus1's suggestion to disable the lazy RAM allocation made the crash appear faster, you can combine that with my suggestion above for a faster initial test.

Re: Guru Meditation error

Posted: 27. Feb 2024, 07:39
by maniacal mator
@fth0: That says, “Could not find a registered machine named “tm1”.

Re: Guru Meditation error

Posted: 27. Feb 2024, 19:24
by maniacal mator
I tried the tm1 command, but now I’m getting a verr_no_page_memory error. That command didn’t take off any of my memory from my virtual machine, did it? I am having this problem when I boot up two vms. Whichever one I boot up second returns the guru meditation error, and the most recent commands you and Scott shared only caused VirtualBox to say this: “No memory available for page table or page directory.” (VERR_NO_PAGE_MEMORY).

Re: Guru Meditation error

Posted: 27. Feb 2024, 23:46
by fth0
That's no surprise, since not everybody names their VM "tm1". ;)

Re: Guru Meditation error

Posted: 28. Feb 2024, 13:00
by Zdenek Kubenka
I have similar problem.
Host: 12cores / 24 threads. RAM: 65445MB (63.9GB) total, 38229MB (37.3GB) available
VMs:
a) 4 cores, 8GB RAM, Debian
b) 4 cores, 16GB RAM, Windows 10

Run of VM ends with Guru Meditation. Tried to
VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1
--> now ends with VERR_NO_PAGE_MEMORY

Also tried to set --largepages off --> did not help.

I have no idea why this keeps happening. After reboot, there is obviously no problem, however later when I shutdown/startup the VMs, this keeps happening. Since its a dev box, I need to cycle the various VMs as needed.

Attaching log and RAMMap screenshot.

Re: Guru Meditation error

Posted: 29. Feb 2024, 01:40
by fth0
You could try if disabling System > Processor > Nested VT-x/AMD-V makes a difference.

Regarding the RAMMap screenshot, did you take it while the Guru Meditation dialog was displayed, or only afterwards?

Re: Guru Meditation error

Posted: 1. Mar 2024, 10:17
by Zdenek Kubenka
As for the Rammap image - since i have "VBoxInternal/RamPreAlloc" set to 1, it tries to allocate full memory during the startup, so I dont get the Guru Mediation error during the VM run; so the rammap screenshot was taken after i tried to start the VM and it failed.


---
So, here i have tested it - disabled the Nested VT-x/AMD-V on the VMs, however it did not do much of a difference.

I was able to start the VM "Oracle Windows 10" (4 cores, 16 GB RAM); then i was able to start VM "Debian" (4 cores, 8 GB RAM); this all went fine. However when i tried to start VM "ORA_RAC_1", it aborted with VERR_NO_PAGE_MEMORY

And at the same time some of the apps started to behave erratically, for example the Chromium based browser froze.

I am uploading the logs for all 3 VMs; and also the jpegs with rammap and also the task manager with performance card opened.

Whats more interesting is the discrepancy of the free memory reported by the task manager and the rammap. I think this may be the real issue here - the VM manager thinks that there is still enough memory and tries to startup the VM; however in reality (as reported by rammap) there isnt any. The fact, that the other apps start behaving erratically may be due the fact the system is indeed out of free memory.

Any idea whats wrong here?

Re: Guru Meditation error

Posted: 6. Mar 2024, 22:53
by fth0
I'm not so sure if there really is an out-of-memory situation here (although VirtualBox claims that):

Like other type 2 hypervisors, VirtualBox uses Driver Locked memory for the VMs' RAM. The 1st and 2nd VM allocated 16 + 8 = 24 GB of RAM, and the 3rd VM crashed when trying to allocate their 16 GB of memory. VirtualBox tries to allocate this memory in chunks of 2 MB (large pages), and has only allocated a bit more than 1 GB (0x45fa1 4k pages), when apparently being refused the next 2 MB by the host OS.

I don't really understand why the Windows host couldn't fulfill the memory allocation request, since >15 GB were still used as disk cache (Standby column). I could imagine that the disk cache was fragmented and contained only 4k pages after some runtime, but then it should suffice to turn off large pages like you also tried (although not in the provided VBox.log file ;)).

From my POV, RAMMap and Task Manager show the same memory situation (compare Standby to Cached). BTW, VirtualBox doesn't care how much memory is available when it starts a VM, it just tries to get the memory page by page. This makes sense because VirtualBox cannot know how much memory the host OS is able to provide. Note that any decent OS always uses all available memory for caches, so there is never really free memory.