Guru Meditation error

Discussions related to using VirtualBox on Windows hosts.
Post Reply
sakeebhoss
Posts: 1
Joined: 26. Sep 2021, 01:30

Guru Meditation error

Post by sakeebhoss »

Running into this crash consistently, happened 3 times in a row. Not really sure how to proceed so I would really appreciate some advice!

Version of VirtualBox: 6.1.26 r145957 (Qt5.6.2)
Using Guest Additions: No
Host OS: Windows Server 2019, 64-bit, 1.5 TB RAM
Guest OS: Red Hat Enterprise Linux 8.4, 64-bit, 300 GB RAM

Description of what I was doing: I was running a web application server on the VM while running a Postman-like tool on the host machine, simply sending requests to the server in the VM.
Attachments
VBox.zip
(166.1 KiB) Downloaded 22 times
VBoxHardening.zip
(23.4 KiB) Downloaded 17 times
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Guru Meditation error

Post by scottgus1 »

Have to say, looks like an awesome server! 96 cores 'hyperthtreaded' (dual AMDs), 1.5TB RAM! Must have hit the war chest a little bit! :D

Despite having "only" :shock: 300 GB out of that 1.5TB RAM set aside for the RedHat VM, the guru meditated over insufficient RAM:
01:34:44.800442 PGMR3PhysAllocateLargePage: allocating large pages takes too long (last attempt 103 ms; nr of timeouts 11); DISABLE
02:27:05.319955
02:27:05.319957 !!R0-Assertion Failed!!
02:27:05.319963 Expression: cRefs == PGMPOOL_TD_CREFS_PHYSEXT
02:27:05.319964 Location : F:\tinderbox\win-6.1\src\VBox\VMM\VMMAll\PGMAllPool.cpp(4132) void __cdecl pgmPoolTrackPhysExtDerefGCPhys(struct PGMPOOL *,struct PGMPOOLPAGE *,union PGMPAGE *,unsigned short)
02:27:05.320021 cRefs=0 pPhysPage=A-:RAM:01bd88200000:001d000:-0000 pPage=ffffa76d9f21d9c0:{.idx=495}
02:27:05.320038 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
02:27:05.320069 Console: Machine state changed to 'GuruMeditation'
02:27:05.320399 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
02:27:05.320401 !!
02:27:05.320402 !! VCPU0: Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)
I surmise this is because Virtualbox has "lazy" RAM allocation for its VMs: it pulls RAM from the host as the VM calls for it. However, if the VM eventually calls for all of it, RAM has to be available for the VM. Since Virtualbox does not by default allocate all of the RAM at once, it is possible for other processes or VMs to take up RAM and leave the VM asking for RAM it expects but that the host cannot now allocate.

Adding to this possibility is that Linux does not touch all RAM at boot, so Linux VMs allow lazy allocation to work. Windows VMs touch all RAM at boot so all VM RAM gets allocated at boot or a RAM error occurs at boot, before you can get into usage of the VM.

A command can be used to turn off lazy allocation for a VM:

cd "C:\Program Files\Oracle\Virtualbox"
VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1


Run it for each VM that you want to avoid lazy allocation.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Guru Meditation error

Post by fth0 »

The following advice may seem counterintuitive at first, but please try if disabling large pages helps, by executing the following command while the VM is not running:

Code: Select all

cd C:\Program Files\Oracle\VirtualBox
VBoxManage modifyvm "tm1" --largepages off
PS: If following scottgus1's suggestion to disable the lazy RAM allocation made the crash appear faster, you can combine that with my suggestion above for a faster initial test.
maniacal mator
Posts: 2
Joined: 27. Feb 2024, 06:50

Re: Guru Meditation error

Post by maniacal mator »

@fth0: That says, “Could not find a registered machine named “tm1”.
Last edited by maniacal mator on 27. Feb 2024, 07:41, edited 1 time in total.
maniacal mator
Posts: 2
Joined: 27. Feb 2024, 06:50

Re: Guru Meditation error

Post by maniacal mator »

I tried the tm1 command, but now I’m getting a verr_no_page_memory error. That command didn’t take off any of my memory from my virtual machine, did it? I am having this problem when I boot up two vms. Whichever one I boot up second returns the guru meditation error, and the most recent commands you and Scott shared only caused VirtualBox to say this: “No memory available for page table or page directory.” (VERR_NO_PAGE_MEMORY).
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Guru Meditation error

Post by fth0 »

That's no surprise, since not everybody names their VM "tm1". ;)
Zdenek Kubenka
Posts: 2
Joined: 25. Sep 2015, 12:18

Re: Guru Meditation error

Post by Zdenek Kubenka »

I have similar problem.
Host: 12cores / 24 threads. RAM: 65445MB (63.9GB) total, 38229MB (37.3GB) available
VMs:
a) 4 cores, 8GB RAM, Debian
b) 4 cores, 16GB RAM, Windows 10

Run of VM ends with Guru Meditation. Tried to
VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1
--> now ends with VERR_NO_PAGE_MEMORY

Also tried to set --largepages off --> did not help.

I have no idea why this keeps happening. After reboot, there is obviously no problem, however later when I shutdown/startup the VMs, this keeps happening. Since its a dev box, I need to cycle the various VMs as needed.

Attaching log and RAMMap screenshot.
Attachments
VBox.log
(74.71 KiB) Downloaded 5 times
rammap.jpg
rammap.jpg (130.68 KiB) Viewed 940 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Guru Meditation error

Post by fth0 »

You could try if disabling System > Processor > Nested VT-x/AMD-V makes a difference.

Regarding the RAMMap screenshot, did you take it while the Guru Meditation dialog was displayed, or only afterwards?
Zdenek Kubenka
Posts: 2
Joined: 25. Sep 2015, 12:18

Re: Guru Meditation error

Post by Zdenek Kubenka »

As for the Rammap image - since i have "VBoxInternal/RamPreAlloc" set to 1, it tries to allocate full memory during the startup, so I dont get the Guru Mediation error during the VM run; so the rammap screenshot was taken after i tried to start the VM and it failed.


---
So, here i have tested it - disabled the Nested VT-x/AMD-V on the VMs, however it did not do much of a difference.

I was able to start the VM "Oracle Windows 10" (4 cores, 16 GB RAM); then i was able to start VM "Debian" (4 cores, 8 GB RAM); this all went fine. However when i tried to start VM "ORA_RAC_1", it aborted with VERR_NO_PAGE_MEMORY

And at the same time some of the apps started to behave erratically, for example the Chromium based browser froze.

I am uploading the logs for all 3 VMs; and also the jpegs with rammap and also the task manager with performance card opened.

Whats more interesting is the discrepancy of the free memory reported by the task manager and the rammap. I think this may be the real issue here - the VM manager thinks that there is still enough memory and tries to startup the VM; however in reality (as reported by rammap) there isnt any. The fact, that the other apps start behaving erratically may be due the fact the system is indeed out of free memory.

Any idea whats wrong here?
Attachments
logs_and_jpegs.zip
(205.18 KiB) Downloaded 3 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Guru Meditation error

Post by fth0 »

I'm not so sure if there really is an out-of-memory situation here (although VirtualBox claims that):

Like other type 2 hypervisors, VirtualBox uses Driver Locked memory for the VMs' RAM. The 1st and 2nd VM allocated 16 + 8 = 24 GB of RAM, and the 3rd VM crashed when trying to allocate their 16 GB of memory. VirtualBox tries to allocate this memory in chunks of 2 MB (large pages), and has only allocated a bit more than 1 GB (0x45fa1 4k pages), when apparently being refused the next 2 MB by the host OS.

I don't really understand why the Windows host couldn't fulfill the memory allocation request, since >15 GB were still used as disk cache (Standby column). I could imagine that the disk cache was fragmented and contained only 4k pages after some runtime, but then it should suffice to turn off large pages like you also tried (although not in the provided VBox.log file ;)).

From my POV, RAMMap and Task Manager show the same memory situation (compare Standby to Cached). BTW, VirtualBox doesn't care how much memory is available when it starts a VM, it just tries to get the memory page by page. This makes sense because VirtualBox cannot know how much memory the host OS is able to provide. Note that any decent OS always uses all available memory for caches, so there is never really free memory.
Post Reply