Guru Meditation error
-
- Posts: 1
- Joined: 26. Sep 2021, 01:30
Guru Meditation error
Running into this crash consistently, happened 3 times in a row. Not really sure how to proceed so I would really appreciate some advice!
Version of VirtualBox: 6.1.26 r145957 (Qt5.6.2)
Using Guest Additions: No
Host OS: Windows Server 2019, 64-bit, 1.5 TB RAM
Guest OS: Red Hat Enterprise Linux 8.4, 64-bit, 300 GB RAM
Description of what I was doing: I was running a web application server on the VM while running a Postman-like tool on the host machine, simply sending requests to the server in the VM.
Version of VirtualBox: 6.1.26 r145957 (Qt5.6.2)
Using Guest Additions: No
Host OS: Windows Server 2019, 64-bit, 1.5 TB RAM
Guest OS: Red Hat Enterprise Linux 8.4, 64-bit, 300 GB RAM
Description of what I was doing: I was running a web application server on the VM while running a Postman-like tool on the host machine, simply sending requests to the server in the VM.
- Attachments
-
- VBox.zip
- (166.1 KiB) Downloaded 41 times
-
- VBoxHardening.zip
- (23.4 KiB) Downloaded 37 times
-
- Site Moderator
- Posts: 20945
- Joined: 30. Dec 2009, 20:14
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Windows, Linux
Re: Guru Meditation error
Have to say, looks like an awesome server! 96 cores 'hyperthtreaded' (dual AMDs), 1.5TB RAM! Must have hit the war chest a little bit!
Despite having "only" 300 GB out of that 1.5TB RAM set aside for the RedHat VM, the guru meditated over insufficient RAM:
Adding to this possibility is that Linux does not touch all RAM at boot, so Linux VMs allow lazy allocation to work. Windows VMs touch all RAM at boot so all VM RAM gets allocated at boot or a RAM error occurs at boot, before you can get into usage of the VM.
A command can be used to turn off lazy allocation for a VM:
cd "C:\Program Files\Oracle\Virtualbox"
VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1
Run it for each VM that you want to avoid lazy allocation.
Despite having "only" 300 GB out of that 1.5TB RAM set aside for the RedHat VM, the guru meditated over insufficient RAM:
I surmise this is because Virtualbox has "lazy" RAM allocation for its VMs: it pulls RAM from the host as the VM calls for it. However, if the VM eventually calls for all of it, RAM has to be available for the VM. Since Virtualbox does not by default allocate all of the RAM at once, it is possible for other processes or VMs to take up RAM and leave the VM asking for RAM it expects but that the host cannot now allocate.01:34:44.800442 PGMR3PhysAllocateLargePage: allocating large pages takes too long (last attempt 103 ms; nr of timeouts 11); DISABLE
02:27:05.319955
02:27:05.319957 !!R0-Assertion Failed!!
02:27:05.319963 Expression: cRefs == PGMPOOL_TD_CREFS_PHYSEXT
02:27:05.319964 Location : F:\tinderbox\win-6.1\src\VBox\VMM\VMMAll\PGMAllPool.cpp(4132) void __cdecl pgmPoolTrackPhysExtDerefGCPhys(struct PGMPOOL *,struct PGMPOOLPAGE *,union PGMPAGE *,unsigned short)
02:27:05.320021 cRefs=0 pPhysPage=A-:RAM:01bd88200000:001d000:-0000 pPage=ffffa76d9f21d9c0:{.idx=495}
02:27:05.320038 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
02:27:05.320069 Console: Machine state changed to 'GuruMeditation'
02:27:05.320399 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
02:27:05.320401 !!
02:27:05.320402 !! VCPU0: Guru Meditation -2701 (VERR_VMM_RING0_ASSERTION)
Adding to this possibility is that Linux does not touch all RAM at boot, so Linux VMs allow lazy allocation to work. Windows VMs touch all RAM at boot so all VM RAM gets allocated at boot or a RAM error occurs at boot, before you can get into usage of the VM.
A command can be used to turn off lazy allocation for a VM:
cd "C:\Program Files\Oracle\Virtualbox"
VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1
Run it for each VM that you want to avoid lazy allocation.
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Guru Meditation error
The following advice may seem counterintuitive at first, but please try if disabling large pages helps, by executing the following command while the VM is not running:
PS: If following scottgus1's suggestion to disable the lazy RAM allocation made the crash appear faster, you can combine that with my suggestion above for a faster initial test.
Code: Select all
cd C:\Program Files\Oracle\VirtualBox
VBoxManage modifyvm "tm1" --largepages off
-
- Posts: 2
- Joined: 27. Feb 2024, 06:50
Re: Guru Meditation error
@fth0: That says, “Could not find a registered machine named “tm1”.
Last edited by maniacal mator on 27. Feb 2024, 07:41, edited 1 time in total.
-
- Posts: 2
- Joined: 27. Feb 2024, 06:50
Re: Guru Meditation error
I tried the tm1 command, but now I’m getting a verr_no_page_memory error. That command didn’t take off any of my memory from my virtual machine, did it? I am having this problem when I boot up two vms. Whichever one I boot up second returns the guru meditation error, and the most recent commands you and Scott shared only caused VirtualBox to say this: “No memory available for page table or page directory.” (VERR_NO_PAGE_MEMORY).
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Guru Meditation error
That's no surprise, since not everybody names their VM "tm1".
-
- Posts: 2
- Joined: 25. Sep 2015, 12:18
Re: Guru Meditation error
I have similar problem.
Host: 12cores / 24 threads. RAM: 65445MB (63.9GB) total, 38229MB (37.3GB) available
VMs:
a) 4 cores, 8GB RAM, Debian
b) 4 cores, 16GB RAM, Windows 10
Run of VM ends with Guru Meditation. Tried to
VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1
--> now ends with VERR_NO_PAGE_MEMORY
Also tried to set --largepages off --> did not help.
I have no idea why this keeps happening. After reboot, there is obviously no problem, however later when I shutdown/startup the VMs, this keeps happening. Since its a dev box, I need to cycle the various VMs as needed.
Attaching log and RAMMap screenshot.
Host: 12cores / 24 threads. RAM: 65445MB (63.9GB) total, 38229MB (37.3GB) available
VMs:
a) 4 cores, 8GB RAM, Debian
b) 4 cores, 16GB RAM, Windows 10
Run of VM ends with Guru Meditation. Tried to
VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" 1
--> now ends with VERR_NO_PAGE_MEMORY
Also tried to set --largepages off --> did not help.
I have no idea why this keeps happening. After reboot, there is obviously no problem, however later when I shutdown/startup the VMs, this keeps happening. Since its a dev box, I need to cycle the various VMs as needed.
Attaching log and RAMMap screenshot.
- Attachments
-
- VBox.log
- (74.71 KiB) Downloaded 14 times
-
- rammap.jpg (130.68 KiB) Viewed 1188 times
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Guru Meditation error
You could try if disabling System > Processor > Nested VT-x/AMD-V makes a difference.
Regarding the RAMMap screenshot, did you take it while the Guru Meditation dialog was displayed, or only afterwards?
Regarding the RAMMap screenshot, did you take it while the Guru Meditation dialog was displayed, or only afterwards?
-
- Posts: 2
- Joined: 25. Sep 2015, 12:18
Re: Guru Meditation error
As for the Rammap image - since i have "VBoxInternal/RamPreAlloc" set to 1, it tries to allocate full memory during the startup, so I dont get the Guru Mediation error during the VM run; so the rammap screenshot was taken after i tried to start the VM and it failed.
---
So, here i have tested it - disabled the Nested VT-x/AMD-V on the VMs, however it did not do much of a difference.
I was able to start the VM "Oracle Windows 10" (4 cores, 16 GB RAM); then i was able to start VM "Debian" (4 cores, 8 GB RAM); this all went fine. However when i tried to start VM "ORA_RAC_1", it aborted with VERR_NO_PAGE_MEMORY
And at the same time some of the apps started to behave erratically, for example the Chromium based browser froze.
I am uploading the logs for all 3 VMs; and also the jpegs with rammap and also the task manager with performance card opened.
Whats more interesting is the discrepancy of the free memory reported by the task manager and the rammap. I think this may be the real issue here - the VM manager thinks that there is still enough memory and tries to startup the VM; however in reality (as reported by rammap) there isnt any. The fact, that the other apps start behaving erratically may be due the fact the system is indeed out of free memory.
Any idea whats wrong here?
---
So, here i have tested it - disabled the Nested VT-x/AMD-V on the VMs, however it did not do much of a difference.
I was able to start the VM "Oracle Windows 10" (4 cores, 16 GB RAM); then i was able to start VM "Debian" (4 cores, 8 GB RAM); this all went fine. However when i tried to start VM "ORA_RAC_1", it aborted with VERR_NO_PAGE_MEMORY
And at the same time some of the apps started to behave erratically, for example the Chromium based browser froze.
I am uploading the logs for all 3 VMs; and also the jpegs with rammap and also the task manager with performance card opened.
Whats more interesting is the discrepancy of the free memory reported by the task manager and the rammap. I think this may be the real issue here - the VM manager thinks that there is still enough memory and tries to startup the VM; however in reality (as reported by rammap) there isnt any. The fact, that the other apps start behaving erratically may be due the fact the system is indeed out of free memory.
Any idea whats wrong here?
- Attachments
-
- logs_and_jpegs.zip
- (205.18 KiB) Downloaded 15 times
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Guru Meditation error
I'm not so sure if there really is an out-of-memory situation here (although VirtualBox claims that):
Like other type 2 hypervisors, VirtualBox uses Driver Locked memory for the VMs' RAM. The 1st and 2nd VM allocated 16 + 8 = 24 GB of RAM, and the 3rd VM crashed when trying to allocate their 16 GB of memory. VirtualBox tries to allocate this memory in chunks of 2 MB (large pages), and has only allocated a bit more than 1 GB (0x45fa1 4k pages), when apparently being refused the next 2 MB by the host OS.
I don't really understand why the Windows host couldn't fulfill the memory allocation request, since >15 GB were still used as disk cache (Standby column). I could imagine that the disk cache was fragmented and contained only 4k pages after some runtime, but then it should suffice to turn off large pages like you also tried (although not in the provided VBox.log file ).
From my POV, RAMMap and Task Manager show the same memory situation (compare Standby to Cached). BTW, VirtualBox doesn't care how much memory is available when it starts a VM, it just tries to get the memory page by page. This makes sense because VirtualBox cannot know how much memory the host OS is able to provide. Note that any decent OS always uses all available memory for caches, so there is never really free memory.
Like other type 2 hypervisors, VirtualBox uses Driver Locked memory for the VMs' RAM. The 1st and 2nd VM allocated 16 + 8 = 24 GB of RAM, and the 3rd VM crashed when trying to allocate their 16 GB of memory. VirtualBox tries to allocate this memory in chunks of 2 MB (large pages), and has only allocated a bit more than 1 GB (0x45fa1 4k pages), when apparently being refused the next 2 MB by the host OS.
I don't really understand why the Windows host couldn't fulfill the memory allocation request, since >15 GB were still used as disk cache (Standby column). I could imagine that the disk cache was fragmented and contained only 4k pages after some runtime, but then it should suffice to turn off large pages like you also tried (although not in the provided VBox.log file ).
From my POV, RAMMap and Task Manager show the same memory situation (compare Standby to Cached). BTW, VirtualBox doesn't care how much memory is available when it starts a VM, it just tries to get the memory page by page. This makes sense because VirtualBox cannot know how much memory the host OS is able to provide. Note that any decent OS always uses all available memory for caches, so there is never really free memory.