VCPU3: Guru Meditation -3817 (VERR_GMM_CHUNK_INSERT)

[flybugz]

Hi, we running 100+ vms on single server

vboxmanage --version
5.2.34r133893


Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
1510 GB RAM

Sometimes wnen sever load above the average

total used free shared buff/cache available
Mem: 1510 1034 31 0 444 468
Swap: 27 0 27

CPU load average: 58.83, 49.70, 48.00

VMs (linux guests) randomly getting stuck state with the same error:

00:16:03.029611 !! VCPU3: Guru Meditation -3817 (VERR_GMM_CHUNK_INSERT)
/** Failed to obtain the GMM instance. */

Can someone explain what means this error ?
It would be nice to get any ideas on how to solve this error also

thanks

[mpack]

That seems to be a paged memory error. Please post a zipped VM log. Make sure the VM is fully shut down, then right click it in the manager UI. Select "Show Log" and save "VBox.log" (no other file) to a zip file. Attach the zip here.

[flybugz]

added log file: VBox.log.1


hapens 2 times with this vm
VBox.log.1:112:57:36.060084 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
VBox.log.1:112:57:36.060126 Console: Machine state changed to 'GuruMeditation'
VBox.log.1:112:57:36.060696 !! VCPU3: Guru Meditation -3817 (VERR_GMM_CHUNK_INSERT)
VBox.log.1:112:57:36.184080 !! VCPU0: Guru Meditation -3817 (VERR_GMM_CHUNK_INSERT)

[scottgus1]

added log file: VBox.log.1

Your host may have picked up Microsoft's habit to have known extensions be hidden. Vbox.log.1 is the second newest log. "Vbox.log" is the newest, and the one Mpack is wanting to see.

[flybugz]

Hi, stoped the Vm and downloaded the last log file, same vm as before

[mpack]

Why are you using such an old version of VirtualBox (5.2.34)? That version (5.2.x) debuted in 2017 and stopped being supported in 2020.

The other thing I notice is the huge amount of RAM you have - around half a terabyte of it. This would be unexpected in 2017.

IMO these two facts are possibly related.

Also you say you are running 100+ VMs on a single server. Surely you don't mean simultaneously? I'm afraid there is no chance of me getting a PC anything like your any time soon so I have no idea if your modest (except for RAM) older Debian host can actually cope with that load.

[flybugz]

We are using some infrastructure which depends on this VBox version. VBox can be upgraded to the latest version, but in this case, we also have to upgrade our infrastructure. It looks like it takes a lot of time and resources, and there's no guarantee we won't face the same issue on the new VBox version.

Also we running simultaneously around 60-70 vms on single server
this issue detected on 2 servers winch have 1.5 TB of tam
10 others servers with 256gb - 700GB working fine with up to 40+ vms running simultaneously

[fth0]

FWIW, this Guru Meditation is a consequential error of the failure to procure handy pages, which is part of VirtualBox's internal memory management. In my early VirtualBox days, when VirtualBox 5.2 was still in common use, I've seen several user reports with similar errors, but they often had to do with host-side RAM shortages, which shouldn't be the case here.

In VirtualBox 6.1 and 7.0, I haven't seen those errors any more (I don't remember VirtualBox 6.0 in this regard). However, new types of memory management errors were reported in large-scale setups that haven't been reproduced, understood or fixed so far.

Additionally, I'm pretty sure that nobody will investigate issues in VirtualBox 5.2 any more.

I do understand that it is a difficult decision for you to try newer VirtualBox versions, but I don't see a good alternative.

[flybugz]

Thanks guys for helping us, in this case we found temporary solution do not use more than 1TB of memory on our servers with vbox 5.2
We have reduced the number of virtual machines that work simultaneously and we no longer see such problems