Ubuntu Server 22.04.3 randomly hangs after ~20 hours of running

[Andrew_241]

Hello everyone,

My ubuntu server cannot run for longer than about a day before it randomly hangs.
After a while, you will see

VMMDev: vmmDevHeartbeatFlatlinedTimer: Guest seems to be unresponsive. Last heartbeat received 4 seconds ago

in the logs, and the guest server become completely unresponsive, no way to properly shut it down.

Checking logs in the server itself after a boot, there is nothing that indicates a failure on Linux. No errors in the logs, just mostly docker containers running. No kernel panic or anything like that.

At this point i dont even know how to collect more information to determine the cause of the stop! The only thing i found a bit irregular was the constant suspend/resume cycle, but the crash happened after 1 hour of the last cycle. The vmmDevHeartbeatFlatlinedTimer was accurate with the detection of the unresponsiveness of the server, i managed to compare when it crashed with the syslogs, and it seemed fairly accurate.

I also installed guest additions in the hopes to see a change or a fix but no difference.

Here is my setup:

VIRTUAL BOX VERSION: Version 7.0.12 r159484 (Qt5.15.2)
HOST: Windows 11 Pro Build 22621
GUEST: Ubuntu Server 22.04.3

Attached a log file:

log.zip

(17.62 KiB) Downloaded 265 times

[Andrew_241]

It looks like downgrading to 6.1.48 fixed my problem. Can't confirm what the issue is, but i saw on reddit that maybe it has to do with the graphics controller. I am now using VMSVGA as opposed to whatever was default to 7.0.12. A nice test would be to bring back 7.0.12 and try different graphics controller.

So far, I am just happy to see that it no longer interrupts after a day

This is the reddit that helped me figure it out:
r/virtualbox/
"windows_11_guest_os_keeps_freezingcrashing"

[mpack]

Congratulations on finding a solution.

I am now using VMSVGA as opposed to whatever was default to 7.0.12.

VMSVGA has been the default for Linux guests for a very long time (6.1.0 I think). If you had some other setting and chose the right template then either the VM was created an even longer time ago, or you over-rode that VM template.

Still, I'm not aware that the old VBoxVGA controller had a tendency to hang. We still sometimes use it for very old guest OSs, e.g. XP. The only circumstances I'm aware of when Linux guests have been reported to hang is when the VM has aggressive power saving settings enabled: these are best turned off since virtual power doesn't really need saving, and it's the host that should manage physical power.

[fth0]

FWIW, it could also be a (rarely reported) consequence of running under Hyper-V hypervisor on the Windows host (aka green turtle mode ).

[Andrew_241]

Thank you for the replies everyone, it appears I might have spoken too soon. It crashed again, but this time it lasted ~35 hours.

@mpack thnx for the oversight, it looks like i was always using VMSVGA, i am going to give VBoxVGA a shot...

@fth0 I did sees that many users suggest disabling Hyper-V hypervisor on Windows 11, so i did disable all relevant settings under "Turn windows features on and off" (hyper-v, windows hypervisor platform) but when i checked systeminfo.exe, i see that "Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed.", meaning maybe hyper-v is not disabled completely after all. Some random users suggest disabling "memory core isolation" on winodws 11, but i feel like some of these claims are baseless... might try it experimentally regardless...

I just wish there was something more relevant in the linux logs or the VitrualBox logs... the lack of information on these systems make me speculate that the true problem might be my OS, windows 11 somehow... maybe some kind of efficiency mode or power saver behaviour...

Either way, I appreciate the support!!

[scottgus1]

@fth0 I did sees that many users suggest disabling Hyper-V hypervisor on Windows 11, so i did disable all relevant settings under "Turn windows features on and off"

Windows Features is often not enough. Please see HMR3Init: Attempting fall back to NEM (Hyper-V is active)

[Andrew_241]

@scottgus1, thank you so much for the guide. I actually did have the green turtle @fth0 was talking about after all!

Now my green turtle turned into a memory chip, so that should indicate hyper-v is out of the way.
In my case, all that needed to be done was disable "memory integrity" under "core isolation".

My instance with VboxVGA also crashed... For documentation purposes, I included the logs of my run with VboxVGA, will compare this with my next run, VMSVGA with no green turtle, brought back virtualbox 7.0.12 since the crash also happened on older version.

VBoxVGA_Green_Turtle_14h.zip

(18.5 KiB) Downloaded 253 times

Thanks again everyone for the support!!

[mpack]

I doubt that anyone will care about your experiences with VBoxVGA. As I already mentioned above, VBoxVGA has been deprecated for modern Linux guests for a very long time. To be blunt, you shouldn't be using it and you were not recommended to use it.

[arQon]

It's a little hard to credit that a Linux server would care about the display driver at all, and even less so that a thread specifically about Windows guests using a different driver would provide anything of use at all, let alone an actual solution. Whatever your problem really is, there are few things less likely to be relevant to it than the behavior of a different guest OS.

7.x doubtless has rare "weird" new bugs kicking around still, but it's much more likely that your initial suspicions were correct and you have something screwy going on with either the host or the guest completely unrelated to VirtualBox. The way to start diagnosing this sort of problem is to create a new VM with a fresh OS install and bisect from there.

[Andrew_241]

After PROPERLY following the link @scottgus1 posted, I have yet to get another crash, and my server has been running for 48 hours now. With 7.0.12 virtualbox and using VMSVGA, everything seems OK.

FWIW, it could also be a (rarely reported) consequence of running under Hyper-V hypervisor on the Windows host (aka green turtle mode ).

I think this was it after all.

@fth0 I did sees that many users suggest disabling Hyper-V hypervisor on Windows 11, so i did disable all relevant settings under "Turn windows features on and off"

Windows Features is often not enough. Please see HMR3Init: Attempting fall back to NEM (Hyper-V is active)

This guide is a must for everyone running windows 11!

@mpack: Noted, will not use deprecated stuff because reddit told me Thanks for the blunt statement.

@arQon: It was OS after all I believe. Nothing in guest linux logs, and nothing noteworthy in virtualbox logs, to me that pointed that it was probably a host issue in general, but the experienced folks here quickly picked up on the Hyper-V pattern of issues. Thanks for your reply!