VM froze; no response or access to guest console

Discussions about using Linux guests in VirtualBox.
Post Reply
HarryMan
Posts: 113
Joined: 15. Jan 2014, 13:54
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: devuan, MX-Linux, others

VM froze; no response or access to guest console

Post by HarryMan »

One of my vbox VMs became unresponsive sometime while I stepped away (maybe 5-6 hours). I was greeted by a black screen. About the only thing I was able to do with it was switch from fullscreen mode to desktop mode. I tried to obtain a console inside the VM (running devuan chimaera) but there was no response. I tried an ACPI shutdown, which did not work. Then I tried to stop the VM from the vbox gui, which also did not work. Finally, I forced the VM to stop with SIGTERM (not SIGKILL) from the host.

The host and all guests are running 6.1.34. The host Devuan Beowulf; guest in question is Devuan Chimaera.

All other VMs appear (so far) to be running normally. I checked the kern.log and syslog on the host and the guest (after restarting it). I do not notice anything of note, other than the guest's kern.log having a lot of errors "drm:vmw_msg_ioctl [vmwgfx]] *ERROR* Failed to open channel" -- but a search of the web seems to indicate these are not real problems.

I am attaching two logs, one which stops at the freeze, and the other after I sent the SIGTERM until it finally shut down with an aborted state.
Attachments
myvm0-2022-04-20-18-45-37.log.gz
Log up until the freeze
(62.76 KiB) Downloaded 9 times
myvm0-2022-04-20-18-39-41.log.gz
Same log, with events subsequent to killing the VM with SIGTERM.
(62.14 KiB) Downloaded 5 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VM froze; no response or access to guest console

Post by fth0 »

myvm0-2022-04-20-18-45-37.log wrote:
20:09:42.302727 06:08:31.949172 AsyncCompletion: Task 0x007fe165312d40 completed after 10 seconds
[...]
14:52:06.782885 144:50:56.426675 AsyncCompletion: Task 0x007fe1650b6f80 completed after 13 seconds
[...]
19:26:19.547412 149:25:09.193786 AHCI#0: Port 1 reset
19:26:19.555154 149:25:09.201593 AHCI#0: Port 0 reset
19:26:19.556323 149:25:09.202754 VD#0: Cancelling all active requests
[...]
19:28:22.923611 149:27:12.569951 VMMDev: Guest Log: vgdrvLinuxIOCtlSlow: copy_to_user(0x7fff9a40b030,,0x1c); uCmd=0xc020560a!
[...]
01:44:52.242834 155:43:41.889277 Changing the VM state from 'RUNNING' to 'SUSPENDING'
01:44:52.242911 155:43:41.889353 PDMR3Suspend: after     0 ms, 1 loops: 1 async tasks - ahci/0
01:44:53.553642 155:43:43.200094 PDMR3Suspend: after  1310 ms, 3 loops: 1 async tasks - ahci/0
01:44:54.553899 155:43:44.200252 PDMR3Suspend: after  2310 ms, 4 loops: 1 async tasks - ahci/0
01:44:56.553495 155:43:46.199884 PDMR3Suspend: after  4310 ms, 6 loops: 1 async tasks - ahci/0
01:45:00.553777 155:43:50.200165 PDMR3Suspend: after  8310 ms, 10 loops: 1 async tasks - ahci/0
01:45:08.552925 155:43:58.199318 PDMR3Suspend: after 16309 ms, 18 loops: 1 async tasks - ahci/0
01:45:24.554815 155:44:14.201194 PDMR3Suspend: after 32311 ms, 34 loops: 1 async tasks - ahci/0
FWIW, those log messages indicate problems with disk I/O. In the past, I've mostly seen this behavior with long-running VMs (like yours).
HarryMan
Posts: 113
Joined: 15. Jan 2014, 13:54
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: devuan, MX-Linux, others

Re: VM froze; no response or access to guest console

Post by HarryMan »

Thank you for the reply. Can you describe what sort of disk problems? Is this limited only to the guest, or does this involve the virtualbox host-side disk i/o also?
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VM froze; no response or access to guest console

Post by fth0 »

AFAIU (but I'm no expert on that), the host-side disk I/O would be my primary suspect. The AsyncCompletion log messages are only written when a task needed at least 10 seconds, which is a very long time IMO. You can check for those log messages while the VM is running. In the provided log, they appeared already after 6 hours runtime, which hints to a general problem.

As a test, you could toggle Storage > Controller: SATA > Use Host I/O Cache. Note that this could make it better or worse.
HarryMan
Posts: 113
Joined: 15. Jan 2014, 13:54
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: devuan, MX-Linux, others

Re: VM froze; no response or access to guest console

Post by HarryMan »

I've just upgraded the host (debian 10 to debian 11). Keep in mind that I've rarely seen this scenario. So it could just be a one-off.

So I'm going to take a wait-and-see approach on this.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VM froze; no response or access to guest console

Post by fth0 »

HarryMan wrote:Keep in mind that I've rarely seen this scenario.
I understand that you're talking about the freezing in the end here. But I suspect that you're having a general problem recognizable by the AsyncCompletion log messages, and it wouldn't hurt to check all your VBox.log files of all your VMs for that. ;)
HarryMan
Posts: 113
Joined: 15. Jan 2014, 13:54
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: devuan, MX-Linux, others

Re: VM froze; no response or access to guest console

Post by HarryMan »

I see these messages on several VMs.

EDIT: I do not see any kernel log errors on the host, at least not ones like "ata0...some error"-- are there some other linux kernel messages I should look for on the host side?
Incidentally, I have not seen a freeze-up again since I reported this initially. I am not sure if the system upgrade to Chimaera (debian 11) did or did not impact things. I'd like to stay on the newer kernel (5.10.0)
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VM froze; no response or access to guest console

Post by fth0 »

HarryMan wrote:I do not see any kernel log errors on the host, at least not ones like "ata0...some error"-- are there some other linux kernel messages I should look for on the host side?
I don't have any concrete idea regarding that, sorry.
HarryMan
Posts: 113
Joined: 15. Jan 2014, 13:54
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: devuan, MX-Linux, others

Re: VM froze; no response or access to guest console

Post by HarryMan »

Since upgrading to devuan chimaera, I've not seen this scenario. OTOH, this type of failure is intermittent and unpredictable.

So I am not sure if this is solved, per se, or if the bug still lurks, lying in wait for me... :)
Post Reply