VirtualBox freezes after Windows inactivity/sleep

Discussions related to using VirtualBox on Windows hosts.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: VirtualBox freezes after Windows inactivity/sleep

Post by arQon »

jep: "1.16-windows" for both.

The old VM (now set to HDA still) managed another hang last night - its first - but it was right as I was heading off to bed, so I haven't really had a chance to poke at it again since.

The other piece that *is* genuinely different between the two VMs is that VBox continues to default to the Intel NIC rather than virtio, so I changed that on the old VM before restarting it.

(The Intel NIC, you'll recall, is actually quite a lot MORE performant than virtio even though it certainly shouldn't be. My guess is I changed it when the Intel driver had that monster CVE, then never changed it back). I'm not *expecting* that to make a difference, but since the .18 situation has gone from "100% lockups" to "random lockups" that unfortunately means I now have to just keep Trying Stuff each time then waiting for the next lockup to happen. :/

I also don't actually have any guarantee that the NEW VM isn't also potentially going to lock up: it's simply that since it managed to not do so ONCE when the old one had a 100% failure rate it was clearly better than that - but I don't really know how MUCH better it actually is. Since the lockup also tends to freeze the VM Manager, I haven't left the new one running, because a lockup of the old VM might also potentially cause the new one to freeze as well, when that new VM *itself* could be blameless. Still, I'll probably just spin it up and leave it idling at some point just out of curiosity, since I can afford the resources it would take.

add> The old VM was frozen again the next time I resumed the host, so that takes out one of the last few "real" deltas between the new machines. The new VM, OTOH, was still running.
The new one doesn't have the XHCI controller in use, or any shared folders. Either one of those clearly COULD be the source of the problem, but I'm going to leave them for later. For now, the next step is to jump to the HWE kernel and see what happens with that.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VirtualBox freezes after Windows inactivity/sleep

Post by fth0 »

Your descriptions branched in several different directions up to now, making it increasingly difficult to follow. I just want to let you know that I'm ok with that because I rather like the full picture than information filtered by the author. BTW, I think we've scared off most other readers by now. ;)

There is one piece of information where I'd also like to take a closer look:
arQon wrote:Since the lockup also tends to freeze the VM Manager
AFAIK, the VirtualBox Manager and the VMs have no direct connections to each other. For example, in a working environment, you can quit the VirtualBox Manager at any time, and this has no influence on the running VMs. The connection is the VBoxSVC process, which is started by any VirtualBox frontend (e.g. VirtualBox Manager, VBoxManage), and that spawns the VirtualBoxVM processes. Another common ground are the host hardware and the host OS.

I can imagine multiple VMs freezing for a common cause. But I'm wondering what can also make the VirtualBox Manager freeze ...

If you are so inclined, you could create and run several simple VMs that only boot to the UEFI shell, one with audio and one without audio (maybe more for other suspected causes), and see if some or all of those VMs freeze.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: VirtualBox freezes after Windows inactivity/sleep

Post by arQon »

So, after another dozen or so rounds of lockups with various settings (including without XHCI) I changed the audio back to HDA, and finally caught a break. The FIRST suspend on this run resumed successfully, but the SECOND one hardlocked. And there's an "easy" difference between the two: a 5th audio device missing from the second run - specifically, the device fth0 called out way back at the start of this. :)

While that doesn't go very far in excusing the crashiness, and even less so in explaining why the NEW VM consistently survives (though that one IS on rust rather than SSD, and can be seen greyed out for several seconds after a resume, whereas the old VM comes back up instantly: faster than the monitor does), I think it's at least a strong voice in support of a race or somesuch there causing VBox to trip over its shoelaces.

I've disabled the AMD HDA device completely, and tomorrow will probably be long enough for me to get a good idea whether that's worked around it or not. (Not least since I'll switch the audio back to AC97 too, which has yet to survive even a single suspend). But before that, sleep!

Regardless of the outcome, I'd like to thank fth0 again for pointing that out, because even if I had noticed the delta in this log without being aware of it already, I probably still wouldn't have paid any attention to it until I'd run out of other things to try - and that's a very long list on a VM this old and this heavily customised. :)

--

heh - crossposted with yours. As you say, I doubt anyone else is still trying to stay on top of this. Not that I'd blame them. :P
Like you, I'd rather have the full picture for nontrivial weirdnesses like this - because without it, you can burn a LOT of time only to be stabbed in the back by a critical piece of information the poster didn't feel was worth mentioning. :)

edit> actually added the logfile, which i always forget needs a "yes, really" confirmation. :P
Attachments
vbox-lockup2.7z
one run, one successful resume, one hardlock
(17.52 KiB) Downloaded 10 times
Last edited by arQon on 6. Feb 2021, 15:04, edited 2 times in total.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: VirtualBox freezes after Windows inactivity/sleep

Post by arQon »

> But I'm wondering what can also make the VirtualBox Manager freeze ...

ANYTHING that makes a VM freeze, AFAICT.

I think there are two cases where it's fairly evident: first, the Manager's attempt to show a thumbnailed copy of the screen; and second, anything to do with issuing commands (like Reset or Shutdown), where it appears to go into a blocking wait for either confirmation that the command was submitted successfully or for some piece of state to change, which never does in hard lockups like this one. Since that leaves what is at a minimum the UI thread of the Manager also permalocked (and it NEVER recovers, even after the VM process itself has been killed, AFAICT), although it's possible that some other threads in the Manager may technically still be running, the program itself can't be interacted with. Since that thread appears to also be the message pump for it, it also doesn't respond to WM_CLOSE/WM_QUIT etc either, so that ALSO can only be "recovered" by task-killing the process itself and then restarting it.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VirtualBox freezes after Windows inactivity/sleep

Post by fth0 »

arQon wrote:VM comes back up instantly: faster than the monitor does
That's an interesting finding that I haven't thought about yet. For the sake of "easier" replication, can you keep the monitor turned off while resuming the host, operate the computer blindly if necessary, and only turn on the monitor seconds after the VMs are (supposed to be ;)) running again? This could show if it is a (difficult to reproduce) race condition or simply an (easier to reproduce) matter of the order.
arQon wrote:until I'd run out of other things to try - and that's a very long list
That's the curse of knowledge: The more you know, the more you know what you don't know, and the more you know what you can try, and its getting more and more difficult to stop yourself. ;)

Regarding the VirtualBox Manager freezing: I understand your reasoning, but for lack of own experience, I'm not convinced yet. When the VM is frozen, can you open the help menu of the VirtualBox Manager? Does it only freeze when you're trying to do anything regarding the frozen VM? Does it only freeze when the thumbnailed screenshot is visible? I don't know how the thumbnailed screenshot is implemented, but would expect it to be a push-type operation, so that missing screenshots should not pose any problem. But what do I know? ;)
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: VirtualBox freezes after Windows inactivity/sleep

Post by arQon »

(many good thoughts snipped - am in a rush)

> can you open the help menu of the VirtualBox Manager?

No. The Manager app is clearly not servicing the message pump at all. (Hence the unusability of the close button etc).

> Does it only freeze when you're trying to do anything regarding the frozen VM?

No - or more accurately, once you DO try to do anything with the frozen VM, the Manager locks up. My RECOLLECTION is that it's actually already locked up locked before you DO do anything (i.e. it can't be closed even if trying to do so is the first action since the VM hung).

> Does it only freeze when the thumbnailed screenshot is visible?

That one I can't remember.

> I don't know how the thumbnailed screenshot is implemented, but would expect it to be a push-type operation, so that missing screenshots should not pose any problem. But what do I know? ;)

I'd think you mean pull "really", though it depends on more factors than I have time to go into right now. :)
Even simple shared memory, or acquiring the framebuffer from the video driver, or etc will likely have SOME kind of sync mechanism. As you say, it's not especially HARD to make it robust in the face of a hard VM lockup like this, and requires one to be expecting it to happen enough to do so, which is not necessarily warranted for it. But that robustness clearly isn't present, given this case.

Obv I'd need to reinstall the devices etc to answer the ones I'm uncertain about, so I need another suitable period to try.

TLDR for the main point though is, this has continued to work perfectly again since the last post, so danke schoen. \o/

More in a few days when I have time.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VirtualBox freezes after Windows inactivity/sleep

Post by fth0 »

arQon wrote:TLDR for the main point though is, this has continued to work perfectly again since the last post, so danke schoen. \o/
You're welcome, or bitte sehr.

We don't have to follow up on the freezing part. Just two short explanations for the background of my questions:

Regarding the Help menu test, my idea was that maybe the message loop only started hanging when the first VM-related action is taken, and the Help menu would be a VM-independent action. But this distinction is probably not important at all, so please ignore.

Regarding the screenshots, I imagined that the VirtualBox Manager initially tells the VBoxSVC that it would be willing to receive screenshots at any time, and later on the VBoxSVC would call some callback function inside the VirtualBox Manager for each new screenshot. In the case of missing screenshots, the VirtualBox Manager would have no reason to freeze. In fact, this happens during nearly all VM startups at least once (you surely know the corresponding log message ;)).
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: VirtualBox freezes after Windows inactivity/sleep

Post by arQon »

sonova...

So, earlier today I plugged my headphones in for a voicechat, and... yeah, the VM hardlocked again. This piece of code is FEROCIOUSLY broken. :(

It did give me a chance to check a couple of things: the Manager IS actually okay until you try to interact with the frozen VM. Not much consolation, since it still means the only thing you can do with the VM is kill the process (i.e. ACPI shutdown etc doesn't work, and if you try to use that you have to kill the Manager as well), but still meaningfully better than the worst case. :)

edit> @scott / mpack - I can't edit the thread title since I didn't start it, but I think it's pretty clear that we've defined the issue much more accurately than just "on resume" if you want to correct it. Thanks.
Martin
Volunteer
Posts: 2560
Joined: 30. May 2007, 18:05
Primary OS: Fedora other
VBox Version: PUEL
Guest OSses: XP, Win7, Win10, Linux, OS/2

Re: VirtualBox freezes after Windows inactivity/sleep

Post by Martin »

Can you check the windows event logs and sound settings if an audio output device disappears when you plug in the headset?
Some systems deactivate the local speakers and microfone when plugging in a headset.
There have been other reports that Virtualbox doesn't like audio devices being removed while they are actively used by a guest.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: VirtualBox freezes after Windows inactivity/sleep

Post by arQon »

I can't check them without freezing the VM, but I'm 100% confident that the speakers DON'T disappear, they're simply no longer the DEFAULT device.

(The reason I'm so confident is that for a while I was using voicechat daily, so I just left the headphones (note: headphones specifically, not headSET, i.e. no mic, in case that matters) plugged in, and switched the active device between them and the speakers depending on what I was doing).

sidenote: At no point in time was the audio actually in use when the VM froze, although the virtual device was obviously "known" to it.
vb_user_6128
Posts: 21
Joined: 15. Jan 2018, 23:54

Re: VirtualBox freezes after Windows inactivity/sleep

Post by vb_user_6128 »

scottgus1 wrote:Start the VM from full normal shutdown, not save-state. Run until you see the problem happen, then shut down the VM from within the VM's OS if possible. If not possible, close the Virtualbox window for the VM with the Power Off option set.

Right-click the VM in the main Virtualbox window's VM list, choose Show Log. Save the far left tab's log and the hardening log next to it, zip them, and post the zip file, using the forum's Upload Attachment tab.
I've recently discovered that this problem doesn't only occur when I leave my computer for a period of time. Yesterday the problem occurred while I was sitting at my computer working. VirtualBox became completely unresponsive and the only way I was able to close it was to keep clicking the red X in the upper right corner of the window. The VirtualBox window title bar displayed "Not Responding" during this time.

I've attached the requested log files to this post.
Attachments
VirtualBox Freeze Logs.zip
(57.11 KiB) Downloaded 9 times
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: VirtualBox freezes after Windows inactivity/sleep

Post by scottgus1 »

The VM seems to run well, aside from a suspend/resume at 38 and a half minutes.

At 2:14:11 the VM re-determines the host audio setup. Comparing the first scan at timestamp 00:00:03.762758 with the second scan at timestamp 02:14:11.605219, two devices drop out:
00:00:03.972341 Audio: Device 'Dell S2716DG (NVIDIA High Definition Audio)'
00:00:03.972370 Audio: Device 'Dell S2716DG (NVIDIA High Definition Audio)'
After some "DSound: DirectSound playback is currently unavailable"s, the VM re-enumerates the host audio again, at timestamp 02:14:13.806776, throws an error:
02:14:13.948976 DSound: DirectSound playback initialization failed with ERROR_TOO_MANY_OPEN_FILES 0x88890004
then sees one of the 'Dell S2716DG's come back, then that's it.

Was the VM active and usable between 0:38:30-ish and 2:14:11?
vb_user_6128
Posts: 21
Joined: 15. Jan 2018, 23:54

Re: VirtualBox freezes after Windows inactivity/sleep

Post by vb_user_6128 »

scottgus1 wrote:The VM seems to run well, aside from a suspend/resume at 38 and a half minutes.

At 2:14:11 the VM re-determines the host audio setup. Comparing the first scan at timestamp 00:00:03.762758 with the second scan at timestamp 02:14:11.605219, two devices drop out:
00:00:03.972341 Audio: Device 'Dell S2716DG (NVIDIA High Definition Audio)'
00:00:03.972370 Audio: Device 'Dell S2716DG (NVIDIA High Definition Audio)'
After some "DSound: DirectSound playback is currently unavailable"s, the VM re-enumerates the host audio again, at timestamp 02:14:13.806776, throws an error:
02:14:13.948976 DSound: DirectSound playback initialization failed with ERROR_TOO_MANY_OPEN_FILES 0x88890004
then sees one of the 'Dell S2716DG's come back, then that's it.

Was the VM active and usable between 0:38:30-ish and 2:14:11?
Unfortunately I wasn't paying close enough attention at the time the error occurred vs. when the VM was started to answer that question. I'll have to make a note of that for future reference.
vb_user_6128
Posts: 21
Joined: 15. Jan 2018, 23:54

Re: VirtualBox freezes after Windows inactivity/sleep

Post by vb_user_6128 »

After a period of time without experiencing this issue, it has returned and now the VirtualBox completely stops responding whenever the monitor powers off. I have attached the updated logs. Can anyone figure out what is going on?
Attachments
VirtualBox Freeze Logs 2021-06-03.zip
(55.7 KiB) Downloaded 8 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VirtualBox freezes after Windows inactivity/sleep

Post by fth0 »

In this thread, the issues of two VirtualBox users are mixed: Regarding the issue of arQon, I'm pretty sure that it is audio related. Regarding the issue of vb_user_6128 (the OP), I suspect it to be. Please test the new VirtualBox test build, which has extensive changes in the audio functionality.

VirtualBox test build(s) for audio problems

On the VirtualBox test builds page, you'll find the VirtualBox stable branch test build 6.1.23r145004 (or newer). The audio functionality underwent extensive changes, and the VirtualBox developers are interested in getting positive as well as negative feedback regarding audio playback and recording. Please mention the host OS, the guest OS, the Host Audio Driver and the Audio Controller (from the VM configuration in Settings > Audio), and describe any misbehavior as detailed as possible. Thanks in advance.
Post Reply