VMs failing to start seemingly randomly

[radaxian]

Hello, people.

For the last 10 to 15 years I have always used VMware personally, but in the last year or so I have been using VirtualBox more and more in place of VMware, because that's what we use during class, so I don't have compatibility issues with the rest of the class.

I'm having, from the begining iirc, issues with VMs not starting up, seemingly randomly, after creating snapshots, after restoring snapshots, after deleting snapshots, and recently, after trying to power them up after doing absolutely nothing, and recently what I discovered is that, in a Windows host machine at least, if I close VirtualBox and terminate all related processes on task manager, then relaunch it, the VMs will start working again, but that is obviously not a good solution, specially when I have another five or six VMs already running. That is already annoying enough at home, but when we have to be present in situ, that's downright infuriating, considering we the students don't have administrative privileges on the machines and can't even open task manager, so the only solution is to reboot the host machine.

So, does anyone know how I can fix this issue so it stops happening, or at least of a way that doesn't require powering off all the running VMs and forcefully terminating VirtualBox?

Thanks in advance.

[scottgus1]

We should be able to help if we can get some info on what happens or what errors appear when the VM doesn't start up.

[radaxian]

I'll have to wait a little for the errors to start popping up again, they aren't easy to reproduce, a little while ago I did some testes with snapshots and everything worked fine, the issues seem to pop up randomly, or at least without a definitive pattern that I have been able to see at this point, the most common factor being the snapshots, but (un)fortunately it's not 100% of the time.

I'll log all the occurrences and come back here when I have more info, but for now, I went through my browsing history and found some info on the most recent error.

E_FAIL (0x80004005) with something about SessionMachine, and I got the message The VM session was closed before any attempt to power it on., I found some mentions of discarding saved states, but there were no save states to discard, again, the only way to start the VM was terminating every VirtualBox process and launching everything up again.

When I gather more info on the errors I receive right after any snapshot activity I'll post them here.

Thanks.

[scottgus1]

E_FAIL (0x80004005) and SessionMachine are common to just about all Virtualbox errors, but "The VM session was closed before any attempt to power it on" can happen when a serious hardening block or other host misconfiguration occurs too.

We'll be here when you have more data. Please use the forum's Upload Attachment tab to upload zipped VM logs when the errors happen again, and the error messages you get, we'll see what can be done.

[radaxian]

And it struck again.

This time right after I created a snapshot, same E_FAIL (0x80004005) SessionMachine error.

I have other 5 VMs running, besides this one that had the error. I finished some stuff on the guest OS, powered off the VM to create a snapshot before carrying on with more stuff, and when I tried to power it up again, that same error, that I'm sure will go away if I terminate all VirtualBox processes, but for that I'll need to power off my VMs.

You can download the logs here bananabanana.xyz/logs.zip, or if you want I can just post them here.

[scottgus1]

if you want I can just post them here.

Please do...

Please use the forum's Upload Attachment tab to upload zipped VM logs

[radaxian]

LOL, completely missed that.

Logs.zip

(88.05 KiB) Downloaded 6 times

[scottgus1]

Thanks for posting the log! This problem is a little fuzzy for me, will have to see what the forum gurus say. However, something that uses Hyper-V is enabled on the host, there's a ton of time offset:

00:50:18.311838 TM: Giving up catch-up attempt at a 60 030 610 539 ns lag; new total: 60 030 610 539 ns
01:10:32.718586 TM: Giving up catch-up attempt at a 60 001 070 885 ns lag; new total: 120 031 681 424 ns
01:27:55.305674 TM: Giving up catch-up attempt at a 60 005 487 138 ns lag; new total: 180 037 168 562 ns
02:01:47.706059 TM: Giving up catch-up attempt at a 60 008 843 043 ns lag; new total: 240 046 011 605 ns
02:27:03.961687 TM: Giving up catch-up attempt at a 60 003 766 186 ns lag; new total: 300 049 777 791 ns
03:32:47.580281 TM: Giving up catch-up attempt at a 60 002 959 908 ns lag; new total: 360 052 737 699 ns
03:59:23.571675 /TM/VirtualSync/CurrentOffset 388260078243 ns

This last time offset has been fingered by forum guru fth0 as a possible indicator of troubles from host Hyper-V being enabled, but not in the 6.1.26 Virtualbox you're running.

You could wait until the masters weigh in, or try turning Hyper-V off on the host, see HMR3Init: Attempting fall back to NEM (Hyper-V is active). This requires rebooting the host.

One possible problem, which may or may not be causing or influencing this problem, but will bite you eventually, is all the snapshots. This VM has 5 disks and 6 snapshots, which makes 30 open disk files that have to be hunted through to gather the VM's disk data. Be sure that you understand that snapshots aren't backups. They're time travel markers in the life of the VM, like Windows' System Restore Points. They store data that cannot change or be deleted, the whole chain must be present for the VM to work, and the VM is a lot more delicate.

Additionally, 5 disks is too much for a modern OS to try to draw from, if the host disk is a spinning platter disk and the VM OS demands high levels of MB from all of them at once. More than two high-demand reads will swamp a platter disk. SSDs should not have this limitation.

[fth0]

I think that using the VirtualBox Guest Additions 6.1.32 in combination with VirtualBox 6.1.26 is a bad idea. I'd suggest to either use the VirtualBox test builds 6.1.35r151866 (or newer), or to disable Hyper-V on the host.

[radaxian]

You could wait until the masters weigh in, or try turning Hyper-V off on the host, see HMR3Init: Attempting fall back to NEM (Hyper-V is active). This requires rebooting the host.

I have Hyper-V disabled, I don't think I ever enabled it on my machine since the last time I installed Windows, I recall enabling it a few years back when I tried to use Docker on Windows instead of using inside a Linux VM, just because I wanted to see how it worked on Windows.

Additionally, 5 disks is too much for a modern OS to try to draw from, if the host disk is a spinning platter disk and the VM OS demands high levels of MB from all of them at once. More than two high-demand reads will swamp a platter disk. SSDs should not have this limitation.

I know, I don't use it for backups, but as snapshots. If something goes horribly wrong with something I'm doing on the guest OS, I just go back to a previous snapshot, I'm also using it on an SSD, I don't have hard drives on this machine.

I think that using the VirtualBox Guest Additions 6.1.32 in combination with VirtualBox 6.1.26 is a bad idea. I'd suggest to either use the VirtualBox test builds 6.1.35r151866 (or newer), or to disable Hyper-V on the host.

I might have created this VM in another machine with a different version of VirtualBox, I don't quite remember at the moment, but one thing I know, I didn't install guest additions on that VM, it doesn't have a graphical interface, and the only time I interact with it using it's own display, is during install and initial setup, after that, only through ssh. Unless Ubuntu detects it's running on VirtualBox and installs it along other packages with apt upgrade (I admit I don't really pay attention to what's being installed in this kind of situation, only in real systems doing real work).

But I'll try to update everything and see, though I'm not very confident it will work, me and my colleagues have been having this exact issue with snapshots (that's the reason almost all of them don't use snapshots, and when things go wrong, instead of reloading a previous snapshot, they need to start from zero, this issue has been wasting quite some time for a number of people) for quite some time, spanning many different versions.

Anyway, thank you guys for the attention so far.

[scottgus1]

I have Hyper-V disabled

Log says otherwise:

00:00:01.017348 HM: HMR3Init: Attempting fall back to NEM: VT-x is not available

Did you read the tutorial? Something on the host has enabled Hyper-V.

one thing I know, I didn't install guest additions on that VM ... Unless Ubuntu detects it's running on VirtualBox and installs it along other packages with apt upgrade

Log says the GAs are there and too modern for the host Virtualbox:

00:00:00.414506 VirtualBox VM 6.1.26 r145957 win.amd64 (Jul 28 2021 17:15:09) release log
...
00:00:26.512038 VMMDev: Guest Additions information report: Version 6.1.32 r149290 '6.1.32_Ubuntu'

[radaxian]

I have Hyper-V disabled

Log says otherwise:

00:00:01.017348 HM: HMR3Init: Attempting fall back to NEM: VT-x is not available

But Windows says other-otherwise.


http://pedrorotoli.com/images/hyper-v.png (don't know why the image isn't showing up here)

Did you read the tutorial? Something on the host has enabled Hyper-V.

Not really, which tutorial exactly?

one thing I know, I didn't install guest additions on that VM ... Unless Ubuntu detects it's running on VirtualBox and installs it along other packages with apt upgrade

Log says the GAs are there and too modern for the host Virtualbox:

00:00:00.414506 VirtualBox VM 6.1.26 r145957 win.amd64 (Jul 28 2021 17:15:09) release log
...
00:00:26.512038 VMMDev: Guest Additions information report: Version 6.1.32 r149290 '6.1.32_Ubuntu'

I just created a new VM and installed Ubuntu Server 20.04 LTS, restarted it a few times, created a new snapshot with the VM powered off, it started up fine again, then updated the system and restarted, and now it won't boot, same issue, but with Guest Additions version (aparently Ubuntu installs it automatically) matching VirtualBox's version.

So, I guess that VM really was created on a different machine, but the issue persists even with a VM created and used only on my machine. So, while I won't discard the possibility of the issue being caused by mismatched versions of VirtualBox and Guest Additions, the issue still happens with matching versions, a single disk and only one snapshot.

I'm at a loss here.

[scottgus1]

which tutorial exactly?

The one I linked to in my earlier post above.

But Windows says other-otherwise.

The tutorial covers this...

don't know why the image isn't showing up here

[IMG] tags don't work for 3rd-party-link sites. [IMG] only works for images previously uploaded using the forum's Upload Attachment tab. You can find your previous attachments under your forum User Control Panel link, Manage Attachments.

I just created a new VM and installed Ubuntu Server 20.04 LTS, restarted it a few times, created a new snapshot with the VM powered off, it started up fine again, then updated the system and restarted, and now it won't boot, same issue,

Until Hyper-V is disabled, we won't know if that's the cause of this issue. But posting a log from a new VM for comparison is a good idea.

[radaxian]

which tutorial exactly?

The one I linked to in my earlier post above.

I see, when you talked about tutorial I was picturing something different, and yes, I already did those Hyper-V steps, and it is disabled, if it weren't I also wouldn't be able to use VMware, which I do.

don't know why the image isn't showing up here

[IMG] tags don't work for 3rd-party-link sites. [IMG] only works for images previously uploaded using the forum's Upload Attachment tab. You can find your previous attachments under your forum User Control Panel link, Manage Attachments.

I guess I'll have to look a little deeper into how the posting system works in this forum.

I just created a new VM and installed Ubuntu Server 20.04 LTS, restarted it a few times, created a new snapshot with the VM powered off, it started up fine again, then updated the system and restarted, and now it won't boot, same issue,

Until Hyper-V is disabled, we won't know if that's the cause of this issue. But posting a log from a new VM for comparison is a good idea.

As I said before, Hyper-V is disabled, was never enabled, and I followed those steps just to double check, the issue persists.

On a side note, while I do have VMware installed, I very rarely use both, VMware and VirtualBox at the same time, haven't done that in quite some time.

Below are the logs of the new VM, and funnily enough, when this test VM failed to start, the previous one also wouldn't start, but other VMs booted just fine.

If you want I can give you a link to download the test VM.

[scottgus1]

00:00:00.899489 HM: HMR3Init: Attempting fall back to NEM: VT-x is not available

Hyper-V is still on.

Please follow this part of the tutorial:

If the green turtle still appears and the tell-tale lines are in the log, try all the steps again. If you don't get the standard virtualization icon, post back exactly what you did and we'll try to help some more.

Also note this part:

If your computer is running in a corporate environment with IT overlords, they may have forced Hyper-V, and none of the above will help. You will have to get IT to turn off Hyper-V so that you can run Virtualbox.