VMs don't start after RHEL 8 update

Discussions related to using VirtualBox on Linux hosts.
Post Reply
Kraut
Posts: 9
Joined: 29. Sep 2021, 01:31
Primary OS: Linux other
VBox Version: OSE other
Guest OSses: Linux and Windows 10
Location: Vancouver, BC

VMs don't start after RHEL 8 update

Post by Kraut »

Since Monday I'm getting this dreaded generic error when starting any of my VMs on this RHEL 8 host (both Linux and Windows 10 guests):
VBoxManage: error: The virtual machine 'XXX Virtual Server' has terminated unexpectedly during startup with exit code 1 (0x1)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine
Usually, there is additional information, e.g., when the kernel modules are not loaded, but not in this case. This failure occurs before VirtualBox opens the log file, i.e., the Logs directory contains only old log files from before Monday. How does one go about diagnosing this when there is not even a log file? Nothing is logged to the messages system log file either.
The significance of Monday is that the system had been powered down on Friday for a planned power outage over the weekend, and RHEL 8 was upgraded to the latest version of all packages. It now runs this kernel:
Linux xxx 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
The Virtual Box version is 6.1.26 r145957 (Qt5.12.5)
When VirtualBox was last working, the system ran this kernel:
4.18.0-305.12.1.el8_4.x86_64
I can still boot into that version, but the same error happens.
During the upgrade and reboot I saw that selinux relabeling occurred. Just in case, I changed selinux from enforcing to permissive, but this did not help. The system uses the latest nvidia video driver, and that was changed to use their prebuilt, presigned kernel modules. I tried starting a VM in headless mode, but got the same error.
This system is a secure boot system, which means I had to generate and sign the VirtualBox kernel modules after the kernel upgrade. This was done, the modules load, but the VMs do not start. I created a new VM, and this does not start either. Reinstalling VirtualBox-6.1 did not help.
Any suggestions what to look at next?
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: VMs don't start after RHEL 8 update

Post by scottgus1 »

I remember reading that selinux is not compatible with Virtualbox. Can you turn that off completely?
Kraut
Posts: 9
Joined: 29. Sep 2021, 01:31
Primary OS: Linux other
VBox Version: OSE other
Guest OSses: Linux and Windows 10
Location: Vancouver, BC

Re: VMs don't start after RHEL 8 update

Post by Kraut »

scottgus1 wrote:I remember reading that selinux is not compatible with Virtualbox. Can you turn that off completely?
Thanks for the suggestion. I tried this just now, but it makes no difference:
[joachim@host ~]$ sestatus
SELinux status: disabled
[joachim@host ~]$ vboxmanage startvm 'XXX Virtual Server'
Waiting for VM "XXX Virtual Server" to power on...
VBoxManage: error: The virtual machine 'XXX Virtual Server' has terminated unexpectedly during startup with exit code 1 (0x1)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine
I considered this a long shot because I saw that the VirtualBox installation scripts are tweaking selinux settings, which suggests to me that it is intended to work with selinux enabled.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VMs don't start after RHEL 8 update

Post by fth0 »

Kraut wrote:How does one go about diagnosing this when there is not even a log file?
One takes a look in the global settings folder $HOME/.config/VirtualBox next. ;)

If you zip and provide the contents of this folder, we'd take a look, too.
Kraut
Posts: 9
Joined: 29. Sep 2021, 01:31
Primary OS: Linux other
VBox Version: OSE other
Guest OSses: Linux and Windows 10
Location: Vancouver, BC

Re: VMs don't start after RHEL 8 update

Post by Kraut »

fth0 wrote: One takes a look in the global settings folder $HOME/.config/VirtualBox next. ;)

If you zip and provide the contents of this folder, we'd take a look, too.
Thanks for the pointer to $HOME/.config/VirtualBox, I was not aware of that directory. This contains a couple of log files, but I don't see any additional details beyond the generic error message, which the VirtualBox GUi or vboxmanage report. I'm uploading an archive of $HOME/.config/VirtualBox with this message, in case somebody else can see something useful.

Is there an option to get more verbose tracing when starting a VM?
Attachments
dot_config_virtualbox.tar.gz
Archive of [i]$HOME/.config/VirtualBox[/i]
(145.92 KiB) Downloaded 5 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VMs don't start after RHEL 8 update

Post by fth0 »

Kraut wrote:Is there an option to get more verbose tracing when starting a VM?
Yes, but then you need to find somebody that can interpret it. ;)

Seriously, reboot your host so that no VirtualBox component is running. In a (BASH type) shell, enter the following commands:

Code: Select all

# VBoxSVC logging
export VBOXSVC_RELEASE_LOG="all=~0"
export VBOXSVC_RELEASE_LOG_FLAGS="thread group flag"
export VBOXSVC_RELEASE_LOG_DEST="file=/tmp/VBoxSVC.log"

# VBox logging
export VBOX_RELEASE_LOG="all=~0"
export VBOX_RELEASE_LOG_FLAGS="thread group flag"
export VBOX_RELEASE_LOG_DEST="file=/tmp/VBox.log"

# Start VM
VirtualBoxVM --startvm "VM" &
Kraut
Posts: 9
Joined: 29. Sep 2021, 01:31
Primary OS: Linux other
VBox Version: OSE other
Guest OSses: Linux and Windows 10
Location: Vancouver, BC

Re: VMs don't start after RHEL 8 update

Post by Kraut »

fth0 wrote:reboot your host so that no VirtualBox component is running.
Will do this as soon as a long-running build is complete. In the meantime, I tried the suggested steps without first rebooting. This produced the following error:

Code: Select all

[joachima@na152 ~]$ VirtualBoxVM: Error -610 in supR3HardenedMainInitRuntime!
VirtualBoxVM: dlopen("/usr/lib/virtualbox/VBoxRT.so",) failed: <NULL>

VirtualBoxVM: Tip! It may help to reinstall VirtualBox.
Why might it fail to dynamically load this library? The ldd command does not report any missing dependencies.
Reinstalling VirtualBox is something I can try again too, but I did this already several times without success, including a downgrade to the previous version.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VMs don't start after RHEL 8 update

Post by fth0 »

Kraut wrote:Why might it fail to dynamically load this library?
Two reasons come to mind: A mixture of different VirtualBox installations (e.g. Oracle vs RHEL versions, different 6.x.y versions), and ownership and access rights of the files and directory tree.

If you're proficient with strace, ... ;)
Kraut
Posts: 9
Joined: 29. Sep 2021, 01:31
Primary OS: Linux other
VBox Version: OSE other
Guest OSses: Linux and Windows 10
Location: Vancouver, BC

Re: VMs don't start after RHEL 8 update [SOLVED]

Post by Kraut »

fth0 wrote: Two reasons come to mind: A mixture of different VirtualBox installations (e.g. Oracle vs RHEL versions, different 6.x.y versions),
According to yum list, there are no RHEL versions of VirtualBox available, I only see the three versions from Oracle's virtualbox repository.

You hit the jackpot with this suggestion:
and ownership and access rights of the files and directory tree.

If you're proficient with strace, ... ;)
The last system call shown by strace before the error message was an lstat call for directory /usr. Problem was, some rogue installer had changed the owner of /usr, and VirtualBox was insisting that it be owned by root. This was a very frustrating episode. Why does VirtualBox not clearly tell us what it is unhappy about?

Thank you for all the suggestions, much appreciated!

Joachim
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: VMs don't start after RHEL 8 update

Post by fth0 »

Kraut wrote:The last system call shown by strace before the error message was an lstat call for directory /usr. Problem was, some rogue installer had changed the owner of /usr, and VirtualBox was insisting that it be owned by root. This was a very frustrating episode. Why does VirtualBox not clearly tell us what it is unhappy about?
Perhaps it's not VirtualBox that is unhappy (about /usr not being owned by root)? ;) I could imagine that the check is part of dlopen(), which is a function of your Linux host, and/or SELinux.
Kraut
Posts: 9
Joined: 29. Sep 2021, 01:31
Primary OS: Linux other
VBox Version: OSE other
Guest OSses: Linux and Windows 10
Location: Vancouver, BC

Re: VMs don't start after RHEL 8 update

Post by Kraut »

fth0 wrote: Perhaps it's not VirtualBox that is unhappy (about /usr not being owned by root)? ;) I could imagine that the check is part of dlopen(), which is a function of your Linux host, and/or SELinux.
I suppose that's possible, but it would surprise me if dlopen() did this. I consider it more likely that these checks are performed by initialization functions in the shared library that failed to load, e.g., constructors of static C++ objects, or explicitly registered load-time init functions.
Post Reply