Suddenly, none of my VMs will start, including after a restore from backup

[ddw]

Hi,

All of a sudden, none of my several snapshots of different VMs will fully boot up. They all hang on the Ubuntu startup screen.

After performing a full clonezilla backup, I restored from a full backup of the host machine from before the problem appeared, and the none of the dang VM snapshots will start after the restore, either. (!). I confirmed the restore was successful.

I can boot the VMs in recovery mode. With snapshotting along the way, I've apt updated and upgraded and made sure the virtualbox kernel module thing got built successfully by the post-kernel etc script, and all that stuff seems OK as far as it looked successful.

Hitting ESC during boot, here's the last stuff that shows up on the console before the hang:

vboxadd.service
[ OK ] Finished vboxadd.service.
Starting GNONE Display Manager...
Started vboxadd-service.service...
[ OK ] Started vboxadd-service.service.
vboxadd-service.service
[ OK ] Started GNOME Display Manager.
gdm.service
home-user-thing\x2dshare.mount

Sometimes it hangs at vboxadd-service.service. Almost always the last message is the share mount. Sending a shutdown signal works to shut the thing down. Waiting a long time doesn't help.

Both host and guest kernels are 5.15.0-87-generic. Both are running Ubuntu 20.04.1.
Virtualbox is 6.1.38-Ubuntu r153438

Host is Intel 12th-gen with 16 GB RAM with plenty of SSD space. VM has plenty of space. x864-64.

VM base memory 4096 MB, Acceleration: VT-x/AMD-V, Nested Paging, KVM, Paravirtualization
1 or 2 CPUs
Video memory 16 MB
Graphics Controller: VMSVGA
Remote Desktop Server: Disabled
Recording: Disabled
NAT ethernet adapter

Looking at kern.log in a VM there are a lot of messages that are like "failed adding user 'vboxadd', data deleted" and many along the lines of "vboxsf: Unknown symbol VBoxGuest_RTLogRelGetDefaultInstanceEx (err -2)" and the like. Clearly not good. But it also says "Successfully loaded version 6.1.38 r153438".

I've attached log.zip containing the log file created from the virtualbox manager.

I'm new to needing to troubleshoot this. I'm reasonably competent at Linux. I've read the docs on how to post and what to provide, etc, but this is my first post, so apologies if I've screwed something up.

[fth0]

The boxy-brown-clone-2023-10-29-10-28-07.log file doesn't indicate any real error to me, but I can perhaps give a few random pointers to follow:

1. While the VirtualBox BIOS is running, a USB fingerprint reader (from Elan Microelectronics) is attached to the VM. This is highly unusual, because USB devices cannot usually be shared between the host and the guest. If you aren't using a special setup with a separate fingerprint reader, this would be worth investigating.

2. After booting from the virtual hard disk image, the video resolution changes to 800x600, which is normal for Linux guests. But there is no subsequent resolution change to the hinted 1390x797 later on. Did perhaps something related to X11 or XWayland change on your host?

3. If the VM has been working after you took the full backup, restoring form the backup restored many things, but not the current calendar date.

[ddw]

Thank you for looking and replying.

Turning off chronyd on the host and setting the date back worked! Thank you, thank you, thank you. Now the question is why. But still.

As to the fingerprint reader, I do not have a special setup for that. I don't see anything in the GUI about it. Is this something I need to look at using VBoxManage and/or is there some config file I should look at?

[ddw]

Solved all the way! Thank you, @ftho. After some number of months a password change was required, but the CIS security scripts broke the password-change process. I had figured this out and fixed it on the host but it didn't seem like the guest needed it since it didn't require a password to login.

Oh but I was wrong.

By turning off chronyd and backdating the system clock I got the VM to run. Setting the clock forward to the current time, it wouldn't. Setting the clock back, starting up, then moving the clock forward by starting chronyd again on the host, and a sudo in the VM demanded a password change and then it was evident it had the same problem. commenting out the pam_pwquality dot so line from /etc/pam.d/common-password and updating the password allows the VM to run again with the system clock set to the current time.

I'm sure there is a better way to solve the problem, like putting the libpam_pwquality dot so file back and disabling password change forcing, but it is good to be back and have the problem solved.

[fth0]

Interesting story, thanks for reporting back!

As to the fingerprint reader, I do not have a special setup for that. I don't see anything in the GUI about it. Is this something I need to look at using VBoxManage and/or is there some config file I should look at?

You could check in the VirtualBox Manager if you have set up a USB filter in the VM configuration. Additionally, you could have a look at the VBoxSVC.log* files in the VirtualBox global configuration folder (~/.config/VirtualBox) and check for USB device 04f3:0c7f.