[Solved] All of a sudden, all my Arch-based VM's fail to launch (boot)

Discussions related to using VirtualBox on Linux hosts.
Post Reply
Jack21
Posts: 21
Joined: 28. Dec 2018, 13:39
Primary OS: Linux other
VBox Version: OSE Debian
Guest OSses: Win 7, Win 11, Arch Linux, Debian, Slackware, Arco
Location: Tarragona. Spain

[Solved] All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by Jack21 »

Debian_boot_success.log.zip
Debian VM boot success
(39.52 KiB) Downloaded 4 times
Arch_fail_to_boot.log.zip
Arch VM boot fail
(31.06 KiB) Downloaded 23 times
Hello all.

This is maddening. A few days ago after updating my various VM's (some based on Arch, some on Debian), upon opening VB later to launch a given VM, it gets to GRUB, boots, and then stops about 1-2 seconds into the process and freezes. At that point I am obligated to go to File > Power > Force to Close the close out the failed VM.

I'm at a loss to understand here. This is not happening with my Debian or Debian-based VM's, only Arch based. On my desktop computer I have several different Linux distros installed to various dedicated SSD's. Two of these distros have VB installed: Fedora 36 (cinnamon) and Linux Mint 20.3 (cinnamon). At first I noticed this failure of Arch-based VM's to open on my Fedora system. I chalked it up to a Fedora issue, then uninstalled VB 6.1.34 and installed 6.1.36 thinking that would solve it, but no joy. A day or so later I noticed the same thing happening on my VB installation on Linux Mint. So I uninstalled VB 6.1.34 and again, installed version 6.1.36 but again no joy.

There seems to be one or two things going on here: A problem between VB and the Linux kernel on the HOST machine, but also a problem with the Linux kernel on the GUEST machine as it relates the host. Tbh I simply don't know what to do here.

I'm attaching two log files to this post. One is the zipped log of the failed Arch VM, and the other is the zipped log of a successful boot of a Debian VM. Can someone look at these and maybe clue me in as to what may be happening? As a side note, the fellows over at Arch forums gave me the usual snotty RTFM response, which of course was no help at all because I have read the Arch wiki and scoured the Internet in an attempt to solve this problem.

Computer specs: Home built, Asus VII Crosshair x470, AMD Ryzen 7 3700x, AMD Radeon 580 GPU, 16MB DDR4
Last edited by Jack21 on 5. Aug 2022, 11:44, edited 1 time in total.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by fth0 »

The biggest difference that I noticed is the 3D Acceleration setting. Please try the other combinations (Debian without and Arch Linux with 3D acceleration).
Jack21
Posts: 21
Joined: 28. Dec 2018, 13:39
Primary OS: Linux other
VBox Version: OSE Debian
Guest OSses: Win 7, Win 11, Arch Linux, Debian, Slackware, Arco
Location: Tarragona. Spain

Re: All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by Jack21 »

fth0 wrote:The biggest difference that I noticed is the 3D Acceleration setting. Please try the other combinations (Debian without and Arch Linux with 3D acceleration).
Yeah that's not it. Disabling 3D Accel on the Debian VM just produces Cinnamon DE complaining about "software rendering mode" (expected behavior). The Arch (this is plain vanilla btw, a basic installation with NO DE) VM still won't boot.
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by scottgus1 »

Jack21 wrote:after updating my various VM's (some based on Arch, some on Debian), upon opening VB later to launch a given VM, it gets to GRUB, boots, and then stops about 1-2 seconds into the process and freezes.
What happens in the VM stays in the VM. OS updates inside the VM can hose the OS or move the OS out of compliance with Virtualbox, but the OS's help features and channels must be used to fix the OS, same as if the problem happened on a real PC.

The "failing" log does not look to me to show a failure. The VM 'hardware' seems to boot fine.

Please try making a new Arch VM, using the Arch ISO you originally used. If it boots, then the updates inside the existing VM are giving trouble.
Jack21
Posts: 21
Joined: 28. Dec 2018, 13:39
Primary OS: Linux other
VBox Version: OSE Debian
Guest OSses: Win 7, Win 11, Arch Linux, Debian, Slackware, Arco
Location: Tarragona. Spain

Re: All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by Jack21 »

scottgus1 wrote:
Jack21 wrote:after updating my various VM's (some based on Arch, some on Debian), upon opening VB later to launch a given VM, it gets to GRUB, boots, and then stops about 1-2 seconds into the process and freezes.
What happens in the VM stays in the VM. OS updates inside the VM can hose the OS or move the OS out of compliance with Virtualbox, but the OS's help features and channels must be used to fix the OS, same as if the problem happened on a real PC.

The "failing" log does not look to me to show a failure. The VM 'hardware' seems to boot fine.

Please try making a new Arch VM, using the Arch ISO you originally used. If it boots, then the updates inside the existing VM are giving trouble.
Exactly. Something in the last kernel update has changed, or something in a recent Arch package update has changed. Yesterday I took the latest Arch iso, and installed entirely by hand a completely vanilla installation sans DE. During the install I added both the linux and linux-lts kernels. After all was said and done with installation finished I shut down the VM with the "shutdown now" command at the Arch root prompt. Upon entering VB again and choosing to start my just built Arch VM, it once again halts after passing the Grub boot loader (same behavior choosing either kernel at Grub).

For "fun" I installed a VB daily build on my Fedora 36 install. Made no difference, same problem. I'm simply at a loss for what to do other than file a bug report with Virtualbox, or should I file one with the Arch devs? Don't know which way to proceed from here.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by fth0 »

FWIW, Debian read ~600 MB from the virtual hard disk, whereas Arch Linux read ~20 MB from the virtual hard disk.

I may have found something:
Arch_fail_to_boot.log wrote:
00:00:00.806585 SVM Feature Identification (leaf A):
00:00:00.806552   Mnemonic - Description                                  = guest (host)
00:00:00.806615   IBPB - Supports the IBPB command in IA32_PRED_CMD       = 0 (1)
[...[
00:00:04.761738 IEM: wrmsr(0x49,0x0`00000001) -> #GP(0)
Although VirtualBox did not provide IBPB to the VM, Arch Linux tried to use it, while Debian didn't. I've found an Arch Linux bug report which seems to fit the bill (Modifications in 5.18.15 addressing Retbleed may prevent booting on AMD CPUs). For a test, please try if adding spectre_v2=off to the kernel parameters helps, and report back.
Jack21
Posts: 21
Joined: 28. Dec 2018, 13:39
Primary OS: Linux other
VBox Version: OSE Debian
Guest OSses: Win 7, Win 11, Arch Linux, Debian, Slackware, Arco
Location: Tarragona. Spain

Re: All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by Jack21 »

fth0 wrote:FWIW, Debian read ~600 MB from the virtual hard disk, whereas Arch Linux read ~20 MB from the virtual hard disk.

I may have found something:
Arch_fail_to_boot.log wrote:
00:00:00.806585 SVM Feature Identification (leaf A):
00:00:00.806552   Mnemonic - Description                                  = guest (host)
00:00:00.806615   IBPB - Supports the IBPB command in IA32_PRED_CMD       = 0 (1)
[...[
00:00:04.761738 IEM: wrmsr(0x49,0x0`00000001) -> #GP(0)
Although VirtualBox did not provide IBPB to the VM, Arch Linux tried to use it, while Debian didn't. I've found an Arch Linux bug report which seems to fit the bill (Modifications in 5.18.15 addressing Retbleed may prevent booting on AMD CPUs). For a test, please try if adding spectre_v2=off to the kernel parameters helps, and report back.
BINGO! That did it. Have added the spectre_v2=off kernel parameter to all my Arch VM's (even the one using rEFInd) and all functions as before. Thank you SO MUCH for that lead. When this all started a few days ago I went through the log files with a fine tooth comb, but clearly my experience level (after nearly six freakin years of using both Arch and Debian linux) just wasn't high enough to catch this. Just wow. Thank you again! :D
Jack21
Posts: 21
Joined: 28. Dec 2018, 13:39
Primary OS: Linux other
VBox Version: OSE Debian
Guest OSses: Win 7, Win 11, Arch Linux, Debian, Slackware, Arco
Location: Tarragona. Spain

Re: SOLVED All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by Jack21 »

As a side thought, why am I able to boot my Fedora and Arch bare metal installations (not VM's, but actual installs to dedicated SSD's) without adding this kernel parameter to my Grub? Just slightly confused here.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: [Solved] All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by fth0 »

In the log messages quoted, you can see that the host's physical CPU provides IBPB, it's just that VirtualBox's virtual CPU does not provide it to the VM. And Arch Linux kernel 5.18.15 has a bug not verifying the availability of IBPB before enabling it.

Can you confirm that for Arch Linux kernel 5.18.16, the spectre_v2=off kernel parameter is no longer necessary?
Jack21
Posts: 21
Joined: 28. Dec 2018, 13:39
Primary OS: Linux other
VBox Version: OSE Debian
Guest OSses: Win 7, Win 11, Arch Linux, Debian, Slackware, Arco
Location: Tarragona. Spain

Re: [Solved] All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by Jack21 »

fth0 wrote:In the log messages quoted, you can see that the host's physical CPU provides IBPB, it's just that VirtualBox's virtual CPU does not provide it to the VM. And Arch Linux kernel 5.18.15 has a bug not verifying the availability of IBPB before enabling it.

Can you confirm that for Arch Linux kernel 5.18.16, the spectre_v2=off kernel parameter is no longer necessary?
Well ok, get this. I removed the spectre kernel parameter from my vanilla Arch VM, and it boots up under kernel 5.18.16 ok. But when choosing kernel 5.15.58-2-lts to boot from, it fails as before. Apparently the kernel devs haven't gotten around to fixing the regression with ltx?
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: [Solved] All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by fth0 »

Should be answered in the Arch Linux ticket.
Jack21
Posts: 21
Joined: 28. Dec 2018, 13:39
Primary OS: Linux other
VBox Version: OSE Debian
Guest OSses: Win 7, Win 11, Arch Linux, Debian, Slackware, Arco
Location: Tarragona. Spain

Re: [Solved] All of a sudden, all my Arch-based VM's fail to launch (boot)

Post by Jack21 »

Update: It appears as of lts kernel 5.15.59-2-lts that this issue is solved as well.
Post Reply