CLI startvm headless hangs with Fedora and Redhat guests

[fth0]

Thank you, too, for sharing your insights!

I'll try and address a few open points (and I'll ignore those you've already solved yourself, kudos for that ):

---

I've read your ticket and your posts here, but I haven't understood if you have tested that the issues still occurred after successfully disabling Hyper-V. This is important, because the core parts of VirtualBox behave completely different depending on the backend (NEM vs. VT-x mode). So did you test using only the E-cores in VT-x mode?

---

I wonder about the message "CPUM: Matched host CPU INTEL 0x6/0xba/0x2 Intel_Atom_Unknown with CPU DB entry 'Intel Pentium N3530 2.16GHz' (INTEL 0x6/0x37/0x8 Intel_Atom_Silvermont)". This is clearly not the right CPU.

Like other type-2 hypervisors, VirtualBox provides only a selection of the host CPU's capabilities and MSRs to the guest. An internal database with only a few selected older CPU types is used to decide to provide even less of the host's CPU to the guest. In your case, this could be a wrong decision, but I haven't seen problems related to that yet. (That doesn't mean that it doesn't play a role here, though! )

---

Another observation which surprises me: while the VM are configured with 6 processors, only 4 E-cores are busy.

The TL;DR would be that you don't provide host CPU resources to the VM at all. What really happens is that the VirtualBoxVM or VBoxHeadless process creates some (OS) threads, one for each vCPU and several more, and then lets the host OS decide when and where to schedule those 42ish threads. Using the power configuration, you just give hints to the host OS scheduler, and VirtualBox doesn't know much (if anything) about that.

[lelegard]

I've read your ticket and your posts here, but I haven't understood if you have tested that the issues still occurred after successfully disabling Hyper-V. This is important, because the core parts of VirtualBox behave completely different depending on the backend (NEM vs. VT-x mode). So did you test using only the E-cores in VT-x mode?

Reading back the order of actions, I don't think so. I first solved the issue by forcing VBox on the P-cores and then fully eradicated Hyper-V. I will try to find some time to revert power throttling and try again.

Another observation which surprises me: while the VM are configured with 6 processors, only 4 E-cores are busy.

The TL;DR would be that you don't provide host CPU resources to the VM at all. What really happens is that the VirtualBoxVM or VBoxHeadless process creates some (OS) threads, one for each vCPU and several more, and then lets the host OS decide when and where to schedule those 42ish threads. Using the power configuration, you just give hints to the host OS scheduler, and VirtualBox doesn't know much (if anything) about that.

Yes, I know. The observation is that there are 8 E-cores in the CPU and only 4 of them were busy. The other 4 were idle.

On the other hand, after moving to the P-cores, I started my massive compilation job with "make -j10" in the guest and I see 6 logical P-cores suddenly going crazy. The CPU has 6 physical P-cores, meaning 12 logical P-cores. So, unless there is something special with the E-cores, the OS is capable of spreading 6 software threads on 6 cores. Being used to develop parallel applications, I have seen this for decades on all decent operating systems.

All tests were done on a "quiet" system, not doing anything else. All other cores are almost idle, just processing the background noise of the operating system. The system is not short of resources, far from that.

[lelegard]

I've read your ticket and your posts here, but I haven't understood if you have tested that the issues still occurred after successfully disabling Hyper-V. This is important, because the core parts of VirtualBox behave completely different depending on the backend (NEM vs. VT-x mode). So did you test using only the E-cores in VT-x mode?

I just did the test. My Hyper-V is still disabled and I re-enabled power throttling on the VBox executables. I don't see any boot problem. The Fedora guest boots normally in headless mode, using the E-cores (as seen on task manager).

So, as a conclusion, the problem occurs when Hyper-V is enabled (VBox in NEM mode) AND the VM runs on the E-cores. I will update the ticket with this information.

To solve the problem, it is consequently sufficient to either disable Hyper-V (completely, including the difficulties you mentioned) or disable power throttling on the VBox executables. Or both. In my case, I will keep both to make sure that all VM's run on the faster P-cores when started in headless mode. This PC is a build automation machine. Its only purpose is to build software on several VM's in automated scripts (and consequently with all VM's running in headless mode).

Two additional points:

1) With Hyper-V disabled, the VM (configured with 6 CPU's) uses 6 E-cores. I reported in a previous post that it used only 4 E-cores when running with Hyper-V enabled. With power throttling disabled, I saw that 6 P-cores were used. However, that was after disabling Hyper-V. I do not know what would happen with Hyper-V enabled and running on P-cores. Honestly, I don't feel like doing all those reconfigurations.

2) Something weird. After re-enabling power throttling on the VBox executables, I expected the VM to run on the E-cores. However, to my surprise, it ran on the P-cores. Then, I realized that I was on battery while all my other tests were done with power plugged. So, I plugged the power and restarted the VM. Then, it ran on the E-cores as expected. This is a bit surprising. Since the E-cores consume less power, I would have expected a larger use of E-cores on battery. But I see the contrary. A background process (the VM in headless mode) runs on P-cores when on battery and E-cores with power plugged. Weird...

[fth0]

Thanks for updating the ticket!

Regarding the weirdness, I have no good explanation either. Nobody really understands Windows.

[squig]

I know a solution has been found already, but I wanted to share my experience from tonight:

I commented on the original post but have since read through this post and the other and have reviewed the logs a bit, comparing them to my own. Generally speaking, I'm seeing the same issues from lelegards logs.

I noted in both of our logs, this was occurring re: Hyper-V:

Code: Select all

00:00:00.691819 NEM: Disable Hyper-V if you need X2APIC for your guests!
00:00:00.691930 NEM:
00:00:00.691930 NEM: NEMR3Init: Snail execution mode is active!
00:00:00.691930 NEM: Note! VirtualBox is not able to run at its full potential in this execution mode.
00:00:00.691931 NEM:       To see VirtualBox run at max speed you need to disable all Windows features
00:00:00.691931 NEM:       making use of Hyper-V.  That is a moving target, so google how and carefully
00:00:00.691931 NEM:       consider the consequences of disabling these features.
00:00:00.691931 NEM:
00:00:00.691942 CPUM: No hardware-virtualization capability detected

On the previous ticket, I posted about how simply adjusting the resolution fixes the issue. This does seem to work, although not as reliably as I initially thought. It still seems to fix the issue when the screen does go black for me, but it doesn't seem to always stop it from re-appearing. That said, I haven't had to adjust it in the devstation I'm working from, so take it with a grain of salt.

I'm also using an Intel alder lake processor, more info below:

Code: Select all

OS Name	Microsoft Windows 11 Enterprise
Version	10.0.22621 Build 22621
Processor	13th Gen Intel(R) Core(TM) i5-1340P, 1900 Mhz, 12 Core(s), 16 Logical Processor(s)
Installed Physical Memory (RAM)	16.0 GB

I haven't previously enabled Hyper-V on this workstation (it's fairly recently provisioned, I'm the only user), so Hyper-V should be in the default state on this machine. Also, this being the default outcome for booting into Fedora on VBox using a 13th Gen Intel host is obviously not great.

I found two potential solutions for the Hyper-V problem in an attempt to find something more scalable.

1) Changing the paravirtualization to KVM instead of Default or Hyper-V

On boot, I'm not seeing any of the NEM lines mentioned above.


2) Similarly by enabling the Nested VT-x/AMD-V toggle and leaving the paravirtualization on default

There is a very easy to find stackoverflow article that mentions how to easily enable the toggle. I'm fairly new to working with
VBox, but I'm betting this is most likely something that can be set in a config file.

After enabling the toggle for the Nested VT-x/AMD-V for the processor, like the above, the NEM lines from the initial logs weren't
present

I'm not really feeling super confident about these solutions yet, but just wanted to mention my findings in-case someone else can roll with it.

[scottgus1]

Thanks for the information!

One thing I don't understand is this:

Changing the paravirtualization to KVM instead of Default or Hyper-V

On boot, I'm not seeing any of the NEM lines mentioned above.

For a Fedora Linux VM, paravirtualization on Default would actually be KVM and never Hyper-V. Paravirtualization is a communication channel that the VM OS understands to talk to the hypervisor with, and does not influence the host Hyper-V setting in any way. Virtualbox sets the Default setting to whatever VM OS type the Create Virtual Machine wizard is set for. Modern Linux always gets KVM, modern Windows always gets Hyper-V.

We would need to see two logs, one where Paravirtualization was set to default and where the lines you quoted are present, and another where the Paravirtualization was set to KVM and where the lines you posted from the log are not present. There are other log lines that may fill in the gaps.

Similarly by enabling the Nested VT-x/AMD-V toggle and leaving the paravirtualization on default

There is a very easy to find stackoverflow article that mentions how to easily enable the toggle. I'm fairly new to working with
VBox, but I'm betting this is most likely something that can be set in a config file.

There an easier way: Check the Nested VT-x/AMD-V checkbox in the VM's System settings, Processor tab. The checkbox will remain checked and recorded in the VM's settings until you uncheck it.

FWIW that setting only passes hardware virtualization into the VM environment so another layer of Virtualbox can use it in Nested Virtualization. It also does not influence host Hyper-V. We would also need to see two logs when the log lines are present or not only by changing the checkbox.

Without the logs we'd have to say there was something else going on that isn't reported yet which is ticking host Hyper-V. Neither of those settings have influenced host Hyper-V in the past.

[fth0]

To clarify one thing:

When we're talking about Hyper-V being active on the host and VirtualBox in NEM, we're not talking about the Hyper-V hypervisor running inside the Windows host OS alongside VirtualBox, but we're talking about the Hyper-V hypervisor running below the Windows host OS.

The Windows host OS itself is running in the Hyper-V Root Partition then, and since the Hyper-V hypervisor is being started before the Windows kernel, no VirtualBox configuration has any influence on that.