Virtualbox 6.1.46 r158378 on FreeBSD Host and nested virtualization

Discussions related to using VirtualBox on other (unsupported) hosts like FreeBSD.
Post Reply
Andreaf74
Posts: 4
Joined: 14. Sep 2023, 11:13

Virtualbox 6.1.46 r158378 on FreeBSD Host and nested virtualization

Post by Andreaf74 »

Hello Everyone,

I'm trying to use Nested virtualization on VirtualBOX 6.1.46 on a FreeBSD host.

Apparently it seems to be supported, and when booted the guest OS (RH 8.8 in this case) shows the SVM support ( I'm on AMD 5950x)

Code: Select all

cat /proc/cpuinfo  | grep svm
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch cpb vmmcall fsgsbase avx2 invpcid rdseed clflushopt arat nrip_save flushbyasid decodeassists
But when I try to boot a VM inside the Guest, it panics with this error:

Code: Select all

[    0.547606] Call Trace:
[    0.548504]  __module_text_address+0xe/0x70
[    0.550094] BUG: unable to handle kernel paging request at ffffb8bb409a39c0
[    0.552056] PGD 113b067 P4D 113b067 PUD 113c067 PMD 24a0067 PTE 0
[    0.553665] Oops: 0000 [#4] SMP NOPTI
[    0.554739] CPU: 0 PID: 1 Comm: init Not tainted 4.18.0-477.10.1.el8_8.x86_64 #1
[    0.556707] Hardware name: Red Hat KVM, BIOS 1.16.0-3.module+el8.8.0+16781+9f4724c2 04/01/2014
[    0.559017] RIP: 0010:__module_address+0x6e/0xf0
[    0.560360] Code: 02 44 89 c1 83 e1 01 48 8d 41 01 48 8b 04 c5 c0 7a e0 b8 48 85 c0 74 de 48 8d 14 09 48 01 ca 48 c1 e2 03 48 f7 da 48 8d 0c 10 <4c> 8b 49 e0 4c 39 cf 73 b1 48 8b 40 10 48 85 c0 75 ea 41 8b 12 41
[    0.565503] RSP: 0018:ffffb8bb4031f568 EFLAGS: 00010046
[    0.566894] RAX: ffffb8bb409a39e0 RBX: ffffffffc019e000 RCX: ffffb8bb409a39e0
[    0.568879] RDX: 0000000000000000 RSI: ffff8a0fc1880000 RDI: ffffffffc019e000
[    0.570804] RBP: 0000000000000000 R08: 000000000000000a R09: ffffffffb9406560
[    0.572754] R10: ffffffffb8e07ac0 R11: 0000000000000003 R12: 0000000000000000
[    0.574716] R13: ffff8a0fc1880000 R14: ffffffffb8725037 R15: 0000000000000000
[    0.576789] FS:  000000000262a3c0(0000) GS:ffff8a100e400000(0000) knlGS:0000000000000000
[    0.579220] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.580726] CR2: ffffb8bb409a39c0 CR3: 0000000047e0a000 CR4: 00000000000506f0
I've found very little information about this feature and it's not clear if this is a problem related to VirtualBox in general or specific to the FreeBSD port.

Does anyone has ever successfully tried nested virtualization on VirtualBOX ?

There'are known constrains about supported nested VM ( kernel version... cpu number...)?

Thanks in advance,

Andrea
scottgus1
Site Moderator
Posts: 20317
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Virtualbox 6.1.46 r158378 on FreeBSD Host and nested virtualization

Post by scottgus1 »

Please start each VM from full normal shutdown, not save-state. Run until you see the problem happen, then shut down the VM from within the VM's OS if possible. If not possible, close the Virtualbox window for the VM with the Power Off option set.

Right-click each VM in the main Virtualbox window's VM list, choose Show in Explorer/Finder/File Manager. In the "Logs" subfolder, zip the VM's "vbox.log", and post the zip file, using the forum's Attachments tab. (Configure your host OS to show all extensions so you can find the "vbox.log", not "vbox.log.1", etc.)
Andreaf74
Posts: 4
Joined: 14. Sep 2023, 11:13

Re: Virtualbox 6.1.46 r158378 on FreeBSD Host and nested virtualization

Post by Andreaf74 »

Hello and thanks for replying,

attached you will find the VBox.log of one of the VM showing the problem

Please note the VM is part of an Openshift cluster running on CoreOS, so operating on the guest VM is a little tricky.

This cluster would run flawlessy except for the kubevirt component that fails when trying to lunch a VM.

I tried also before with a simpler setup, failing the same way.

Regards,

Andrea
Attachments
vboxlog.zip
(19.94 KiB) Downloaded 10 times
fth0
Volunteer
Posts: 5387
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Virtualbox 6.1.46 r158378 on FreeBSD Host and nested virtualization

Post by fth0 »

Unfortunately, the VBox.log file is missing the statistics at the end. Did you close the VM window and/or error message windows before copying the log file?

In general, I've seen similar issues reported by other users from time to time, where memory management errors occurred with at least 32 GB provided to a VM, with and without nested virtualization. For a test, I'd suggest to only provide 8 GB to the first guest and 4 GB to the nested guest, and to scale up later when everything works.
Andreaf74
Posts: 4
Joined: 14. Sep 2023, 11:13

Re: Virtualbox 6.1.46 r158378 on FreeBSD Host and nested virtualization

Post by Andreaf74 »

Hello,

I've tried with a simple setup, using a virtual vm based on RHEL 8.8 as guest and an rhel8.8 as guest's guest.
The error is always the same:

? [ 1.051010] BUG: unable to handle kernel paging request at ffffa5b3407b79c0
? [ 1.053152] PGD 113b067 P4D 113b067 PUD 113c067 PMD 22ae067 PTE 0
? [ 1.055016] Oops: 0000 [#20] SMP NOPTI
? [ 1.056149] CPU: 0 PID: 1 Comm: init Not tainted 4.18.0-477.10.1.el8_8.x86_64 #1
? [ 1.058383] Hardware name: Red Hat KVM, BIOS 1.16.0-3.module+el8.8.0+16781+9f4724c2 04/01/2014
? [ 1.060969] RIP: 0010:__module_address+0x6e/0xf0
? [ 1.062405] Code: 02 44 89 c1 83 e1 01 48 8d 41 01 48 8b 04 c5 c0 7a 60 ac 48 85 c0 74 de 48 8d 14 09 48 01 ca 48 c1 e2 03 48 f7 da 48 8d 0c 10 <4c> 8b 49 e0 4c 39 cf 73 b1 48 8b 40 10 48 85 c0 75 ea 41 8b 12 41
? [ 1.068086] RSP: 0018:ffffa5b340194768 EFLAGS: 00010046
? [ 1.069639] RAX: ffffa5b3407b79e0 RBX: ffffffffc0213000 RCX: ffffa5b3407b79e0
? [ 1.071700] RDX: 0000000000000000 RSI: ffff95c3c12f8000 RDI: ffffffffc0213000
? [ 1.073765] RBP: 0000000000000000 R08: 000000000000000a R09: ffffffffacc06560
? [ 1.076151] R10: ffffffffac607ac0 R11: 0000000000000003 R12: 0000000000000000
? [ 1.078301] R13: ffff95c3c12f8000 R14: ffffffffabf25037 R15: 0000000000000000
? [ 1.080457] FS: 00000000017313c0(0000) GS:ffff95c3fe800000(0000) knlGS:0000000000000000
? [ 1.082877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
? [ 1.084885] CR2: ffffa5b3407b79c0 CR3: 000000001e00a000 CR4: 00000000000506f0
? [ 1.087060] Call Trace:
? [ 1.087809] __module_text_address+0xe/0x70

Attached will find a zip file containing VBox.log e KVM.log.

THanks,

Andrea
Attachments
logs.zip
(46.27 KiB) Downloaded 5 times
fth0
Volunteer
Posts: 5387
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Virtualbox 6.1.46 r158378 on FreeBSD Host and nested virtualization

Post by fth0 »

The VBox.log file shows the POV of VirtualBox on the FreeBSD host. Does the KVM.log file show the POV of the first/outer guest or the second/inner guest (I assume it's the latter)? What does the other guest think about what happened?
Andreaf74
Posts: 4
Joined: 14. Sep 2023, 11:13

Re: Virtualbox 6.1.46 r158378 on FreeBSD Host and nested virtualization

Post by Andreaf74 »

Hello,

sorry for the late reply but I've been very busy at work past month..

to answer your question, the first outer guest kvm shows no error, and it does believe machine is running while is freezed in the kernel loop shown before.

I've upgraded VIrtualBox to the lastest 6.1.48 r159471, but not signs of improvement.


THanks,

ANdrea
Post Reply