Virtualbox 6.1.46 r158378 on FreeBSD Host and nested virtualization

[Andreaf74]

Hello Everyone,

I'm trying to use Nested virtualization on VirtualBOX 6.1.46 on a FreeBSD host.

Apparently it seems to be supported, and when booted the guest OS (RH 8.8 in this case) shows the SVM support ( I'm on AMD 5950x)

Code: Select all

cat /proc/cpuinfo  | grep svm
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch cpb vmmcall fsgsbase avx2 invpcid rdseed clflushopt arat nrip_save flushbyasid decodeassists

But when I try to boot a VM inside the Guest, it panics with this error:

Code: Select all

[    0.547606] Call Trace:
[    0.548504]  __module_text_address+0xe/0x70
[    0.550094] BUG: unable to handle kernel paging request at ffffb8bb409a39c0
[    0.552056] PGD 113b067 P4D 113b067 PUD 113c067 PMD 24a0067 PTE 0
[    0.553665] Oops: 0000 [#4] SMP NOPTI
[    0.554739] CPU: 0 PID: 1 Comm: init Not tainted 4.18.0-477.10.1.el8_8.x86_64 #1
[    0.556707] Hardware name: Red Hat KVM, BIOS 1.16.0-3.module+el8.8.0+16781+9f4724c2 04/01/2014
[    0.559017] RIP: 0010:__module_address+0x6e/0xf0
[    0.560360] Code: 02 44 89 c1 83 e1 01 48 8d 41 01 48 8b 04 c5 c0 7a e0 b8 48 85 c0 74 de 48 8d 14 09 48 01 ca 48 c1 e2 03 48 f7 da 48 8d 0c 10 <4c> 8b 49 e0 4c 39 cf 73 b1 48 8b 40 10 48 85 c0 75 ea 41 8b 12 41
[    0.565503] RSP: 0018:ffffb8bb4031f568 EFLAGS: 00010046
[    0.566894] RAX: ffffb8bb409a39e0 RBX: ffffffffc019e000 RCX: ffffb8bb409a39e0
[    0.568879] RDX: 0000000000000000 RSI: ffff8a0fc1880000 RDI: ffffffffc019e000
[    0.570804] RBP: 0000000000000000 R08: 000000000000000a R09: ffffffffb9406560
[    0.572754] R10: ffffffffb8e07ac0 R11: 0000000000000003 R12: 0000000000000000
[    0.574716] R13: ffff8a0fc1880000 R14: ffffffffb8725037 R15: 0000000000000000
[    0.576789] FS:  000000000262a3c0(0000) GS:ffff8a100e400000(0000) knlGS:0000000000000000
[    0.579220] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.580726] CR2: ffffb8bb409a39c0 CR3: 0000000047e0a000 CR4: 00000000000506f0

I've found very little information about this feature and it's not clear if this is a problem related to VirtualBox in general or specific to the FreeBSD port.

Does anyone has ever successfully tried nested virtualization on VirtualBOX ?

There'are known constrains about supported nested VM ( kernel version... cpu number...)?

Thanks in advance,

Andrea

[scottgus1]

Please start each VM from full normal shutdown, not save-state. Run until you see the problem happen, then shut down the VM from within the VM's OS if possible. If not possible, close the Virtualbox window for the VM with the Power Off option set.

Right-click each VM in the main Virtualbox window's VM list, choose Show in Explorer/Finder/File Manager. In the "Logs" subfolder, zip the VM's "vbox.log", and post the zip file, using the forum's Attachments tab. (Configure your host OS to show all extensions so you can find the "vbox.log", not "vbox.log.1", etc.)

[Andreaf74]

Hello and thanks for replying,

attached you will find the VBox.log of one of the VM showing the problem

Please note the VM is part of an Openshift cluster running on CoreOS, so operating on the guest VM is a little tricky.

This cluster would run flawlessy except for the kubevirt component that fails when trying to lunch a VM.

I tried also before with a simpler setup, failing the same way.

Regards,

Andrea

[fth0]

Unfortunately, the VBox.log file is missing the statistics at the end. Did you close the VM window and/or error message windows before copying the log file?

In general, I've seen similar issues reported by other users from time to time, where memory management errors occurred with at least 32 GB provided to a VM, with and without nested virtualization. For a test, I'd suggest to only provide 8 GB to the first guest and 4 GB to the nested guest, and to scale up later when everything works.

[Andreaf74]

Hello,

I've tried with a simple setup, using a virtual vm based on RHEL 8.8 as guest and an rhel8.8 as guest's guest.
The error is always the same:

? [ 1.051010] BUG: unable to handle kernel paging request at ffffa5b3407b79c0
? [ 1.053152] PGD 113b067 P4D 113b067 PUD 113c067 PMD 22ae067 PTE 0
? [ 1.055016] Oops: 0000 [#20] SMP NOPTI
? [ 1.056149] CPU: 0 PID: 1 Comm: init Not tainted 4.18.0-477.10.1.el8_8.x86_64 #1
? [ 1.058383] Hardware name: Red Hat KVM, BIOS 1.16.0-3.module+el8.8.0+16781+9f4724c2 04/01/2014
? [ 1.060969] RIP: 0010:__module_address+0x6e/0xf0
? [ 1.062405] Code: 02 44 89 c1 83 e1 01 48 8d 41 01 48 8b 04 c5 c0 7a 60 ac 48 85 c0 74 de 48 8d 14 09 48 01 ca 48 c1 e2 03 48 f7 da 48 8d 0c 10 <4c> 8b 49 e0 4c 39 cf 73 b1 48 8b 40 10 48 85 c0 75 ea 41 8b 12 41
? [ 1.068086] RSP: 0018:ffffa5b340194768 EFLAGS: 00010046
? [ 1.069639] RAX: ffffa5b3407b79e0 RBX: ffffffffc0213000 RCX: ffffa5b3407b79e0
? [ 1.071700] RDX: 0000000000000000 RSI: ffff95c3c12f8000 RDI: ffffffffc0213000
? [ 1.073765] RBP: 0000000000000000 R08: 000000000000000a R09: ffffffffacc06560
? [ 1.076151] R10: ffffffffac607ac0 R11: 0000000000000003 R12: 0000000000000000
? [ 1.078301] R13: ffff95c3c12f8000 R14: ffffffffabf25037 R15: 0000000000000000
? [ 1.080457] FS: 00000000017313c0(0000) GS:ffff95c3fe800000(0000) knlGS:0000000000000000
? [ 1.082877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
? [ 1.084885] CR2: ffffa5b3407b79c0 CR3: 000000001e00a000 CR4: 00000000000506f0
? [ 1.087060] Call Trace:
? [ 1.087809] __module_text_address+0xe/0x70

Attached will find a zip file containing VBox.log e KVM.log.

THanks,

Andrea

[fth0]

The VBox.log file shows the POV of VirtualBox on the FreeBSD host. Does the KVM.log file show the POV of the first/outer guest or the second/inner guest (I assume it's the latter)? What does the other guest think about what happened?

[Andreaf74]

Hello,

sorry for the late reply but I've been very busy at work past month..

to answer your question, the first outer guest kvm shows no error, and it does believe machine is running while is freezed in the kernel loop shown before.

I've upgraded VIrtualBox to the lastest 6.1.48 r159471, but not signs of improvement.


THanks,

ANdrea