Presenting the guest two different nvme controllers

Discussions about using Linux guests in VirtualBox.
Post Reply
bwoebi
Posts: 3
Joined: 25. Aug 2020, 12:31

Presenting the guest two different nvme controllers

Post by bwoebi »

Setup: Standard Linux Host & Guest, guest booted to a live Linux system to inspect the disks; vbox version 6.1.10.

I'm currently trying to emulate the setup of our servers with virtualbox, which is nearly working, but only for disks which aren't pcie nvmes (i.e. SATA works without issues). Primarly, we want to test some pretty complicated disk setup workflows (involving reboots, partprobes etc., which make meaningful testing by manually faking the drive nodes via mknod defacto impossible).
It's possible to add two nvmes, but only on the same controller. This leads to the linux guest recognizing them as /dev/nvme0n1 and /dev/nvme0n2. Our servers all have two different nvme controllers with each one nvme, which then appear ass /dev/nvme0n1 and /dev/nvme1n1.

Is there a way to make the linux host recognize both disks as /dev/nvme0n1 and /dev/nvme1n1?

My research up to now:
You can add a second nvme controller if you set the chipset to ich9; the default pxii3 chipset refuses to add multiple controllers of the same family. Did that:

Code: Select all

vboxmanage modifyvm $NAME --chipset ich9
vboxmanage storagectl $NAME --name "NVMe Controller 1" --add pcie --controller NVMe --portcount 1 --bootable on
vboxmanage storageattach $NAME --storagectl "NVMe Controller 1" --device 0 --port 0 --type hdd --medium ${VHD1} --nonrotational on
vboxmanage storagectl $NAME --name "NVMe Controller 2" --add pcie --controller NVMe --portcount 1
vboxmanage storageattach $NAME --storagectl "NVMe Controller 2" --device 0 --port 0 --type hdd --medium ${VHD2} --nonrotational on
But then Linux (quite recent 5.4.32 kernel) does not like that and complains (nvme0 works perfectly):

Code: Select all

[Mon Aug 24 21:35:37 2020] nvme nvme1: I/O 28 QID 0 timeout, completion polled
[Mon Aug 24 21:35:37 2020] nvme nvme1: Duplicate cntlid 0 with nvme0, rejecting
[Mon Aug 24 21:35:37 2020] nvme nvme1: Removing after probe failure status: -22
I'm absolutely stuck at what these error messages even mean.
Maybe that's the wrong approach, maybe a bug (I've seen ich9 being described as experimental) - I have no idea what to try next.
Attachments
VBox.log
(72.88 KiB) Downloaded 10 times
bwoebi
Posts: 3
Joined: 25. Aug 2020, 12:31

Re: Presenting the guest two different nvme controllers

Post by bwoebi »

As nobody seems to have any suggestion (maybe 2 days of wait is too short? :-D) - where shall I try asking next?
Shall I go to the bug tracker? Or is there a way to connect to the VirtualBox devs for that issue?
scottgus1
Site Moderator
Posts: 20945
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Presenting the guest two different nvme controllers

Post by scottgus1 »

I got an Ubuntu 2004 ISO, set up an ICH9 guest with two NVMe controllers and a disk on each controller.

'uname -a' reports the guest OS as Ubuntu 5.4.0-26-generic.

Here is the configuration, from the guest's .vbox file:
<HardDisks>
<HardDisk uuid="{3016646e-0a45-40a1-9fe1-5d784970260a}" location="NewVirtualDisk1.vdi" format="VDI" type="Normal"/>
<HardDisk uuid="{48b63885-44f2-4bcc-a6e1-38bf5c24efb2}" location="NewVirtualDisk2.vdi" format="VDI" type="Normal"/>
</HardDisks>
<DVDImages>
<Image uuid="{8f02c47e-5de7-41e2-a2e6-4dbc78461a89}" location="C:/Users/SnC/VirtualBox VMs/ubuntu-20.04-desktop-amd64.iso"/>
</DVDImages>
.....
<StorageControllers>
<StorageController name="IDE" type="PIIX4" PortCount="2" useHostIOCache="true" Bootable="true">
<AttachedDevice passthrough="false" type="DVD" hotpluggable="false" port="1" device="0">
<Image uuid="{8f02c47e-5de7-41e2-a2e6-4dbc78461a89}"/>
</AttachedDevice>
</StorageController>
<StorageController name="NVMe Controller 1" type="NVMe" PortCount="1" useHostIOCache="false" Bootable="true">
<AttachedDevice nonrotational="true" type="HardDisk" hotpluggable="false" port="0" device="0">
<Image uuid="{3016646e-0a45-40a1-9fe1-5d784970260a}"/>
</AttachedDevice>
</StorageController>
<StorageController name="NVMe Controller 2" type="NVMe" PortCount="1" Instance="1" useHostIOCache="false" Bootable="false">
<AttachedDevice nonrotational="true" type="HardDisk" hotpluggable="false" port="0" device="0">
<Image uuid="{48b63885-44f2-4bcc-a6e1-38bf5c24efb2}"/>
</AttachedDevice>
</StorageController>
</StorageControllers>
NewVirtualDisk1.vdi is 10GB, NewVirtualDisk2.vdi is 20GB

The 'Disks' app only shows one disk, the 10GB one, which for some reason shows as 11GB in the app:
ubuntu 2 nvmes.png
ubuntu 2 nvmes.png (126.25 KiB) Viewed 2853 times
I am only proficient in Windows, so I am not certain what else to do to see if the other disk is waiting somewhere to be activated, like Windows Disk Management requires, or if there's some command line wizardry that's needed. What would you do next, please?
bwoebi
Posts: 3
Joined: 25. Aug 2020, 12:31

Re: Presenting the guest two different nvme controllers

Post by bwoebi »

I basically have no idea what to do next :-)

I found that, in the initial minute after boot, there is a /dev/nvme1 (as in: raw device for the controller) which then gets removed with the couple error messages in the kernel. You can make linux reload that controller, but it will always fail to issue the IRQ request with a timeout.
I am suspecting either a bug in the emulator os that I need to set ssome magical extradata for linux to recognize?
scottgus1
Site Moderator
Posts: 20945
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Presenting the guest two different nvme controllers

Post by scottgus1 »

Here's what I found after remembering Linux has a syslog and googling for where it is:
ubuntu 2 nvme syslog.zip
(37.25 KiB) Downloaded 11 times
This is the live-CD booting with the setup in my previous post. Here's every tline mentioning 'nvme':
Aug 28 17:38:58 ubuntu kernel: [ 2.624446] nvme nvme0: pci function 0000:00:0e.0
Aug 28 17:38:58 ubuntu kernel: [ 2.625557] nvme nvme0: failed to register the CMB
Aug 28 17:38:58 ubuntu kernel: [ 2.627082] nvme nvme0: 2/0/0 default/read/poll queues
Aug 28 17:38:58 ubuntu kernel: [ 2.627278] nvme nvme1: pci function 0000:03:00.0
Aug 28 17:38:58 ubuntu kernel: [ 2.628370] nvme nvme1: failed to register the CMB
Aug 28 17:38:58 ubuntu kernel: [ 2.628878] nvme nvme1: Duplicate cntlid 0 with nvme0, rejecting
Aug 28 17:38:58 ubuntu kernel: [ 2.629189] nvme nvme1: Removing after probe failure status: -22
So the second NVMe controller has the same 'cntlid' (whatever that is) as the first controller. Whether that's a bug or what else it might mean, I have no idea. Someone who swims in the source code might know.
fth0
Volunteer
Posts: 5677
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Presenting the guest two different nvme controllers

Post by fth0 »

scottgus1 wrote:Someone who swims in the source code might know.
Unfortunately, the extended swimming pool is probably not available to me (NVMe controller -> VirtualBox Extension Pack).
bwoebi wrote:Shall I go to the bug tracker?
I'd suggest to do so.

Background info: The NVMe standard requires Controller IDs to be unique inside a NVMe Subsystem. My understanding is that a physical NVMe device represents a NVMe subsystem with one or multiple controllers. I don't know if VirtualBox creates one or two NVMe subsystems when configured with two NVMe controllers. If you have a chance, you could try executing sudo nvme list at the right time.
scottgus1
Site Moderator
Posts: 20945
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Presenting the guest two different nvme controllers

Post by scottgus1 »

fth0 wrote:extended swimming pool is probably not available to me
Ah, the EP is closed source, gotcha.

I tried 'sudo nvme list' and 'sudo list nvme' in a terminal and got "command not found" for 'nvme' and 'list'.

In /dev/disk/by-id, there's this:
dev disk byid.png
dev disk byid.png (19.22 KiB) Viewed 2770 times
/dev/disk/py-path has this:
dev disk bypath.png
dev disk bypath.png (9.45 KiB) Viewed 2770 times
/dev has these two nvme's:
dev nvme.png
dev nvme.png (5.18 KiB) Viewed 2770 times
fth0
Volunteer
Posts: 5677
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Presenting the guest two different nvme controllers

Post by fth0 »

scottgus1 wrote:I tried 'sudo nvme list' and 'sudo list nvme' in a terminal and got "command not found" for 'nvme' and 'list'.
Depending on the Linux distribution used, you'll have to install the nvme tools first (e.g. sudo apt install nvme-cli).

Your pictures show only one NVMe controller and device (nvme-1 on PCI bus 00 device 0e function 0, nvme0, namespace nvme0n1).

The OP mentioned they can "make linux reload" the 2nd controller, and my "at the right time" was hinted at that. During booting, there seems to be a 60 seconds timeout starting nvme1, so there is maybe a "right time" somewhere. Additionally, the OP could use this command on the original servers they are trying to replicate, for comparison.
Alexsd
Posts: 1
Joined: 3. Oct 2020, 03:37

Re: Presenting the guest two different nvme controllers

Post by Alexsd »

I'm having the same issue. I can add multiple NVMe controllers, but I can't see them in the guest(Ubuntu 20.04). Is there no way to get this working?
Post Reply