multiple nvme contoller

[kenzo]

Hi,

I'm new on this site and i've been in read mode these past few days.

I'm trying to create a VM which houses 2 nvme controller as per my setup file:

config.vm.provider "virtualbox" do |vb|
    vb.customize ["modifyvm", :id, "--ioapic", "on"]
    vb.customize ["modifyvm", :id, "--chipset", "ich9"]
    vb.memory = "#{vmram}"
    vb.cpus = "#{vmcpu}"

    nvme_disk1 = 'nvme1.vdi'
    unless File.exist? (nvme_disk1)
        vb.customize ["createmedium", "--filename", nvme_disk1, "--variant", "Fixed", "--size", "1024"]
        vb.customize ["storagectl", :id, "--name", "nvme1", "--add", "pcie", "--controller", "NVMe", "--portcount", "2", "--bootable", "off"]
        vb.customize ["storageattach", :id, "--storagectl", "nvme1", "--type", "hdd", "--medium", nvme_disk1, "--port", "0"]
    end

    nvme_disk2 = 'nvme2.vdi'
    unless File.exist? (nvme_disk2)
        vb.customize ["createmedium", "--filename", nvme_disk2, "--variant", "Fixed", "--size", "512"]
        vb.customize ["storagectl", :id, "--name", "nvme2", "--add", "pcie", "--controller", "NVMe", "--portcount", "1", "--bootable", "off"]
        vb.customize ["storageattach", :id, "--storagectl", "nvme2", "--type", "hdd", "--medium", nvme_disk2, "--port", "0"]
    end

It successfully created what i want, as show in lspci

00:0e.0 Non-Volatile memory controller: InnoTek Systemberatung GmbH Device 4e56
00:18.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f2)
00:19.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f2)
00:1f.0 ISA bridge: Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge (rev 02)
00:1f.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
02:18.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f2)
03:00.0 Non-Volatile memory controller: InnoTek Systemberatung GmbH Device 4e56

problem is, after sometime, the 2nd controller will timeout and eventually removing it from the list:

[ 61.948231] nvme 0000:03:00.0: I/O 0 QID 0 timeout, disable controller
[ 61.948352] ------------[ cut here ]------------
[ 61.948425] WARNING: CPU: 3 PID: 0 at /build/linux-Ay7j_C/linux-4.4.0/kernel/irq/manage.c:1438 __free_irq+0x1d2/0x280()
[ 61.948488] Trying to free IRQ 17 from IRQ context!
[ 61.948579] Modules linked in: vboxsf(OE) binfmt_misc ppdev crct10dif_pclmul vboxvideo(OE) lpc_ich ttm crc32_pclmul drm_kms_helper aesni_intel aes_x86_64 lrw gf128mul glue_helper drm ablk_helper cryptd input_leds fb_sys_fops serio_raw syscopyarea sysfillrect i2c_piix4 sysimgblt vboxguest(OE) parport_pc parport 8250_fintek mac_hid sunrpc autofs4 psmouse pata_acpi video nvme fjes e1000
[ 61.948765] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE 4.4.0-21-generic #37-Ubuntu
[ 61.948808] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 61.948814] 0000000000000286 b5573404d83674e0 ffff880123d83b10 ffffffff813e93c3
....
[ 61.949489] Trying to free nonexistent resource <000000000000e000-000000000000e007>
[ 61.949689] nvme 0000:03:00.0: Cancelling I/O 0 QID 0
[ 61.949927] nvme 0000:03:00.0: Identify Controller failed (-4)
[ 61.950565] nvme 0000:03:00.0: Removing after probe failure

Has anyone tried this? Do we have any workaround or solution for this kind of configuration?

TIA..

[socratis]

1. That looks like a part of a script. We need to see the whole thing, along with the resulting .vbox.
2. You didn't mention your host, your guest or your VirtualBox. Makes and versions, complete versions, including builds and revisions please.
3. What happens if you create the VM from the GUI?

[kenzo]

I am running Vagrant on top of VB. The snippet above is from the VagrantFile used to create VM. Vagrant use VB as a provider.

I am running on Ubuntu 16.04 LTS on 4.4.0-21-kernel Host.I have the above issue happening when creating Ubuntu16 and Ubuntu18. VB is 5.2.18r124319 version.

I wasn't able to create VM from GUI since I am only running on a command line.

The concept is to create a VM that has 2 NVMe controller so I can test the P2P capability of the SPDK. And somehow, I was able to make it work initially, but has some issues during runtime. If you can share a reading material or point me to a right direction on how to successfully create 2 instance of NVme controller. Based on my search, I only see VMs created with multiple disks under 1 controller but multiple ports, but never on a separate controller.

[socratis]

I am running Vagrant on top of VB.

Vagrant is a program that relies on VirtualBox but modifies its configuration files in unknown ways to us. It is not supported on these VirtualBox user forums, they have their own Vagrant support channels. If you are having this problem with a standalone version of VirtualBox, then we can continue this discussion.

I only see VMs created with multiple disks under 1 controller but multiple ports, but never on a separate controller.

That is true. The GUI for example won't let you add a second controller from the same family, that option is greyed out, disabled. More ports? Yes. More controllers? Yes. More controller of the same family? No.

Same thing if you try from the CLI, you get an error:

$ VBoxManage storagectl Test --name "NVMe1" --add pcie --controller NVMe
$ VBoxManage storagectl Test --name "NVMe2" --add pcie --controller NVMe
VBoxManage: error: Too many storage controllers of this type
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component StorageControllerWrap, interface IStorageController, callee nsISupports
VBoxManage: error: Context: "AddStorageController(Bstr(pszCtl).raw(), StorageBus_PCIe, ctl.asOutParam())" at line 1110 of file VBoxManageStorageController.cpp

[kenzo]

That is true. The GUI for example won't let you add a second controller from the same family, that option is greyed out, disabled. More ports? Yes. More controllers? Yes. More controller of the same family? No.

Thank you! This is what I am looking for.

You may now close this thread.

[socratis]

vb.customize ["modifyvm", :id, "--chipset", "ich9"]

I just noticed the above modification in your script; you switched to ICH9. Thing is that ICH9 is labelled as "experimental", and "OSX-guests-only". It's ... half-baked, a work in progress.

Keeping all that in mind, and finding the following from the 5.2.10 changelog...

• Storage: fixed having multiple NVMe controllers with ICH9 enabled (5.2.0 regression)

... I changed my guinea-pig VM to have the ICH9 chipset instead of the default PIIX3. And here's what happened:

• I could add a 2nd NVMe (or any other) controller from the GUI.
• I did not get an error from the CLI when adding a 2nd NVMe controller.

I also tested an Ubuntu 18.04, a Win10 and an OSX 10.13 guests, based on that assumption, 2 NVMe controllers that each one had a 10 GB NVMe "hard drive" attached:

• Ubuntu only saw the 1st adapter. Installed fine, but there was no way that I could see the 2nd NVMe controller or HD.
• Win10 didn't even bother. No hard drives there.
• Same with OSX 10.13. Nothing to see here folks...

So, I think that what you're trying to do is still "experimental" and not fully baked. I would definitely *not* consider it a bug, because a bug is something that's advertised as working and it's not. That's not the case for either ICH9 or NVMe. In a couple of years maybe, depending on the demand of paying customers and/or the source code contributions of people that need it...