[Help] PCI-passthrough for NIC Intel 82599 10G bit Ethernet Card

Discussions related to using VirtualBox on Linux hosts.
Post Reply
Cancer1987
Posts: 11
Joined: 11. Aug 2017, 05:05

[Help] PCI-passthrough for NIC Intel 82599 10G bit Ethernet Card

Post by Cancer1987 »

Hi all,

I'm trying to set up a virtualbox-5.1 with PCI passthrough on
host Ubuntu 16.04 (4.8.0-36-generic #36~16.04.1-Ubuntu SMP Fri Jun 30 05:32:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux)
installed on PC with
CPU Intel® Core™ i7-3612QE CPU @ 2.10GHz
RAM 12GB
integrated video
BIOS Vendor American Megatrends
BIOS Version CS720008

PCI device trying to passthrough is an Intel 82599 10Gigabit Ethernet Controller

Here is my pci device information.
#lspci
00:00.0 Host bridge: Intel Corporation 3rd Gen Core processor DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation 3rd Gen Core processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4)
00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 3 (rev c4)
00:1c.3 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 4 (rev c4)
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation QM77 Express Chipset LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
02:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
06:00.0 Serial controller: Exar Corp. Device 0358 (rev 03)
07:00.0 PCI bridge: Pericom Semiconductor PCI Express to PCI-XPI7C9X130 PCI-X Bridge (rev 04)
guest OS is Windows server 2003.

I've already made guest OS to recognized the Ethernet Controller, via the cmd
VBoxManage modifyvm windows2003 --pciattach 02:00.0@02:00.0
but it cannot work properly in Guest OS, in the Windows Device Manager of guest OS, there is a exclamation mark on its icon. that is why I'd like to ask for some help.

I'm sure the VT-d and IOMMU are enabled
# dmesg | grep -e IOMMU -e DMAR
[    0.000000] ACPI: DMAR 0x00000000DABC13E8 0000B8 (v01 INTEL  SNB      00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.037657] DMAR: Host address width 36
[    0.037658] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.037666] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.037667] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.037670] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.037671] DMAR: RMRR base: 0x000000dae9d000 end: 0x000000daeacfff
[    0.037672] DMAR: RMRR base: 0x000000db800000 end: 0x000000df9fffff
[    0.037674] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.037674] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.037675] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.037984] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.962717] DMAR: No ATSR found
[    0.963435] DMAR: dmar0: Using Queued invalidation
[    0.963442] DMAR: dmar1: Using Queued invalidation
[    0.963767] DMAR: Setting RMRR:
[    0.963822] DMAR: Setting identity map for device 0000:00:02.0 [0xdb800000 - 0xdf9fffff]
[    0.964293] DMAR: Setting identity map for device 0000:00:14.0 [0xdae9d000 - 0xdaeacfff]
[    0.964340] DMAR: Setting identity map for device 0000:00:1a.0 [0xdae9d000 - 0xdaeacfff]
[    0.964387] DMAR: Setting identity map for device 0000:00:1d.0 [0xdae9d000 - 0xdaeacfff]
[    0.964398] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.964436] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.964721] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    1.218255] [drm] DMAR active, disabling use of stolen memory
[    3.403875] vboxpci: IOMMU found
after I start the VM, it appear some more ...

[  105.514597] vboxpci: created IOMMU domain ffff9059a4933e88
[  105.666532] vboxpci 0000:02:00.0: failed to attach to IOMMU, error -22
Some other situations:
1. I'v added intel_iommu=on in /etc/default/grub, and executed "update-grub2", otherwise "dmesg" won't show IOMMU is found.
2. if I executed "ifconfig eth0 down", to unbind the ethernet card, then start the VM, in device manager of Guest OS,
there WON'T be a exclamation mark on it, but in Network Connection Window, its network state is disconnected. after disable and enable the connection,
the connection will disapear, and the exclamation mark will appear on it again in device manager window.
3. I've heard someone made it successfully , using same CPU and NIC card, so maybe I miss some config(???) or mainboard chipset is not supported IOMMU ?

Thank you in advance for any help and/or suggestions
Perryg
Site Moderator
Posts: 34369
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: [Help] PCI-passthrough for NIC Intel 82599 10G bit Ethernet Card

Post by Perryg »

As you have seen PCI pass-through is difficult and only a hand few of people have been able to get it to work. I am not even sure that the function has been updated to the newer releases, but with that said I wonder why you need to pass a NIC directly to the guest? What is the reason that you can not use the normal feature?
Cancer1987
Posts: 11
Joined: 11. Aug 2017, 05:05

Re: [Help] PCI-passthrough for NIC Intel 82599 10G bit Ethernet Card

Post by Cancer1987 »

Perryg wrote:As you have seen PCI pass-through is difficult and only a hand few of people have been able to get it to work. I am not even sure that the function has been updated to the newer releases, but with that said I wonder why you need to pass a NIC directly to the guest? What is the reason that you can not use the normal feature?
The reason is that a PCI-passthrough NIC may has higher performance than VIRTIO-net or software simulated NIC.

My work is to improve the guest NIC performance as rapid as possible.

I'v tested the performance of VIRTIO-net under different circumstances via iperf3, here is the avarage speed test result.
(using cmd iperf3 -c 192.168.1.3 -w 1024k)
kvm + Ubuntu 16.04 5.69Gb/s
kvm + Windows 2003 3.40Gb/s
VirtualBox + Ubuntu 16.04 2.80Gb/s
VirtualBox+ Windows 2003 2.40Gb/s

(VirtualBox version is 5.0.40, it has better performance in version 5.1.24)

Obviously, virtio-net has higher performance in kvm than in VBox, and virtio-net guest dirver has higher performance in Ubuntu than in windows.

Unfortunately, Boss persists in using VirtualBox and Windows 2003 guest OS, the baddest situation in the test result you see.
What's more, he demands the iperf3 test result should not be less than 4Gb/s :( :(
I have 2 choices, improving the virtio-net performance, or using PCI-passthrough instead of, just like one of our rivals did and they succeed.

What's more, I'm reading source code of virtualbox now, try to improve the virtio-net performance by edit source code, so where should I start my reading ? :)


Another question, why the iperf3 test result is asymmetrical. It has pooler performance when guest OS acts as the iperf server than host machine acts.

the test result is:
GUEST OS(iperf client) ----------------> HOST (iperf server) nealy 9Gb/s
GUEST OS(iperf server) <---------------- HOST (iperf client) 3Gb/s
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: [Help] PCI-passthrough for NIC Intel 82599 10G bit Ethernet Card

Post by socratis »

Cancer1987 wrote:I'm reading source code of VirtualBox now, try to improve the virtio-net performance by edit source code, so where should I start my reading ?
Developer question
You have several options. See: https://www.virtualbox.org/wiki/Community :
  • If you want an answer to a simple question like the one above, try the [url=irc://irc.freenode.net/vbox-dev]vbox-dev channel[/url] over IRC (warning: IRC link).
  • For more convoluted questions (that cannot be answered in 2-3 lines), try the mailing list.
  • And do not forget the forums, the "VirtualBox OSE" and the "Developers corner" sub-forums. Ask once, a lot of people might read the answer and benefit... ;)
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: [Help] PCI-passthrough for NIC Intel 82599 10G bit Ethernet Card

Post by mpack »

Cancer1987 wrote: The reason is that a PCI-passthrough NIC may has higher performance than VIRTIO-net

I would doubt that. The performance restriction on network cards is most often the speed of the network itself. Plus with Virtio-NET you've essentially got the same access as a host app. Nothing simulated, so no simulation overhead.

Performance benchmarks run inside the guest are usually a waste of time, since the timer is simulated too.
Perryg
Site Moderator
Posts: 34369
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: [Help] PCI-passthrough for NIC Intel 82599 10G bit Ethernet Card

Post by Perryg »

Both Socratis and mpack are corrrect. But I have done a lot of testing with network pass-through and VirtIO is by far the fastest.

Code: Select all

mint@mint-17 ~ $ iperf -c 192.168.1.10 -w 1024k
------------------------------------------------------------
Client connecting to 192.168.1.10, TCP port 5001
TCP window size:  416 KByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 192.168.1.128 port 55340 connected with 192.168.1.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  4.65 GBytes  4.00 Gbits/sec
It allows the guest to directly use the hosts NIC with very little translation ( overhead ) and as you see the numbers are subjective. I also find that the test will slow down when increasing the size of the test but I suspect this has to do with disk IO more than the actual network speed, since IO has always been a factor in virtual. Try messing with the host I/O cache in the guest settings and see if that improves the speed, but numbers and test results are just that, numbers. As for your competition and them getting PCI pass-through to work properly I would think that odd myself knowing just how hard it is ( I did get it to work in the past ) and I did not see all that much improvement. Not enough to warrant the hassle anyway.
Post Reply