Intel 82575/82576 NICs and igb driver with VBox Bridging

[crash0veride]

Just want to put some information out there for any users who may be encountering network issues in their VBox guests with a Linux host using VirtualBox bridging with Intel 82575/82576 based ethernet controllers.

I have only observed this in VBox guests running on Nehalem and Westmere based servers or on any Intel PCI-E based adapters, which utilize this Intel silicon with linux as a host OS.

The symptoms of this are:
- Slow networking performance in the guest with I/O intensive network operations.
- Slow network performance in the guest with certain network protocols and operations.
- Excessively large amounts of TCP re-transmissions
- Constant out of order TCP packet deliveries
- Excessive TCP segmentation
- Excessive TCP recv window overflows
- Adapter resets

The above symptoms can be observed by utilizing wireshark to observe a VBox nictrace (--nictrace1 on --nictracefile1 /path/to/tracefile). Additionally a simple "netstat -s ethX" where "ethX" is the interface to which the guest is bridged to will show some of the above as well.

The cause of the above as known at the moment is some improper interaction with VBox bridging and the Generic Receive Offloading (GRO) feature which is now defaulted “on” in the igb driver in Linux.
Below is the description of GRO from the Intel igb driver kernel documentation:
- Generic Receive Offload -
The igb driver supports the in-kernel software implementation of GRO. GRO has shown that by coalescing Rx traffic into larger chunks of data, CPU utilization can be significantly reduced when under large Rx load. GRO is an evolution of the previously-used LRO interface. GRO is able to coalesce other protocols besides TCP. It's also safe to use with configurations that are problematic for LRO, namely bridging and iSCSI.

The workaround for this at the moment is to disable GRO on the host. This can be done via ethtool by issuing the following: "ethtool –K gro off"
Note however that only newer or patched versions of ethtool have the ability to disable GRO. I have checked for and tried this with ethtool in a number of Linux distributions.

I found the following distributions include a newer of patched version of ethtool which can disable GRO:
- OEL 5.5
- RHEL 5.5
- CentOS 5.5
- Fedora 12
- Fedora 13
- Ubuntu 10.4

I found the following distributions did not have this feature present or patched into their ethtool:
- OpenSUSE 11.1, 11.2, 11.3
- SLES 11 and SLES 11 SP1
- SLED 11 and SLED 11 SP1

[crash0veride]

*Update*

Fixed in VirtualBox Version 3.2.8

[smaser]

I run Virtual Box 3.2.8 hosting a Windows 2003 server. My hardware is supermicro and the ethernet interface uses the intel 82575EE. I am seeing performance problems. The linux version i am running is 2.6.9-89.0.11.ELsmp. The workaround mentioned in the article does not appear to be available through ethtool. One option I'm considering is to install a nic card using a different chipset.

These are the cards listed in Virtual Networking section of VirtualBox help. I'm assuming any one of these will provide the best shot at avoiding hwr compatibility issues (at least with VirtualBox). Any other recommendations for dealing wth this issue ?

AMD PCNet PCI II (Am79C970A);
AMD PCNet FAST III (Am79C973, the default);
Intel PRO/1000 MT Desktop (82540OEM);
Intel PRO/1000 T Server (82543GC);
Intel PRO/1000 MT Server (82545EM);
Paravirtualized network adapter (virtio-net).

Thanks.

[fixedwheel]

My hardware is supermicro and the ethernet interface uses the intel 82575EE. I am seeing performance problems. The linux version i am running is 2.6.9-89.0.11.ELsmp.

are you sure your problem is related to the OP: is the supermicro (-v please) hardware running a "Nehalem" CPU?
2.6.9-89.0.11.ELsmp is Red Hat EL 4 (or Centos/OEL clone) no wonder your ethtool doesn't have the new option ... and does the igb module from old 2.6.9 kernel have the GRO option, and if: enabled?

One option I'm considering is to install a nic card using a different chipset.

These are the cards listed in Virtual Networking section of VirtualBox help. I'm assuming any one of these will provide the best shot at avoiding hwr compatibility issues (at least with VirtualBox).

no, these are emulated by VBox because most guest operating systems have drivers for them: e.g. the emulated intel as new Windows 64bit doesn't have drivers for AMD PCNet chips any more.
You can't buy a real Paravirtualized NIC (virtio-net) and the AMD PCNet ones are 10Mbit ...


for the moment i would examine the reason of degraded performance more closely before buying a new NIC


additional question: do you need to run this host on old RHEL 4 (or compatible)?

[crash0veride]

As long as you are running VBox Version 3.2.8 the fix itself for this issue was incorporated into the vboxnetflt code. No need to disable GRO or otherwise worry about it. Can you be more verbose about the network performance issues you are seeing?