Just want to put some information out there for any users who may be encountering network issues in their VBox guests with a Linux host using VirtualBox bridging with Intel 82575/82576 based ethernet controllers.
I have only observed this in VBox guests running on Nehalem and Westmere based servers or on any Intel PCI-E based adapters, which utilize this Intel silicon with linux as a host OS.
The symptoms of this are:
- Slow networking performance in the guest with I/O intensive network operations.
- Slow network performance in the guest with certain network protocols and operations.
- Excessively large amounts of TCP re-transmissions
- Constant out of order TCP packet deliveries
- Excessive TCP segmentation
- Excessive TCP recv window overflows
- Adapter resets
The above symptoms can be observed by utilizing wireshark to observe a VBox nictrace (--nictrace1 on --nictracefile1 /path/to/tracefile). Additionally a simple "netstat -s ethX" where "ethX" is the interface to which the guest is bridged to will show some of the above as well.
The cause of the above as known at the moment is some improper interaction with VBox bridging and the Generic Receive Offloading (GRO) feature which is now defaulted “on” in the igb driver in Linux.
Below is the description of GRO from the Intel igb driver kernel documentation:
- Generic Receive Offload -
The igb driver supports the in-kernel software implementation of GRO. GRO has shown that by coalescing Rx traffic into larger chunks of data, CPU utilization can be significantly reduced when under large Rx load. GRO is an evolution of the previously-used LRO interface. GRO is able to coalesce other protocols besides TCP. It's also safe to use with configurations that are problematic for LRO, namely bridging and iSCSI.
The workaround for this at the moment is to disable GRO on the host. This can be done via ethtool by issuing the following: "ethtool –K gro off"
Note however that only newer or patched versions of ethtool have the ability to disable GRO. I have checked for and tried this with ethtool in a number of Linux distributions.
I found the following distributions include a newer of patched version of ethtool which can disable GRO:
- OEL 5.5
- RHEL 5.5
- CentOS 5.5
- Fedora 12
- Fedora 13
- Ubuntu 10.4
I found the following distributions did not have this feature present or patched into their ethtool:
- OpenSUSE 11.1, 11.2, 11.3
- SLES 11 and SLES 11 SP1
- SLED 11 and SLED 11 SP1