Major packet loss between physical hosts and VMs w/ bridged network adapters

[itwerks4me]

Hello,

I recently upgraded our server to Ubuntu 16.04 64bit and took the opportunity to also upgrade Virtualbox from 5.0 to 5.1.4 and I'm now suffering crippling packet loss when physical hosts on the network access any of the guest VMs. I have two instances of Ubuntu-server 64bit, one Windows XP VM, and a Windows 7 Pro 64bit VM hosted on this machine. Physical hosts on the network can communicate with the Host server fine, iperf3 shows traffic moving between 900-1000Mbit/s. VMs can communicate with the host server without issue as well, iperf3 shows traffic moving well above 1Gbit/s with no dropped packets. But i am suffering crippling packet loss when testing communications between any of my physical hosts and any of the VMs... an example of this poor performance:

>iperf3 -c 10.0.10.205
Connecting to host 10.0.10.205, port 5201
[ 4] local 10.0.10.12 port 62835 connected to 10.0.10.205 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 4] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.00-3.00 sec 256 KBytes 2.10 Mbits/sec
[ 4] 3.00-4.00 sec 384 KBytes 3.14 Mbits/sec
[ 4] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 7.00-8.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 8.01-9.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 9.01-10.01 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 896 KBytes 733 Kbits/sec sender
[ 4] 0.00-10.01 sec 673 KBytes 551 Kbits/sec receiver


I have tried every adapter available to each machine within the VB settings, all Intel NICs and PCFast-III suffer the same crippling behaviour.

This setup worked fine until the upgrade to Ubuntu 16.04. I've downgraded Virtualbox to 5.0.24 and the issue remains. All VMs have Guest Additions installed. Virtualbox Extension Pack is installed. I see no errors within the Host machine's syslog from Virtualbox. I see no errors with VBox.log for any of the machines. The Host machine isn't getting hammered, load is normal, iotop shows nothing of concern, system memory isn't over-allocated (1GB+ free). The issue persists whether I have a single VM running or all four.

I'm baffled.

Please advise, critical applications have failed and I must resolve this issue asap.

I'm happy to provide any further information on demand and TIA for any assistance.

[Perryg]

I don't have a 16.04 host to test this on but assuming this is a clean install and not an upgrade which sometimes causes weird things to happen, you probably should post a ticket at bugtracker

[itwerks4me]

This is indeed an upgrade not a clean install.

[itwerks4me]

I'm adding some attachments, I ran netstat and ethtool on the host nic and am adding the vbox.log as well. I turned on nictitating for the most critical vm and have that capture should anyone want it.

ethtool-k_eth0.txt

ethtool

(1.28 KiB) Downloaded 24 times

JAA-EZ-MASTER-2016-08-31-00-32-38.log

Vbox.log

(54.57 KiB) Downloaded 9 times

netstat.txt

Netstat

(2.87 KiB) Downloaded 14 times

[itwerks4me]

Well... i ran iperf3 using UDP to measure packet loss after examining the nictrace file in Wireshark and discovering there were no drops.... and iperf3 says the same. No dropped packets:

iperf3.exe -V -u -c 10.0.10.205
iperf 3.1.3
CYGWIN_NT-10.0 JAAWS01 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Thu, 01 Sep 2016 01:38:53 GMT
Connecting to host 10.0.10.205, port 5201
Cookie: JAAWS01.1472693933.057693.0172e5da1e
[ 4] local 10.0.10.10 port 60020 connected to 10.0.10.205 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.02 sec 128 KBytes 1.03 Mbits/sec 16
[ 4] 1.02-2.02 sec 136 KBytes 1.11 Mbits/sec 17
[ 4] 2.02-3.02 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 3.02-4.00 sec 120 KBytes 998 Kbits/sec 15
[ 4] 4.00-5.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 8.00-9.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 16
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 4.200 ms 0/159 (0%)
[ 4] Sent 159 datagrams
CPU Utilization: local/sender 0.9% (0.8%u/0.2%s), remote/receiver 0.0% (0.0%u/0.0%s)

The host's NIC is an Nvidia MCP65.

Ubuntu is loading the forcedeth driver.

Here's more info on the host:

Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial

Linux jaasrv02 4.4.14-040414-generic #201606241434 SMP Fri Jun 24 18:36:45 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[itwerks4me]

Bump.

Anyone? Bueller?

Somewhere between my Host's fully functional nic and my vm network traffic is broken. I switched the most critical vms to virtio nics, speed increased but the problem remains. How can I troubleshoot further? Should I turn on vm debugging? Should I file a bug even if my host was upgraded to 16.04 instead of a clean install?

Need advice on how to proceed please.

[Perryg]

As I said earlier for myself I can not offer more since I can not replicate this and do not have a 16.04 host. I believe your best approach is to post a ticket at bugtracker and see what the DEVs have to say about this. Keep in mind that they are going through vacations right now and it might take longer for someone to reply.

[itwerks4me]

Thanks, I do appreciate your input. I will file a bug report over the weekend.

[NinjaPenguin]

Was a resolution ever found for this issue? I'm seeing a very similar problem with VirtualBox on Ubuntu Server 16.04, version 5.0.24. All the physical interfaces and connections check out, but the virtual guests experience a ~10% packet loss on any communications to/from other hosts on the physical network. This is on an HP ML350 G9, with multiple NICs, and no load on it (yet).
I have tried configurations with one active physical network interface, two active interfaces, and two bonded interfaces, all with almost the exact same results, 7-12% packet loss every time.