Ubuntu Xenial (16.04 LTS) network interface bouncing

[mabnhdev]

I'm running the new Ubuntu Xenial (16.04 LTS) in a VBox (5.0.16) on a Windows 7 host. The VBox Adapter 1 is Intel PRO/1000 MT Desktop (NAT).

I'm running a large build on the VBox. During one phase, the build becomes very network I/O intensive. Shortly after entering this phase of the build, I always lose my network connection and it usually takes about 1-2 minutes to come back. The network interface continues to bounce throughout this phase of the build. This happens every time I perform this phase of the build on my Xenial VBox - it is 100% reproducible.

I haven't noticed that the interface bounces at any other time while I'm running other things on the VBox.

I'm using all default network configuration - I haven't changed any traffic shaping settings, etc.

I don't think this is a problem with my host machine or my host network since I run the exact same build on a Debian Jessie VBox configured the same as the Xenial VBox on the same host machine with no problems at all. I guess for that reason, I'm not at all convinced that this is a VBox problem either, but...

Any troubleshooting guidance would be appreciated.

Thanks, Mike

[Perryg]

I think you really answered your own question. If other guests do not behave the same as the 16.04 guest I would think the place to look would be the 16.04 guest. What does dmesg and or syslog show on the guest? Use wireshark to isolate the failing point.

[mabnhdev]

Update: The following accounts for losing the network interface for almost a minute. Again, 100% reproducible when running a large build at the same point within that build. Very predictable. Running same build on a Debian Jessie VBox on same host is fine. I have seen many e1000 reset adapter messages without the netdev_watchdog also during this phase of the build.

I'll see if WireShark shows anything...

Code: Select all

[ 3274.670123] ------------[ cut here ]------------
[ 3274.670185] WARNING: CPU: 0 PID: 0 at /build/linux-Ay7j_C/linux-4.4.0/net/sched/sch_generic.c:303 dev_watchdog+0x237/0x240()
[ 3274.670207] NETDEV WATCHDOG: enp0s3 (e1000): transmit queue 0 timed out
[ 3274.670225] Modules linked in: vboxsf(OE) snd_intel8x0 snd_ac97_codec ac97_bus crct10dif_pclmul snd_pcm crc32_pclmul snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq aesni_intel snd_seq_device aes_x86_64 lrw snd_timer gf128mul joydev glue_helper ablk_helper cryptd input_leds snd vboxvideo(OE) serio_raw i2c_piix4 drm soundcore vboxguest(OE) 8250_fintek mac_hid parport_pc ppdev lp parport autofs4 hid_generic usbhid hid ahci fjes psmouse libahci video e1000 pata_acpi
[ 3274.670309] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE   4.4.0-21-generic #37-Ubuntu
[ 3274.670324] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 3274.670326]  0000000000000286 72c1feb3be21c3bb ffff88011fc03d98 ffffffff813e93c3
[ 3274.670330]  ffff88011fc03de0 ffffffff81d622b8 ffff88011fc03dd0 ffffffff81080f62
[ 3274.670333]  0000000000000000 ffff8800355f6480 0000000000000000 ffff880035994000
[ 3274.670337] Call Trace:
[ 3274.670339]  <IRQ>  [<ffffffff813e93c3>] dump_stack+0x63/0x90
[ 3274.670350]  [<ffffffff81080f62>] warn_slowpath_common+0x82/0xc0
[ 3274.670353]  [<ffffffff81080ffc>] warn_slowpath_fmt+0x5c/0x80
[ 3274.670358]  [<ffffffff81741217>] dev_watchdog+0x237/0x240
[ 3274.670361]  [<ffffffff81740fe0>] ? qdisc_rcu_free+0x40/0x40
[ 3274.670366]  [<ffffffff810ec345>] call_timer_fn+0x35/0x120
[ 3274.670369]  [<ffffffff81740fe0>] ? qdisc_rcu_free+0x40/0x40
[ 3274.670373]  [<ffffffff810eccfa>] run_timer_softirq+0x23a/0x2f0
[ 3274.670376]  [<ffffffff810859a1>] __do_softirq+0x101/0x290
[ 3274.670380]  [<ffffffff81085ca3>] irq_exit+0xa3/0xb0
[ 3274.670384]  [<ffffffff81826fa2>] smp_apic_timer_interrupt+0x42/0x50
[ 3274.670388]  [<ffffffff81825262>] apic_timer_interrupt+0x82/0x90
[ 3274.670389]  <EOI>  [<ffffffff810645e6>] ? native_safe_halt+0x6/0x10
[ 3274.670398]  [<ffffffff81038dde>] default_idle+0x1e/0xe0
[ 3274.670400]  [<ffffffff810395ef>] arch_cpu_idle+0xf/0x20
[ 3274.670405]  [<ffffffff810c3d0a>] default_idle_call+0x2a/0x40
[ 3274.670407]  [<ffffffff810c4071>] cpu_startup_entry+0x2f1/0x350
[ 3274.670412]  [<ffffffff81817f2c>] rest_init+0x7c/0x80
[ 3274.670418]  [<ffffffff81f5a011>] start_kernel+0x481/0x4a2
[ 3274.670430]  [<ffffffff81f59120>] ? early_idt_handler_array+0x120/0x120
[ 3274.670433]  [<ffffffff81f59339>] x86_64_start_reservations+0x2a/0x2c
[ 3274.670436]  [<ffffffff81f59485>] x86_64_start_kernel+0x14a/0x16d
[ 3274.670463] ---[ end trace c22f760451207699 ]---
[ 3274.670784] e1000 0000:00:03.0 enp0s3: Reset adapter
[ 3274.726396] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 3384.650054] e1000 0000:00:03.0 enp0s3: Reset adapter
[ 3384.707703] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

[Perryg]

Look at the power management features on the guest and make sure they are all shut off.


What exactly is this large build you reference? I only ask because I have never had a network drop issue on any of my guests and I do some heavy work on most of them.

[mabnhdev]

OpenSwitch...

I'm running on a Ubuntu 16.04 LTS 64-bit guest with 4 CPUs and 4GB memory configured. If you only have one CPU, it will take much longer and may not trigger the event.

Prerequisites for the build environment...
> sudo apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib build-essential chrpath screen curl device-tree-compiler libsdl1.2-dev xterm libfontconfig1

To build...
> git clone https://git.openswitch.net/openswitch/ops-build.git
> cd ops-build
> make configure genericx86-64
> make

After you see the message 'Building fs (openswitch-disk-image)...'
Very shortly after you see the message 'NOTE: Executing SetScene Tasks', that is the point that I get netdev watchdog crash. It happens every time for me.

[mabnhdev]

I've verified that all power options are off.

[Perryg]

Ah now I see what is going on. There were reports of network issues and multiple vCPUs in a guest. Not sure what ever happened with that but you can search bugtracker tickets and see. I also only use VirtIO-net on guests so it may be something for you to look into. Probably why I don't have network issues like some do.

[Martin]

You are running the guest with four CPU. Do you have more than four physical CPU cores in the host?

[mabnhdev]

Yes, 8 core host machine with 16GB memory.

Windows 7 Enterprise SP1

Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz

[mabnhdev]

Using VirtIO-net helps the situation quite a bit. I'm still getting many network "glitches" during that problematic phase of the build, but the build gracefully recovers from glitches. No more watchdogs, so the build is now making it through. Thanks!

[Martin]

The i7-4810MQ is a quadcore CPU with eight threads. Please try a test with two or three vCPU for the guest if you don't have two of these CPUs in your system.