No Ethernet Ubuntu/Windows guest VM's after computer wake from sleep

[bertje]

The Ethernet connection in the guest is lost after resuming the host computer from sleep. Before entering sleep mode of the Windows 10 host, Ethernet works on the host and guest.

Setup:

• VirtualBox v6.1.32
• Windows 10 20H2 host (updates installed of today)
• Ubuntu 21.04 guest (updates + guest additions installed)
• VM Intel (R) i211 Gigabit Network connection (default)
• VM Bridge network

Investigated:

• Guest VM uncheck/check "Connect Network Adapter" does not work.
• Disabling/enabling guest Ethernet does not work.
• Unplug/reconnect Ethernet cable host computer does not work.
• Reboot the host PC to get network working in the guest. (Not a solution)
• Reproducible on AMD Ryzen 7/9 and Intel Core I7 laptops/desktops.

Any suggestions how to proceed?

[scottgus1]

I'm not certain how to troubleshoot this myself, but whoever does will need a VM log showing the entire sequence from cold VM boot with good network, through host sleep and wakeup and network loss in VM, to full shutdown of the VM.

Start the VM from full normal shutdown, not save-state. Run until you see the problem happen, then shut down the VM from within the VM's OS if possible. If not possible, close the Virtualbox window for the VM with the Power Off option set.

Right-click the VM in the main Virtualbox window's VM list, choose Show Log. Save the far left tab's log, zip it, and post the zip file, using the forum's Upload Attachment tab.

[bertje]

Thanks for your fast reply. Please find attached the zipped logs as you requested.

Additional info

• Also reproducible with VirtualBox v6.1.30.
• Ethernet works after computer wake-up from sleep and restarting each Ubuntu guest VM.
• When running multiple Ubuntu VM's in parallel (I've 3 running), the Ethernet connection is lost in all VM's after computer wake-up.
• When creating the logs, I noticed one time a good wake-up where only one Ubuntu guest VM was running, so it looks like random behavior when Ethernet is lost.

Logs in Logs-EthernetWorks: Logs after full normal power-on, Ethernet works, graceful shutdown via Guest OS.
Logs in Logs-EthernetBrokenAfterWake: Logs after full normal power-on, computer sleep, computer wake-up, Ethernet guest works, computer sleep, computer wake, Ethernet guest lost, graceful shutdown via Guest OS.

[bertje]

More details after broken Ethernet connection in VM guest(s) after computer wake-up from sleep:

• Computer Ethernet cable disconnect/reconnect does not work
• Disable/re-enable Ethernet in Ubuntu VM guest | Settings | Network | Wired does not work
• Disable/re-enable Ethernet in Windows 10 host Control Panel | Network Connections does not work
• Save machine state Ubuntu VM, restart computer (host), restore machine state VM, disable/re-enable Ethernet in VM guest -> Ethernet works
• Save machine state Ubuntu VM, set computer (host) in sleep mode, wake-up, restore machine state VM -> Ethernet works

Maybe related to viewtopic.php?f=3&t=105687 (Ubuntu (Guest), Windows 11 (Host) Bridge Connection problem)?

It is fully reproducible with latest VirtualBox v6.1.32 on all computers/laptops I tested so far.
Oracle / VirtualBox, can you please try to reproduce this problem? Requested logs are included in my previous post. Thanks.

[scottgus1]

I looked at the logs, and I noticed that the log showing the failed network actually runs for only 5 seconds and does not show that the VM OS booted. Might need to run the tests again.

Could you please try a couple of things:

1. Make a Windows VM, no need to activate the OS, and see if the network returns or not after host sleep?

2. In the Linux VM, after network has failed to return, run this command:

/etc/init.d/network restart && dhclient

Please let us know what happens for each test.

[bertje]

This command does not work on Ubuntu:

Code: Select all

$ sudo /etc/init.d/network restart && dhclient
sudo: /etc/init.d/network: command not found

$ sudo systemctl restart NetworkManager.service
# <= No Ethernet

$ sudo nmcli networking off
$ sudo nmcli networking on
# <= No Ethernet

# Link up/down
<= No Ethernet

Please find attached the logs of Windows 10 guest:

• Boot Windows 10 host computer
• Boot Windows 10 guest
• Set computer in sleep mode
• Wake computer from sleep mode => No Ethernet Windows 10 guest
• Graceful shutdown Windows 10 guest
• <== Attached VBoxHardening.log
• -------------------------------------
• Reboot computer
• Save machine state guests or power-off guests
• Restart computer
• <== Ethernet works in all guests

Note: Guest additions + latest Windows updates are installed in the Windows 10 guest.

Repeat steps. 75% change of broken Ethernet in any Linux/Windows guest after wake-up.

The problem looks like located on Windows 10 host and not in the guest.
Reproducible with Linux and Windows (changed title of this issue).

Are there any services on Windows 10 host which can be restarted or investigated?

[scottgus1]

I regret that the hardening log isn't the one that was needed. The far left tab's log will be the one to be used. You don't have any hardening issues, though; that's good.

It looks like you tried other Ubuntu-specific commands to take the network up and down inside the VM, good. Also, the Windows VM having the same problem does indicate it may be a host issue.

Virtualbox Bridged uses a driver and relies on the host network driver playing nicely together. I unfortunately don't know how to troubleshoot the problem. I haven't found a way to reset the Windows host network without restarting the host. Maybe a Windows forum could help with that one.

Moving to Windows hosts, as this is probably a host issue.

[bertje]

Please find attached the saved VBox.log via the VM Show Log... window.

[scottgus1]

Thanks for the log. Unfortunately, better eyes than mine will have to loo it over. I can't see any network problems in it.

This might be happening in the host network outside of Virtualbox. Try the test again, and look through the Windows Event Viewer to see if any network issues show themselves.

********************
One side issue I see that may or may not be causing the issue but might be causing slower VM operation:

00:00:01.925635 VirtualBox VM 6.1.30 r148432 win.amd64 (Nov 22 2021 15:15:30) release log
00:00:02.218386 HM: HMR3Init: AMD-V w/ nested paging
00:03:24.668467 /TM/VirtualSync/CurrentOffset 60016505757 ns

Your VM has over 60 seconds of offset, which is prime indication of the 6.1 Hyper-V bug that crept in around 6.1.28, when Windows 11 came out.

Our forum guru 'fth0' has done some troubleshooting and pins down these possibilities to try (HCVI = Microsoft 'Memory Integrity'):
6.1.32 without Hyper-V
6.1.30 with Hyper-V but without HVCI
6.1.26 with Hyper-V and HVCI.

[bertje]

Relevant warnings/errors Windows 10 Guest VM Event log:

Code: Select all

Event ID: 1003
Your computer was not able to renew its address from the network (from the DHCP Server) for the Network Card with network address 0x080027D99C3B.  The following error occurred: 0x79. Your computer will continue to try and obtain an address on its own from the network address (DHCP) server.

Event ID: 1014
Name resolution for the name wpad timed out after none of the configured DNS servers responded.

Event ID: 10005
DCOM got error "1115" attempting to start the service SecurityHealthService with arguments "Unavailable" in order to run the server:
{8C9C0DB7-2CBA-40F1-AFE0-C55740DD91A0}

Relevant warnings/errors Windows 10 Host Event log:

Code: Select all

Event ID: 1001, Status code: 121
Your computer was not assigned an address from the network (by the DHCP Server) for the Network Card with network address 0x0A002700000E.  The following error occurred: 0x79. Your computer will continue to try and obtain an address on its own from the network address (DHCP) server.

Event ID:7043
The Intel(R) PROSet Monitoring Service service did not shut down properly after receiving a pre-shutdown control.

[fth0]

Can you capture all DHCP negotiations either on the DHCP server or somewhere in between the host and the DHCP server?

[bertje]

Ethernet receive stops working in all VM's at the same time after computer wake-up. See Wireshark output below:

Code: Select all

ICMP packet from guest to gateway:
0000   00 00 03 04 00 06 00 00 00 00 00 00 01 3d 08 00   .............=..
0010   45 c0 00 70 2e 62 00 00 40 01 36 2a 0a 00 00 a1   E..p.b..@.6*....
0020   0a 00 00 a1 03 01 fc fe 00 00 00 00 45 00 00 54   ............E..T
0030   a8 8d 40 00 40 01 7c f1 0a 00 00 a1 0a 00 00 8a   ..@.@.|.........
0040   08 00 2e 71 00 02 00 2e 1c cc 6a 62 00 00 00 00   ...q......jb....
0050   7d 5d 06 00 00 00 00 00 10 11 12 13 14 15 16 17   }]..............
0060   18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27   ........ !"#$%&'
0070   28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37   ()*+,-./01234567

No response

ARP request from guest to gateway:
0000   00 01 00 01 00 06 08 00 27 3c 32 76 00 00 08 06   ........'<2v....
0010   00 01 08 00 06 04 00 01 08 00 27 3c 32 76 0a 00   ..........'<2v..
0020   00 c0 00 00 00 00 00 00 0a 00 00 8a 00 00 00 00   ................
0030   00 00 00 00 00 00 00 00 00 00 00 00 00 00         ..............

No response

DHCP discover from guest to network:
0000   00 04 00 01 00 06 08 00 27 6e 14 fc 00 00 08 00   ........'n......
0010   45 c0 01 3b 00 00 40 00 40 11 38 f3 00 00 00 00   E..;..@.@.8.....
0020   ff ff ff ff 00 44 00 43 01 27 35 4c 01 01 06 00   .....D.C.'5L....
0030   1d 52 41 b8 00 40 00 00 00 00 00 00 00 00 00 00   .RA..@..........
0040   00 00 00 00 00 00 00 00 08 00 27 6e 14 fc 00 00   ..........'n....
0050   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0060   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0070   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0080   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0090   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00a0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00b0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00c0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00d0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00e0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00f0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0100   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0110   00 00 00 00 00 00 00 00 63 82 53 63 35 01 01 3d   ........c.Sc5..=
0120   07 01 08 00 27 6e 14 fc 37 11 01 02 06 0c 0f 1a   ....'n..7.......
0130   1c 79 03 21 28 29 2a 77 f9 fc 11 39 02 02 40 32   .y.!()*w...9..@2
0140   04 0a 00 00 a1 0c 03 61 30 32 ff                  .......a02.

No response

Also changing Ethernet from DHCP in the VM to a fixed IP address does not help.

Please notice that the VM is configured in Bridge network

I'm not able to capture DHCP Ethernet traffic on the DHCP server.

No Ethernet in/out trafic captured with Wireshark between host and VM.

The computer should be restarted in order to recover from this Ethernet hang-up situation. Restarting each VM does not recover.

[fth0]

At first sight (*), the packet dumps look a bit strange to me: All Ethernet frames start with an unusual structured Ethernet header (two bytes inserted before the EtherType?). The first packet is an ICMP Destination Unreachable sent from 10.0.0.161 to itself.

(*) I just looked at the hexdumps, because I was too lazy to convert them back into a PCAP file.

Since you obviously know Wireshark, can you provide (zipped) PCAP files containing the working DHCP negotiation and the non-working case? Of course, you're allowed to provide more packets/frames. I'd like to check if the frames are different in crucial parts. Perhaps you can capture on the host and inside the guest simultaneously, by playing with the (virtual) Ethernet cable and/or pausing the guest before the host sleep for a coordinated host/guest wakeup ...

[bertje]

Please find attached the Wireshark PCAP file in the VM as you requested with linkdown and linkup and DHCP requests before and after computer wake. I only exported relevant packets. Same behavior Ubuntu and Windows 10 guest.

Disconnecting (virtual) Ethernet cable or setting VM in paused mode before enter computer sleep mode does not make sense.

Saving the machine state (guest) before entering sleep mode, computer wake, restore VM machine state (guest) Ethernet keeps working. (I tested this several times, hopefully no lucky shots...)

Ethernet on the host is still working after each computer wake.

I could not capture any Ethernet pack between the VM and host after computer wakeup, so it looks like located in the host VirtualBox Ethernet handling which enters an unrecoverable state on TX and RX.

I'm wondering, are you able to reproduce this problem? The testcase is very simple and we're able to reproduce it on several computers and laptops.

[bertje]

I've reported a new defect https://www.virtualbox.org/ticket/20922 to find a solution.