Strange behavior with packets not reaching guest

Discussions related to using VirtualBox on Windows hosts.
RoyG
Posts: 18
Joined: 18. Mar 2019, 22:14

Strange behavior with packets not reaching guest

Post by RoyG »

My environment: Windows 10 host, Ubuntu 16 and Ubuntu 20 guests. These have been working for a long time without issues.

Yesterday, a problem started where my Ubuntu guest machines could no longer reach any websites outside of my internal (work) network. I think this happened when changing a setting on the host to allow SMB 1.0 connections (should be completely unrelated) and updating Virtualbox to 6.1.16. It may have started happening before this update however.

The behavior I see is that the guests cannot connect to port 80 or port 443 outside of my work network. When I run a Wireshark trace on the guest, I see the SYN packets being sent and then retransmitted over and over. When I run a Wireshark trace on the host, I see the SYN, ACK packets from the remote server being received, and then the retransmissions from both sides. I have turned off the Windows firewall, I have downgraded Virtualbox to 6.0.24, I have reset the SMB setting to what it was before yesterday, but I can't change this behavior. Other ports seem to be working well. I can telnet or ssh outside my internal network, I can ping the same hosts that I can't connect via port 80 or 443 to, NTP seems to be working, etc.

I hope there is someone here who has seen this before, or knows more about Windows than I do and can tell me or help me figure out what changed on my Windows host.
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Strange behavior with packets not reaching guest

Post by scottgus1 »

If you're able to telnet, ssh, ping, etc, then the Virtualbox network is working and the problem probably lies somewhere else.

If you have a work network, did IT put a filter in? Ports 80 and 443 are internet access. Can your guests ping 8.8.8.8 (Google)? If so, they can still access internet and something somewhere not in Virtualbox is blocking those ports.
RoyG
Posts: 18
Joined: 18. Mar 2019, 22:14

Re: Strange behavior with packets not reaching guest

Post by RoyG »

The packets seem to be coming back from the outside world, as I see them in the Wireshark trace on the host system. Ping works fine to everywhere, as I said. I'm sure the problem is not in VirtualBox but I am at a loss to figure out where it could be.

Thanks for any insights.
Last edited by scottgus1 on 29. Oct 2020, 18:23, edited 2 times in total.
Reason: took out quote of complete previous post
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Strange behavior with packets not reaching guest

Post by scottgus1 »

RoyG wrote:I can ping the same hosts that I can't connect via port 80 or 443 to
The above quote is the only mention of ping in the first post, and only mentions these hosts, so it doesn't quite mean you can ping the internet. So,
RoyG wrote:Ping works fine to everywhere, as I said.
Not quite, based on the first post's data.

But since you can ping the internet, look outside Virtualbox for this problem. It appears Virtualbox is working, and Virtualbox does not have any filtering to prevent traffic from getting out of a guest.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Strange behavior with packets not reaching guest

Post by fth0 »

You can create a Wireshark trace at a third position, between the guest and the host, to check if the TCP SYN/ACK packets appear inside the VirtualBox internal network stack. Search the VirtualBox User Manual for the nictrace options.

Additionally, you could tell us some details about the network configuration: Which networking mode do you use (NAT, NAT Network, Bridged)? Are you using an Ethernet or a Wi-Fi network adapter?
RoyG
Posts: 18
Joined: 18. Mar 2019, 22:14

Re: Strange behavior with packets not reaching guest

Post by RoyG »

fth0 wrote:You can create a Wireshark trace at a third position, between the guest and the host, to check if the TCP SYN/ACK packets appear inside the VirtualBox internal network stack. Search the VirtualBox User Manual for the nictrace options.

Additionally, you could tell us some details about the network configuration: Which networking mode do you use (NAT, NAT Network, Bridged)? Are you using an Ethernet or a Wi-Fi network adapter?
All the network configurations are Bridged. I've never had any luck with anything else. The host is hard-wired so I guess that means an Ethernet adapter? Or do you mean in the guest? The guests both have wired connections. It only says Ethernet explicitly in the Ubuntu 16 guest, the Ubuntu 20 guest just says that it is a wired connection. I'll look into the nictrace, thank you.
RoyG
Posts: 18
Joined: 18. Mar 2019, 22:14

Re: Strange behavior with packets not reaching guest

Post by RoyG »

RoyG wrote:
fth0 wrote:You can create a Wireshark trace at a third position, between the guest and the host, to check if the TCP SYN/ACK packets appear inside the VirtualBox internal network stack. Search the VirtualBox User Manual for the nictrace options.

Additionally, you could tell us some details about the network configuration: Which networking mode do you use (NAT, NAT Network, Bridged)? Are you using an Ethernet or a Wi-Fi network adapter?
All the network configurations are Bridged. I've never had any luck with anything else. The host is hard-wired so I guess that means an Ethernet adapter? Or do you mean in the guest? The guests both have wired connections. It only says Ethernet explicitly in the Ubuntu 16 guest, the Ubuntu 20 guest just says that it is a wired connection. I'll look into the nictrace, thank you.
I guess unsurprisingly, the nictrace shows the same thing as Wireshark within the guest OS. That is, the SYN packets leave the guest but no responses are received. I'm sure this problem must be in Windows somewhere but I just don't know where it could possibly be.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Strange behavior with packets not reaching guest

Post by fth0 »

RoyG wrote:The host is hard-wired so I guess that means an Ethernet adapter? Or do you mean in the guest?
Yes, and no. ;)

The reason for asking about a Wi-Fi network adapter was that using bridged mode with a Wi-Fi network adapter can pose some problems (which isn't the case here).

The reason for asking about the networking mode is that VirtualBox behaves quite differently: With bridged mode, VirtualBox injects/captures the frames from/for the guest quite low in the host OS's networking stack (but above the Wireshark NPCAP filter). With the two NAT modes, VirtualBox behaves like a standard host OS application and uses IP sockets to send/receive the packets.
RoyG wrote:I guess unsurprisingly, the nictrace shows the same thing as Wireshark within the guest OS. That is, the SYN packets leave the guest but no responses are received. I'm sure this problem must be in Windows somewhere but I just don't know where it could possibly be.

Can you provide two short Wireshark traces (captured on the host and the guest simultaneously) containing a working and a non-working TCP handshake (SSH, HTTP(S), both to external addresses)? Just in case I see something you don't.
RoyG
Posts: 18
Joined: 18. Mar 2019, 22:14

Re: Strange behavior with packets not reaching guest

Post by RoyG »

fth0 wrote: Can you provide two short Wireshark traces (captured on the host and the guest simultaneously) containing a working and a non-working TCP handshake (SSH, HTTP(S), both to external addresses)? Just in case I see something you don't.
Thanks very much for your offer. Attached is I hope a zip file with the two trace files.
Attachments
trace-files.zip
(8.37 KiB) Downloaded 10 times
Last edited by RoyG on 30. Oct 2020, 00:51, edited 1 time in total.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: Strange behavior with packets not reaching guest

Post by arQon »

> All the network configurations are Bridged.

Ever suspend/sleep the host?
RoyG
Posts: 18
Joined: 18. Mar 2019, 22:14

Re: Strange behavior with packets not reaching guest

Post by RoyG »

arQon wrote:> All the network configurations are Bridged.

Ever suspend/sleep the host?
No, it's a desktop PC in the office, so it's on 24/7.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Strange behavior with packets not reaching guest

Post by fth0 »

Well, it took some time, but here is my analysis:

1. The VirtualBox guest uses the IP address 172.17.0.10 and the MAC address 08:00:27:a8:6b:16. All frames are sent to the destination IP address 104.239.142.24 and the MAC address 00:00:5e:00:01:01, which is a VRRP (Virtual Router Redundancy Protocol) MAC address. This means that the next hop can be one of multiple redundant routers. So far, so good.

2. On the Windows host, the frames are duplicated and additionally sent from the MAC address 70:20:84:0f:e6:53. This is probably the actual problem.

3. In the SSH (and the Two Towers) case, the original TCP SYN packet is answered by a TCP SYN/ACK packet, received from the IP address 104.239.142.24 and the MAC address a0:36:9f:9f:88:76, and sent to the MAC address 08:00:27:a8:6b:16. This works as expected. Additionally, the frame is received after 38 ms, which is typical for an Internet destination.

4. In the HTTP (and HTTPS) case, the duplicate TCP SYN packet is answered by a TCP SYN/ACK packet, received from the IP address 104.239.142.24 and the MAC address 00:00:5e:00:01:01, and sent to the MAC address 70:20:84:0f:e6:53. This doesn't work. Additionally, the frame is received after less than 1 ms, which is typical for a LAN destination.

What does that tell us? There probably is a content filter firewall in your LAN, and either one of the VRRP routers or the firewall is answering the duplicated frames instead of the original frames. But the main question is: Who is duplicating the frames?

What are the IP address and the MAC address of the network adapter in the Windows host?

There is the possibility that the installation of the VirtualBox network filter driver is corrupt. You can proceed as follows: Uninstall VirtualBox (the VMs and settings will not be deleted by this), boot the Windows host, and install VirtualBox again. Are the frames still getting duplicated?
RoyG
Posts: 18
Joined: 18. Mar 2019, 22:14

Re: Strange behavior with packets not reaching guest

Post by RoyG »

fth0 wrote:Well, it took some time, but here is my analysis:
Thank you very much indeed. The MAC address in (2) (source) and (4) (destination) is in fact the MAC address of the host system. Are you saying that (2) is a problem in VirtualBox? or is it a problem in Windows? The source MAC address in (4) being the same as the outbound MAC address seems very suspect to me.

I will contact my IT staff and see if they have mucked things up.

Cheers...
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Strange behavior with packets not reaching guest

Post by fth0 »

RoyG wrote:Are you saying that (2) is a problem in VirtualBox? or is it a problem in Windows?
Yes, that's the question, and the last paragraph of my previous post was an attempt to find out ...
RoyG
Posts: 18
Joined: 18. Mar 2019, 22:14

Re: Strange behavior with packets not reaching guest

Post by RoyG »

fth0 wrote: You can proceed as follows: Uninstall VirtualBox (the VMs and settings will not be deleted by this), boot the Windows host, and install VirtualBox again. Are the frames still getting duplicated?
I did this, with no change in the behavior. (I went from 6.0.24, which I installed last week in the hope that it was something in 6.1 which was broken, to 6.1.12, which is the version I had installed the last time this was working.)

Another weirdness in this whole thing is that for sites on our internal network, the duplicate SYN packets are still sent, but the response comes to the initial packet not the duplicate packet, so internal sites work. Perhaps this is because the remote host can respond to the initial SYN packet before the duplicate is received?

I'm going to have my IT staff set up another Windows PC host with VirtualBox and an Ubuntu VM, and see what happens on that machine too.

Thanks once more for looking at this.
Post Reply