Strange behavior with packets not reaching guest
Strange behavior with packets not reaching guest
My environment: Windows 10 host, Ubuntu 16 and Ubuntu 20 guests. These have been working for a long time without issues.
Yesterday, a problem started where my Ubuntu guest machines could no longer reach any websites outside of my internal (work) network. I think this happened when changing a setting on the host to allow SMB 1.0 connections (should be completely unrelated) and updating Virtualbox to 6.1.16. It may have started happening before this update however.
The behavior I see is that the guests cannot connect to port 80 or port 443 outside of my work network. When I run a Wireshark trace on the guest, I see the SYN packets being sent and then retransmitted over and over. When I run a Wireshark trace on the host, I see the SYN, ACK packets from the remote server being received, and then the retransmissions from both sides. I have turned off the Windows firewall, I have downgraded Virtualbox to 6.0.24, I have reset the SMB setting to what it was before yesterday, but I can't change this behavior. Other ports seem to be working well. I can telnet or ssh outside my internal network, I can ping the same hosts that I can't connect via port 80 or 443 to, NTP seems to be working, etc.
I hope there is someone here who has seen this before, or knows more about Windows than I do and can tell me or help me figure out what changed on my Windows host.
Yesterday, a problem started where my Ubuntu guest machines could no longer reach any websites outside of my internal (work) network. I think this happened when changing a setting on the host to allow SMB 1.0 connections (should be completely unrelated) and updating Virtualbox to 6.1.16. It may have started happening before this update however.
The behavior I see is that the guests cannot connect to port 80 or port 443 outside of my work network. When I run a Wireshark trace on the guest, I see the SYN packets being sent and then retransmitted over and over. When I run a Wireshark trace on the host, I see the SYN, ACK packets from the remote server being received, and then the retransmissions from both sides. I have turned off the Windows firewall, I have downgraded Virtualbox to 6.0.24, I have reset the SMB setting to what it was before yesterday, but I can't change this behavior. Other ports seem to be working well. I can telnet or ssh outside my internal network, I can ping the same hosts that I can't connect via port 80 or 443 to, NTP seems to be working, etc.
I hope there is someone here who has seen this before, or knows more about Windows than I do and can tell me or help me figure out what changed on my Windows host.
-
- Site Moderator
- Posts: 20945
- Joined: 30. Dec 2009, 20:14
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Windows, Linux
Re: Strange behavior with packets not reaching guest
If you're able to telnet, ssh, ping, etc, then the Virtualbox network is working and the problem probably lies somewhere else.
If you have a work network, did IT put a filter in? Ports 80 and 443 are internet access. Can your guests ping 8.8.8.8 (Google)? If so, they can still access internet and something somewhere not in Virtualbox is blocking those ports.
If you have a work network, did IT put a filter in? Ports 80 and 443 are internet access. Can your guests ping 8.8.8.8 (Google)? If so, they can still access internet and something somewhere not in Virtualbox is blocking those ports.
Re: Strange behavior with packets not reaching guest
The packets seem to be coming back from the outside world, as I see them in the Wireshark trace on the host system. Ping works fine to everywhere, as I said. I'm sure the problem is not in VirtualBox but I am at a loss to figure out where it could be.
Thanks for any insights.
Thanks for any insights.
Last edited by scottgus1 on 29. Oct 2020, 18:23, edited 2 times in total.
Reason: took out quote of complete previous post
Reason: took out quote of complete previous post
-
- Site Moderator
- Posts: 20945
- Joined: 30. Dec 2009, 20:14
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Windows, Linux
Re: Strange behavior with packets not reaching guest
The above quote is the only mention of ping in the first post, and only mentions these hosts, so it doesn't quite mean you can ping the internet. So,RoyG wrote:I can ping the same hosts that I can't connect via port 80 or 443 to
Not quite, based on the first post's data.RoyG wrote:Ping works fine to everywhere, as I said.
But since you can ping the internet, look outside Virtualbox for this problem. It appears Virtualbox is working, and Virtualbox does not have any filtering to prevent traffic from getting out of a guest.
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Strange behavior with packets not reaching guest
You can create a Wireshark trace at a third position, between the guest and the host, to check if the TCP SYN/ACK packets appear inside the VirtualBox internal network stack. Search the VirtualBox User Manual for the nictrace options.
Additionally, you could tell us some details about the network configuration: Which networking mode do you use (NAT, NAT Network, Bridged)? Are you using an Ethernet or a Wi-Fi network adapter?
Additionally, you could tell us some details about the network configuration: Which networking mode do you use (NAT, NAT Network, Bridged)? Are you using an Ethernet or a Wi-Fi network adapter?
Re: Strange behavior with packets not reaching guest
All the network configurations are Bridged. I've never had any luck with anything else. The host is hard-wired so I guess that means an Ethernet adapter? Or do you mean in the guest? The guests both have wired connections. It only says Ethernet explicitly in the Ubuntu 16 guest, the Ubuntu 20 guest just says that it is a wired connection. I'll look into the nictrace, thank you.fth0 wrote:You can create a Wireshark trace at a third position, between the guest and the host, to check if the TCP SYN/ACK packets appear inside the VirtualBox internal network stack. Search the VirtualBox User Manual for the nictrace options.
Additionally, you could tell us some details about the network configuration: Which networking mode do you use (NAT, NAT Network, Bridged)? Are you using an Ethernet or a Wi-Fi network adapter?
Re: Strange behavior with packets not reaching guest
I guess unsurprisingly, the nictrace shows the same thing as Wireshark within the guest OS. That is, the SYN packets leave the guest but no responses are received. I'm sure this problem must be in Windows somewhere but I just don't know where it could possibly be.RoyG wrote:All the network configurations are Bridged. I've never had any luck with anything else. The host is hard-wired so I guess that means an Ethernet adapter? Or do you mean in the guest? The guests both have wired connections. It only says Ethernet explicitly in the Ubuntu 16 guest, the Ubuntu 20 guest just says that it is a wired connection. I'll look into the nictrace, thank you.fth0 wrote:You can create a Wireshark trace at a third position, between the guest and the host, to check if the TCP SYN/ACK packets appear inside the VirtualBox internal network stack. Search the VirtualBox User Manual for the nictrace options.
Additionally, you could tell us some details about the network configuration: Which networking mode do you use (NAT, NAT Network, Bridged)? Are you using an Ethernet or a Wi-Fi network adapter?
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Strange behavior with packets not reaching guest
Yes, and no.RoyG wrote:The host is hard-wired so I guess that means an Ethernet adapter? Or do you mean in the guest?
The reason for asking about a Wi-Fi network adapter was that using bridged mode with a Wi-Fi network adapter can pose some problems (which isn't the case here).
The reason for asking about the networking mode is that VirtualBox behaves quite differently: With bridged mode, VirtualBox injects/captures the frames from/for the guest quite low in the host OS's networking stack (but above the Wireshark NPCAP filter). With the two NAT modes, VirtualBox behaves like a standard host OS application and uses IP sockets to send/receive the packets.
RoyG wrote:I guess unsurprisingly, the nictrace shows the same thing as Wireshark within the guest OS. That is, the SYN packets leave the guest but no responses are received. I'm sure this problem must be in Windows somewhere but I just don't know where it could possibly be.
Can you provide two short Wireshark traces (captured on the host and the guest simultaneously) containing a working and a non-working TCP handshake (SSH, HTTP(S), both to external addresses)? Just in case I see something you don't.
Re: Strange behavior with packets not reaching guest
Thanks very much for your offer. Attached is I hope a zip file with the two trace files.fth0 wrote: Can you provide two short Wireshark traces (captured on the host and the guest simultaneously) containing a working and a non-working TCP handshake (SSH, HTTP(S), both to external addresses)? Just in case I see something you don't.
- Attachments
-
- trace-files.zip
- (8.37 KiB) Downloaded 10 times
Last edited by RoyG on 30. Oct 2020, 00:51, edited 1 time in total.
-
- Posts: 231
- Joined: 1. Jan 2017, 09:16
- Primary OS: MS Windows 7
- VBox Version: PUEL
- Guest OSses: Ubuntu 16.04 x64, W7
Re: Strange behavior with packets not reaching guest
> All the network configurations are Bridged.
Ever suspend/sleep the host?
Ever suspend/sleep the host?
Re: Strange behavior with packets not reaching guest
No, it's a desktop PC in the office, so it's on 24/7.arQon wrote:> All the network configurations are Bridged.
Ever suspend/sleep the host?
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Strange behavior with packets not reaching guest
Well, it took some time, but here is my analysis:
1. The VirtualBox guest uses the IP address 172.17.0.10 and the MAC address 08:00:27:a8:6b:16. All frames are sent to the destination IP address 104.239.142.24 and the MAC address 00:00:5e:00:01:01, which is a VRRP (Virtual Router Redundancy Protocol) MAC address. This means that the next hop can be one of multiple redundant routers. So far, so good.
2. On the Windows host, the frames are duplicated and additionally sent from the MAC address 70:20:84:0f:e6:53. This is probably the actual problem.
3. In the SSH (and the Two Towers) case, the original TCP SYN packet is answered by a TCP SYN/ACK packet, received from the IP address 104.239.142.24 and the MAC address a0:36:9f:9f:88:76, and sent to the MAC address 08:00:27:a8:6b:16. This works as expected. Additionally, the frame is received after 38 ms, which is typical for an Internet destination.
4. In the HTTP (and HTTPS) case, the duplicate TCP SYN packet is answered by a TCP SYN/ACK packet, received from the IP address 104.239.142.24 and the MAC address 00:00:5e:00:01:01, and sent to the MAC address 70:20:84:0f:e6:53. This doesn't work. Additionally, the frame is received after less than 1 ms, which is typical for a LAN destination.
What does that tell us? There probably is a content filter firewall in your LAN, and either one of the VRRP routers or the firewall is answering the duplicated frames instead of the original frames. But the main question is: Who is duplicating the frames?
What are the IP address and the MAC address of the network adapter in the Windows host?
There is the possibility that the installation of the VirtualBox network filter driver is corrupt. You can proceed as follows: Uninstall VirtualBox (the VMs and settings will not be deleted by this), boot the Windows host, and install VirtualBox again. Are the frames still getting duplicated?
1. The VirtualBox guest uses the IP address 172.17.0.10 and the MAC address 08:00:27:a8:6b:16. All frames are sent to the destination IP address 104.239.142.24 and the MAC address 00:00:5e:00:01:01, which is a VRRP (Virtual Router Redundancy Protocol) MAC address. This means that the next hop can be one of multiple redundant routers. So far, so good.
2. On the Windows host, the frames are duplicated and additionally sent from the MAC address 70:20:84:0f:e6:53. This is probably the actual problem.
3. In the SSH (and the Two Towers) case, the original TCP SYN packet is answered by a TCP SYN/ACK packet, received from the IP address 104.239.142.24 and the MAC address a0:36:9f:9f:88:76, and sent to the MAC address 08:00:27:a8:6b:16. This works as expected. Additionally, the frame is received after 38 ms, which is typical for an Internet destination.
4. In the HTTP (and HTTPS) case, the duplicate TCP SYN packet is answered by a TCP SYN/ACK packet, received from the IP address 104.239.142.24 and the MAC address 00:00:5e:00:01:01, and sent to the MAC address 70:20:84:0f:e6:53. This doesn't work. Additionally, the frame is received after less than 1 ms, which is typical for a LAN destination.
What does that tell us? There probably is a content filter firewall in your LAN, and either one of the VRRP routers or the firewall is answering the duplicated frames instead of the original frames. But the main question is: Who is duplicating the frames?
What are the IP address and the MAC address of the network adapter in the Windows host?
There is the possibility that the installation of the VirtualBox network filter driver is corrupt. You can proceed as follows: Uninstall VirtualBox (the VMs and settings will not be deleted by this), boot the Windows host, and install VirtualBox again. Are the frames still getting duplicated?
Re: Strange behavior with packets not reaching guest
Thank you very much indeed. The MAC address in (2) (source) and (4) (destination) is in fact the MAC address of the host system. Are you saying that (2) is a problem in VirtualBox? or is it a problem in Windows? The source MAC address in (4) being the same as the outbound MAC address seems very suspect to me.fth0 wrote:Well, it took some time, but here is my analysis:
I will contact my IT staff and see if they have mucked things up.
Cheers...
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Strange behavior with packets not reaching guest
Yes, that's the question, and the last paragraph of my previous post was an attempt to find out ...RoyG wrote:Are you saying that (2) is a problem in VirtualBox? or is it a problem in Windows?
Re: Strange behavior with packets not reaching guest
I did this, with no change in the behavior. (I went from 6.0.24, which I installed last week in the hope that it was something in 6.1 which was broken, to 6.1.12, which is the version I had installed the last time this was working.)fth0 wrote: You can proceed as follows: Uninstall VirtualBox (the VMs and settings will not be deleted by this), boot the Windows host, and install VirtualBox again. Are the frames still getting duplicated?
Another weirdness in this whole thing is that for sites on our internal network, the duplicate SYN packets are still sent, but the response comes to the initial packet not the duplicate packet, so internal sites work. Perhaps this is because the remote host can respond to the initial SYN packet before the duplicate is received?
I'm going to have my IT staff set up another Windows PC host with VirtualBox and an Ubuntu VM, and see what happens on that machine too.
Thanks once more for looking at this.