NAT connection stops ARPing - NAT: Can't allocate mbuf
Posted: 16. Mar 2015, 23:47
Hello,
We have virtual machines on a number of hosts (all laptops running OSX) exhibiting a network connection issue.
Machines are configured with a single network adapter set to 'NAT'. There are also about 10 port forwards set up to connect into the machine. When first turned on they function as expected, but after some interval (not predictable or consistent) we lose network access to the VM. It is impossible to connect to any of the forwarded ports. We can still access the VM via remote desktop, and when we do so we see the machine cannot make outbound connections either.
Investigating with TCP dump shows that the machine stops receiving responses to its ARP requests. When we attempt to establish a connection to the VM from outside we see the SYN packet arrive with the VM. The VM then sends an ARP request to the gateway IP but gets no response. We have shown that manually adding a relevant entry to the ARP table restores connections. When this happens the VBox log contains "NAT: Can't allocate mbuf". There is no sign of any other allocations failing on these machines.
I have attached a copy of VBox.log corresponding to this issue.
We have found we can restore connectivity by turning nictrace on and off again, but it will then fail again at some later point.
This started happening around the beginning of March. We have been unable to figure out what if anything changed then. We have reproduced the issue with both VirtualBox 4.3.20 and 4.3.22.
We have also today once observed a full VBoxHeadless crash that has some signs of being related. It was in the NAT thread, and was an attempt to access memory at 0xa, suggesting it could be the result of an unchecked alloc failure. I've attached the crash report.
Does anyone have any suggestions for what we can do to resolve or better diagnose this issue?
We have virtual machines on a number of hosts (all laptops running OSX) exhibiting a network connection issue.
Machines are configured with a single network adapter set to 'NAT'. There are also about 10 port forwards set up to connect into the machine. When first turned on they function as expected, but after some interval (not predictable or consistent) we lose network access to the VM. It is impossible to connect to any of the forwarded ports. We can still access the VM via remote desktop, and when we do so we see the machine cannot make outbound connections either.
Investigating with TCP dump shows that the machine stops receiving responses to its ARP requests. When we attempt to establish a connection to the VM from outside we see the SYN packet arrive with the VM. The VM then sends an ARP request to the gateway IP but gets no response. We have shown that manually adding a relevant entry to the ARP table restores connections. When this happens the VBox log contains "NAT: Can't allocate mbuf". There is no sign of any other allocations failing on these machines.
I have attached a copy of VBox.log corresponding to this issue.
We have found we can restore connectivity by turning nictrace on and off again, but it will then fail again at some later point.
This started happening around the beginning of March. We have been unable to figure out what if anything changed then. We have reproduced the issue with both VirtualBox 4.3.20 and 4.3.22.
We have also today once observed a full VBoxHeadless crash that has some signs of being related. It was in the NAT thread, and was an attempt to access memory at 0xa, suggesting it could be the result of an unchecked alloc failure. I've attached the crash report.
Does anyone have any suggestions for what we can do to resolve or better diagnose this issue?