virtualbox.org

Environment:

10.8.0 OSX running VirtualBox: 4.2.12 r84980

Guest OS: Oracle Enterprise Linux 5.9 x86_64
192.168.1.140 /24

Guest NIC: Intel Pro / 1000 MT Desktop All VM Promiscuous Mode
Bridged Adapter Connected to my En0 interface

Wireless Router/DHCP Server - DD-WRT v24-sp1 (07/27/08) std


Symptoms:
From the OSX terminal that's running Virtual Box, no problems accessing my VMs via SSH. All the rest of the test below were done from another OSX 10.8.0 client and not the one that's running Virtual Box.


From another OSX client on my subnet, I can only ping and get a good arp on the guest OS for 1 min after bootup.


Now, this is happening because at bootup, the interface of the VM is talking to the gateway while getting its DHCP and getting a new ARP entry on the DD-WRT router.

Now, this arp only lasts for 60 seconds (this is the default arp timeout setting on the DD-WRT :
cat /proc/sys/net/ipv4/neigh/`nvram get wan_ifname`/gc_stale_time

So, that makes sense. During those 60 seconds, the Guest VM can ping his gateway and my other OSX client can ping the Guest VM.

The arp entry on the OSX Client shows that IP/ARP as good and accordingly gets a good ping:

hawk_ws:~ hawk$ arp -a
dd-wrt (192.168.1.1) at 0:24:a5:6f:55:d4 on en0 ifscope [ethernet]
? (192.168.1.120) at 90:2b:34:68:ca:91 on en0 ifscope [ethernet]
? (192.168.1.133) at 0:25:22:60:d4:91 on en0 ifscope [ethernet]
? (192.168.1.140) at 8:0:27:4f:95:9b on en0 ifscope [ethernet]
? (192.168.1.148) at 0:d:88:ba:93:de on en0 ifscope [ethernet]

hawk_ws:~ hawk$ ping 192.168.1.140
PING 192.168.1.140 (192.168.1.140): 56 data bytes
64 bytes from 192.168.1.140: icmp_seq=0 ttl=64 time=0.512 ms
c64 bytes from 192.168.1.140: icmp_seq=1 ttl=64 time=0.587 ms
^C
--- 192.168.1.140 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.512/0.549/0.587/0.037 ms


The Guest Os is 192.168.1.140.


Now, I wait 60 seconds for the arp to timeout on DD-WRT and naturally, the VM is supposed to re-arp. I will test this by pinging the VM from another server in the subnet. If the arp is still present on the DD-WRT arp-table, then it will ping fine. If the arp was never added back after the expiration (which should be automatic), then the ping will fail

60 seconds later (from the OSX Client)
hawk_ws:~ hawk$ ping 192.168.1.140
PING 192.168.1.140 (192.168.1.140): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
^C
--- 192.168.1.140 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

The ping failed to the Guest VM. Looking at the arp table on OSX Client:
hawk_ws:~ hawk$ arp -a
dd-wrt (192.168.1.1) at 0:24:a5:6f:55:d4 on en0 ifscope [ethernet]
? (192.168.1.120) at 90:2b:34:68:ca:91 on en0 ifscope [ethernet]
? (192.168.1.133) at 0:25:22:60:d4:91 on en0 ifscope [ethernet]
? (192.168.1.140) at (incomplete) on en0 ifscope [ethernet]
? (192.168.1.148) at 0:d:88:ba:93:de on en0 ifscope [ethernet]

We see an incomplete arp for the Guest VM's IP address. So the OSX client tried to send a message, but never got a response.


Once, I go back to the Guest VM and ping the gateway (thus forcing a re-arp):

My arp table on my OSX client sees the Guest VM as good:
hawk_ws:~ hawk$ arp -a
dd-wrt (192.168.1.1) at 0:24:a5:6f:55:d4 on en0 ifscope [ethernet]
? (192.168.1.120) at 90:2b:34:68:ca:91 on en0 ifscope [ethernet]
? (192.168.1.133) at 0:25:22:60:d4:91 on en0 ifscope [ethernet]
? (192.168.1.140) at 8:0:27:4f:95:9b on en0 ifscope [ethernet]
? (192.168.1.148) at 0:d:88:ba:93:de on en0 ifscope [ethernet]

And when I try to ping it:
hawk_ws:~ hawk$ ping 192.168.1.140
PING 192.168.1.140 (192.168.1.140): 56 data bytes
64 bytes from 192.168.1.140: icmp_seq=0 ttl=64 time=0.527 ms
64 bytes from 192.168.1.140: icmp_seq=1 ttl=64 time=0.534 ms
c64 bytes from 192.168.1.140: icmp_seq=2 ttl=64 time=0.506 ms
^C
--- 192.168.1.140 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.506/0.522/0.534/0.012 ms

All is good. So I know the Guest VM is not re-arping like it should.

Now the issue will repeat over and over each 60 seconds (as long as I force the VM Guest to ping his gateway to force the re-arp)


Changing the arp timeout on the DD-WRT wireless router is not the answer. This has to do with why VirtualBox isn't allowing its Guest VMs to re-arp naturally. Or its due to the way Bridged Networking is working on OSX 10.8.0.

I've seen many other posts about the same problem, but this is a more detailed workflow of seeing why and how the arp is not getting re-arped after normal arp cache expiration times.

AFAIK it is not your VM that will do the "re-arp" but your dd-wrt that will send an ARP request if it doesn't have a match in its ARP table for a given IP address.
But in your case it wouldn't be the ARP, but the MAC table used in the switching, but even that doesn't make sense in your case.

Unless there is firewalling prevent any kind of reply from your CentOS, in which case there will never be an update of the router's ARP/MAC table since no request will ever be answerd. But on the other hand, as soon as your guest talks, its MAC/IP will be registered in the cache, and obviously it will work.

So my first guest would be firewalling somewhere.

Well, I'm sure there are no firewalls blocking ICMP (ping). All the linux servers (including the OEL server I stood up all have iptables turned off) as well as my MAC(s). My DDWRT router isn't blocking any requests and since its all the same same subnet anyway, it wouldn't really matter.

So, I tried the exact same test again, but used my

OSX 10.7.3 Desktop w/ VirtualBox
Windows 7 VM
OEL 5.9 x86_64 VM

With all the same config settings for Network Bridging.


Then I used my OSX 10.8.0 as the one who tries to ping the Guest VMs.


I have the exact same issue. I'm thinking it may be due to my router and how its handling its arp table. So I'll see if switching that out helps.

So far, nobody reported issues of the sort with Virtualbox. But since DDWRT is a open source firmware, it's possible some bugs are still around, and this would look like one. I would try with another switch/router first.

So I just switched out my DDWRT Router with a Linksys/Cisco WRT54GL that I just flashed with the latest DDWRT Firmware.

I fired up the testing again after setting up the new Device as my head-end router.


Everything is now working fine. So it looks like the old DDWRT linux OS may have had some issues with its internal handling of the ARP tables on the existing firmware version.


Issue solved!