Can't get public and private subnets to work

[Craig5479Anderson]

I am trying to create a 2-node DB cluster (Oracle 11gR2 RAC) on a Win 7 host (8 CPU, 16 GB RAM). I have 2 guests stood up with Oracle Linux 6.3. Each has 3 network adapters, all with Type=PCnet-FAST III (Am79C973), as follows:
1 - NAT
2 - Host-Only, Name=VirtualBox Host-Only Ethernet Adapter, Promiscuous Mode=Allow VMs
3 - Host-Only, Name=VirtualBox Host-Only Ethernet Adapter, Promiscuous Mode=Allow VMs

Adapters 1, 2 and 3 are set up in each VM the same way, as eth0, 1, and 2. Both guests also have the same hosts file as shown below. VM #1 is RAC1 (ol6-112-rac1) and Node# 2 is RAC2 (ol6-112-rac2)

------ /etc/hosts ----------
127.0.0.1 localhost.localdomain localhost
# Public
192.168.0.111 ol6-112-rac1.localdomain ol6-112-rac1
192.168.0.112 ol6-112-rac2.localdomain ol6-112-rac2
# Private
192.168.1.111 ol6-112-rac1-priv.localdomain ol6-112-rac1-priv
192.168.1.112 ol6-112-rac2-priv.localdomain ol6-112-rac2-priv
# Virtual
192.168.0.113 ol6-112-rac1-vip.localdomain ol6-112-rac1-vip
192.168.0.114 ol6-112-rac2-vip.localdomain ol6-112-rac2-vip
# SCAN
192.168.0.115 ol6-112-scan.localdomain ol6-112-scan
192.168.0.116 ol6-112-scan.localdomain ol6-112-scan
192.168.0.117 ol6-112-scan.localdomain ol6-112-scan

From each guest I can ping the private IP of the other guest, but when I ping the public IP of the other guest I see 'host unreachable'. the instructions said to unset the gateway IP on eth2. Not sure what magic is supposed to take place but I cannot get around this. The requirement for Oracle RAC is to have the same interface names (eth1 and eth2 in my case) on each node. The contents of /etc/sysconfig/network-scripts/ifcfg-eth* are as follows for RAC2 (RAC1 are similar but with ...111 for the IP):

----- ifcfg-eth0 -----
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=eth3
ONBOOT=yes
HWADDR=08:00:27:33:0D:1C
PEERDNS=yes
PEERROUTES=yes
LAST_CONNECT=1365652742

----- ifcfg-eth1 -----
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.0.112
PREFIX=24
GATEWAY=192.168.0.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=eth1
ONBOOT=yes
HWADDR=08:00:27:35:EC:54
UUID=9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04
LAST_CONNECT=1366246694

----- ifcfg-eth2 -----
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.1.112
PREFIX=24
GATEWAY=
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=eth2
ONBOOT=yes
HWADDR=08:00:27:81:8F:04
LAST_CONNECT=1365887801

The output of ifconfig seems as expected except eth1/2 show Tx and Rx errors. The NAT eth0 does not work but I don't really care, I just need eth1/2 to work:

------ ifconfig -------
[oracle@ol6-112-rac2 network-scripts]$ ifconfig
eth0 Link encap:Ethernet HWaddr 08:00:27:33:0D:1C
inet6 addr: fe80::a00fe33:d1c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:372 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:41879 (40.8 KiB) TX bytes:4752 (4.6 KiB)
Interrupt:17 Base address:0xd260

eth1 Link encap:Ethernet HWaddr 08:00:27:35:EC:54
inet addr:192.168.0.112 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:19 Base address:0xd020

eth2 Link encap:Ethernet HWaddr 08:00:27:81:8F:04
inet addr:192.168.1.112 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::a00fe81:8f04/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:399 errors:0 dropped:0 overruns:0 frame:0
TX packets:42 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:47458 (46.3 KiB) TX bytes:4978 (4.8 KiB)
Interrupt:16 Base address:0xd240

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:20 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1656 (1.6 KiB) TX bytes:1656 (1.6 KiB)

I hope I am overlooking something silly and easy to correct, but after farting around for a week I cannot seem to improve upon things. Any advice on how to pursue further?

[Craig5479Anderson]

Also, my host (a Lenovo laptop) has an Ethernet wired port and a wireless port. I see the VirtualBox Host-Only Ethernet Adapter so that was installed successfully. I am on VirtualBox 4.2.12 and have the Guest Additions installed (same version). I am following an excellent post at oracle-base.com but my new status on this forum does not allow me to post URLs for the next 24 hours

[Craig5479Anderson]

You will notice the VirtualBox Host-Only Adapter has an unusual IP address. I set it back to the default 192.168.56.1 using the File > Properties in the VMB Manager. Also I see a post from JRitchie (June 16, 2012) with similar issues. From the post it looks like he found a simple fix, but I have already defined my adapters as Host-Only. I don't see anything else different from his post.

In case the same questions come up, the private IP settings are for the intra-cluster communications between the database nodes so no one else needs to access this subnet. Static IP is fine. Static is also fine on public subnet since this not going to be exposed to the rest of my network. This is an exercise in understanding how to set up an Oracle RAC cluster. No one else needs to access it.

Thanks in advance for anyone willing to offer advice!

[noteirak]

Food for thoughts:
1. Why do you have a NAT interface? If you don't need it, remove it.
2. Why don't you use Internal mode if only the VMs need to talk to each other? Do you need the host to talk to the VMs for any reason? If not, use Internal and not Host-Only.
3. What are these other IPs for (scan, etc)?
4. Who has the 192.168.0.1 IP?

Your "public" interface does not have the "RUNNING" status, wherehas all the others do. It also never sent any packets at all. Seems like you forgot to enable the interface in the guest.

[Craig5479Anderson]

Thanks for the food
1. I was overly optimistic and set up NAT interface so I can play with the finished cluster. I have now removed it
2. Thought Host-Only mode was safer. I have now changed it to Internal mode
3. Scan IPs are used by the cluster software but are not defined anywhere in the network setups
4. I thought I needed to set a gateway for the interfaces so they could find each other (ok I was guessing at that after things did not work). I have removed GATEWAY= in the ifcfg-eth* files

After rebooting on this new configuration I get the guest eth0 interfaces to ping each other. However both VM guests skipped over defining eth1 even though I think they are configured correctly (hardware addr are correct per contents of /etc/udev/rules.d/70-persistent-net.rules

Instead, they both have eth2 defined but not with IPv4 addresses. I cannot figure out why it is defining eth2 and not eth1 Here are some files.

----- ol6-112-rac1 -----
[oracle@ol6-112-rac1 Desktop]$ ifconfig
eth0 Link encap:Ethernet HWaddr 08:00:27:F5:5A:96
inet addr:192.168.0.111 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00fef5:5a96/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:21 errors:0 dropped:0 overruns:0 frame:0
TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3871 (3.7 KiB) TX bytes:3858 (3.7 KiB)
Interrupt:16 Base address:0xd240

eth2 Link encap:Ethernet HWaddr 08:00:27:3B:3C:16
inet6 addr: fe80::a00fe3b:3c16/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:840 (840.0 b) TX bytes:468 (468.0 b)
Interrupt:17 Base address:0xd260

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:72 errors:0 dropped:0 overruns:0 frame:0
TX packets:72 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5036 (4.9 KiB) TX bytes:5036 (4.9 KiB)

[oracle@ol6-112-rac1 Desktop]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.0.111
PREFIX=24
GATEWAY=
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=eth0
ONBOOT=yes
HWADDR=08:00:27:F5:5A:96
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
LAST_CONNECT=1366420848

[oracle@ol6-112-rac1 Desktop]$ cat /etc/sysconfig/network-scripts/ifcfg-eth1
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.1.111
PREFIX=24
GATEWAY=
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=eth1
ONBOOT=yes
HWADDR=08:00:27:ef:eb:dc
LAST_CONNECT=1365716178
[oracle@ol6-112-rac1 Desktop]$ cat /etc/sysconfig/network-scripts/ifcfg-eth2


----- ol6-112-rac2 -----
[root@ol6-112-rac2 network-scripts]# ifconfig
eth0 Link encap:Ethernet HWaddr 08:00:27:33:0D:1C
inet addr:192.168.0.112 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00fe33:d1c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24 errors:0 dropped:0 overruns:0 frame:0
TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2112 (2.0 KiB) TX bytes:5273 (5.1 KiB)
Interrupt:17 Base address:0xd260

eth2 Link encap:Ethernet HWaddr 08:00:27:81:8F:04
inet6 addr: fe80::a00fe81:8f04/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:660 (660.0 b) TX bytes:552 (552.0 b)
Interrupt:16 Base address:0xd240

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:72 errors:0 dropped:0 overruns:0 frame:0
TX packets:72 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5036 (4.9 KiB) TX bytes:5036 (4.9 KiB)

[root@ol6-112-rac2 network-scripts]# cat ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.0.112
PREFIX=24
GATEWAY=
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=eth0
ONBOOT=yes
HWADDR=08:00:27:33:0D:1C
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
LAST_CONNECT=1366420807

[root@ol6-112-rac2 network-scripts]# cat ifcfg-eth1
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.1.112
PREFIX=24
GATEWAY=
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=eth1
ONBOOT=yes
HWADDR=08:00:27:35:EC:54
LAST_CONNECT=1366246694

[noteirak]

what is the output of this on both guests?

Code: Select all

ifconfig -a

Note the -a switch, which will also show you the down interfaces...

[Craig5479Anderson]

I see no difference in the output...

[oracle@ol6-112-rac1 Desktop]$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 08:00:27:F5:5A:96
inet addr:192.168.0.111 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00fef5:5a96/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:240 (240.0 b) TX bytes:3870 (3.7 KiB)
Interrupt:16 Base address:0xd240

eth2 Link encap:Ethernet HWaddr 08:00:27:3B:3C:16
inet6 addr: fe80::a00fe3b:3c16/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:480 (480.0 b) TX bytes:636 (636.0 b)
Interrupt:17 Base address:0xd260

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:480 (480.0 b) TX bytes:480 (480.0 b)

[oracle@ol6-112-rac1 Desktop]$

[noteirak]

Your guests are just not configured properly.
The first thing that strikes me is that from your ifcfg-ethX, there are MAC adresses that do not show up in your interfaces list.

On your first guest :
You expect to configure interfaces with the following MACs : 08:00:27:F5:5A:96 & 08:00:27:ef:eb:dc
But in your interface list, there is 2 interfaces with 08:00:27:F5:5A:96 & 08:00:27:3B:3C:16

So far this is not a Virtualbox problem but a guest configuration one. Please ensure that first you configured properly your interfaces.

[Craig5479Anderson]

Outstanding! It works! Now I need to understand some basic principles because the changes I have been making were not persisting as I expected. I would like to understand the relationship between:

1. The MAC address in the adapter setup dialog in the VM Manager (I just noticed the 'refresh' icon makes the MAC change according to a pattern I can't figure out)
2. The MAC address/adapter name combinations that appear in /etc/udev/rules.d/70-persistent-net.rules
3. The MAC address/adapter name that one can set in /etc/sysconfig/network-scripts/ifcfg-eth* files as well as the Network Connections utility in Oracle Linux 6.3

Are the MAC addresses in 70-persistent-net.rules taken from VM Manager? Are they refreshed every time the VM starts up?

I believe I got into trouble because the instructions I was following had me set up one VM then clone it to create the second. That made the MAC ID/Adapter Names identical. So I commented out the first 4 MAC lines in rules file and renamed the last 4 adapter names starting from eth0. I thought that made sense and that I was being careful, but somehow in that process I framised up everything.

Thank you for your patience and for sharing your expertise!

[Craig5479Anderson]

Oh, I left out the reason that identical rule files were a problem. Oracle clustering requires the same adapter names for public and private subnets on all nodes. So eth1 has to be on the private subnet on both my RAC nodes, same for eth0 on public subnet.

[noteirak]

I don't have much experience with any RedHat-based distro, so I can't tell you much, except that the MAC will not change unless you do it yourself.
The scripts are mererly generated when you install and modifed later on either by you or some networking daemon tool / GUI tool.

The basic idea is : MAC will not change without a manual change from your side, and the config file must match it or there is no way to map a config file to an interface and a config.

[Craig5479Anderson]

OK, I probably made wrong changes out of ignorance of how MAC Addresses are defined.

I was moving on to the next step (installing the clusterware) when I see that I need to download new software to the VM via yum. That requires access to the Internet so I took your earlier suggestion and defined the first appropriate adapter as Bridged (I had both adapters as Internal) and restarted the VM. Now I cannot ping anything on the Internet nor can I ping the other VM on the public subnet (eth0).

My first thought was the role of the VirtualBox Host-Only Adapter on my Win 7 host. I no longer have a Host-only adapter defined. Is it still involved in the networking? Do I need to set something else up or is this just supposed to know how to route things?

[Craig5479Anderson]

OK, I learn slowly. I put things back so eth0/1 work between the VMs. Then I set up another adapter (eth3) as Bridged and made sure the MAC address issue didn't get me again, and set up eth3 as DHCP. Now I see all 3 adapters, the VMs can ping each other, but the bridged adapter does not show an IPv4 address and cannot reach any other subnet. I tried adding GATEWAY= to the eth3 setup but not sure what to put there.

[root@ol6-112-rac1 network-scripts]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 08:00:27:F5:5A:96
inet addr:192.168.0.111 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00fef5:5a96/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:143 errors:0 dropped:0 overruns:0 frame:0
TX packets:86 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:23201 (22.6 KiB) TX bytes:13914 (13.5 KiB)
Interrupt:16 Base address:0xd240

eth1 Link encap:Ethernet HWaddr 08:00:27:EF:EB:DC
inet addr:192.168.1.111 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::a00feef:ebdc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:121 errors:0 dropped:0 overruns:0 frame:0
TX packets:94 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:18530 (18.0 KiB) TX bytes:14443 (14.1 KiB)
Interrupt:17 Base address:0xd260

eth2 Link encap:Ethernet HWaddr 08:00:27:3B:3C:16
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:19 Base address:0xd020

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:480 (480.0 b) TX bytes:480 (480.0 b)

[noteirak]

Well first of all you don't have a eth3, only a eth2, which I assume it is that one, and it is not enabled. Do so first.

If I may, none of your issues so far were related to Virtualbox, only to the OS configuration.
I would strongly suggest you read more on networking and usage of Linux & Redhat-based in particular.

[Craig5479Anderson]

I guess I see it differently. Normally the MAC Address is provided by a physical interface, here it is set by VirtualBox using logic I am not familiar with. Normally I set the OS to use DHCP and it automatically sets things up, here it does not. Normally if I need to set up static IPs I enter my router IP as the GATEWAY but here that does not work. The way I see it everything so far has been about how VituralBox provides networking in a totally virtual environment. Read up on networking is not likely to help me understand what VirtualBox is doing or how to set it up. If I am way off please point out the flaw in my thinking.