Cannot make internal network work on oel6 guests...

[Max0911]

Hi

I am trying to create an Oracle 12c database RAC using 2 OEL6 guests on a Win laptop.
Both my guests have 3 network interfaces, a Host Only Network, a NAT and an Internal Network.
I am using the Internal Network as dedicated RAC interconnect.
The problem is my guests are not able to ping the other guest's private IP which uses the internal network.
Following are my network settings:
Node 1
----------
eth0 Link encap:Ethernet HWaddr 08:00:27:56:E2:BE
inet addr:10.154.137.101 Bcast:10.154.137.255 Mask:255.255.255.0
inet6 addr: fe80::a00fe56:e2be/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:136 errors:0 dropped:0 overruns:0 frame:0
TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:14047 (13.7 KiB) TX bytes:7173 (7.0 KiB)

eth1 Link encap:Ethernet HWaddr 08:00:27:7C:9B:DE
inet addr:10.154.138.101 Bcast:10.154.138.255 Mask:255.255.255.0
inet6 addr: fe80::a00fe7c:9bde/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:210 (210.0 b) TX bytes:914 (914.0 b)

eth2 Link encap:Ethernet HWaddr 08:00:27:F9:83:D7
inet6 addr: fe80::a00fef9:83d7/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:468 (468.0 b)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:10 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:642 (642.0 b) TX bytes:642 (642.0 b)

Node 2
---------
eth0 Link encap:Ethernet HWaddr 08:00:27:29:2C:A3
inet addr:10.154.137.102 Bcast:10.154.137.255 Mask:255.255.255.0
inet6 addr: fe80::a00fe29:2ca3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:137 errors:0 dropped:0 overruns:0 frame:0
TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:14203 (13.8 KiB) TX bytes:7173 (7.0 KiB)

eth1 Link encap:Ethernet HWaddr 08:00:27:CE:65:42
inet addr:10.154.138.102 Bcast:10.154.138.255 Mask:255.255.255.0
inet6 addr: fe80::a00fece:6542/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:914 (914.0 b)

eth2 Link encap:Ethernet HWaddr 08:00:27:CE:20:9E
inet6 addr: fe80::a00fece:209e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:480 (480.0 b) TX bytes:468 (468.0 b)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:54 errors:0 dropped:0 overruns:0 frame:0
TX packets:54 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6348 (6.1 KiB) TX bytes:6348 (6.1 KiB)

Please note that eth2 is down on both the nodes. I only make it up when I want the guests to be able to connect to the internet.

The problem is:

[root@ol6r01 ~]# ping ol6r02-priv
PING ol6r02-priv (10.154.138.102) 56(84) bytes of data.
From ol6r01-priv (10.154.138.101) icmp_seq=2 Destination Host Unreachable
From ol6r01-priv (10.154.138.101) icmp_seq=3 Destination Host Unreachable
From ol6r01-priv (10.154.138.101) icmp_seq=4 Destination Host Unreachable

[root@ol6r02 ~]# ping ol6r01-priv
PING ol6r01-priv (10.154.138.101) 56(84) bytes of data.
From ol6r02-priv (10.154.138.102) icmp_seq=2 Destination Host Unreachable
From ol6r02-priv (10.154.138.102) icmp_seq=3 Destination Host Unreachable
From ol6r02-priv (10.154.138.102) icmp_seq=4 Destination Host Unreachable

This is working:

[root@ol6r01 ~]# ping ol6r02
PING ol6r02 (10.154.137.102) 56(84) bytes of data.
64 bytes from ol6r02 (10.154.137.102): icmp_seq=1 ttl=64 time=1.14 ms
64 bytes from ol6r02 (10.154.137.102): icmp_seq=2 ttl=64 time=0.370 ms
64 bytes from ol6r02 (10.154.137.102): icmp_seq=3 ttl=64 time=0.629 ms

[root@ol6r02 ~]# ping ol6r01
PING ol6r01 (10.154.137.101) 56(84) bytes of data.
64 bytes from ol6r01 (10.154.137.101): icmp_seq=1 ttl=64 time=0.568 ms
64 bytes from ol6r01 (10.154.137.101): icmp_seq=2 ttl=64 time=0.637 ms
64 bytes from ol6r01 (10.154.137.101): icmp_seq=3 ttl=64 time=0.604 ms

Screenshot for network adapter settings for internal network on node 1

Node 1 internal network.PNG (39.91 KiB) Viewed 2023 times

Screenshot for network adapter settings for internal network on node 2

Node 2 internal network.PNG (41.83 KiB) Viewed 2023 times

Appreciate any inputs about this issue.

[Max0911]

Hi Guys

I am really stuck at this step and cannot proceed to create my 12c RAC database.
I would really appreciate if you could provide some pointers as to what is missing in my setup.

Thanks

[Perryg]

First thing I would do is simplify the network and get it working then add extra adapters one at a time to achieve what ever it is you are wanting.

If the guest just need to talk to each other and none other then use the internal and assign static IP and subnet only.
If the host and the guest need to talk to each other than use host-only and start with using dhcp to see if it works and if it does then use static on the guests.
If you need the internet then use NAT in addition to the one you select above. You do not need three adapters at all.

[Max0911]

Hi Perryg

Thanks for your response.
I have 3 adapters because I need a host-only network which servers the VIP and Public IPs for the RAC.
I have NAT so that I can connect to the internet from time to time and download any packages if required. Please note that NAT is down all the time unless required.
I have an internal network which I need to use for Private Interconnect for the RAC.
The host-only network and NAT works perfectly however only my Internal Network is not working properly. I could ping the private IP on the same machine but not the private of of the other machine.

Thanks

[Perryg]

Well you pretty much just said the same thing as before. What I said is to simplify the network and do it one step at a time to make sure that all of them are really working. Why you think you need all of these active is up to you but it has been my experience that all you really should need is two. As for disabling the NAT that you need for the Internet it should not make a difference one way or the other.

If I were diagnosing this I would do as I said one at a time and use wireshark to see where it is failing. Other that that it is just throwing time at the project using trial and error.

[Max0911]

Problem solved !
My adapter settings got mixed up when I copied my guest to another vm.
On my main VM the adapter sequencing was:
Adapter 1 - eth0
Adapter 2 - eth1
Adapter 3 - eth2

However after copying the machines the sequencing changed to:
Adapter 1 - eth0
Adapter 2 - eth2
Adapter 3 - eth1

I just configured the /etc/sysconfig/network-scripts/ifcfg-eth* on the second machine same as the first machine without noticing the adapter sequencing mixed up.

Correct the adpater settings on the second machine and everything works.

Thanks Perryg, your recommendation about disabling the extra adapters helped me to pin point the issue.

Cheers.