Restart a node in a 11.2.0.3 (GI and DB both) cluster and the node vip is not reachable from the external network, restart the other nodes too and now port 1521 on all the cluster vip addresses and scan addresses is not reachable.
This is a rather serious issue even in a dev cluster and all the cluster services are actually up and running including SCAN vip addresses, SCAN listeners, local listeners etc.
However the port 1521 on the node vip addresses is reachable from the other nodes of the cluster or from the same subnet to be more specific.
This took a while to troubleshoot involving security and sysadmin teams and finally the issue was due to a weird mac address entry at the firewall arp cache and clearing the cache resolved it and the root cause is in the Oracle clusterware due to a strange Oracle bug 13440962 which says
"After upgrading to 11.2.0.3, after vip failover, the ip address is
not pingable from a different subnet on Linux."
And the fix is to run the below:
After vip failover, run command
/sbin/arping -U -c 3 -I public NIC for vip vip ip address
to update the ARP table of router.
No comments:
Post a Comment