Upgrading OS in 11gR2 RAC Environment

Oracle clusterware 11gR2 requires relinking whenever an OS patch or OS is upgraded. According to GI Installation guide "You must relink the Oracle Clusterware and Oracle ASM binaries every time you apply an operating system patch or after an operating system upgrade. Before relinking executables, you must shut down all executables that run in the Oracle home directory that you are relinking. In addition, shut down applications linked with Oracle shared libraries."
Relink is done with the following steps, again from the GI installation guide

As root:
# cd Grid_home/crs/install
# perl rootcrs.pl -unlock

As the Oracle Grid Infrastructure for a Cluster owner:
$ export ORACLE_HOME=Grid_home
$ Grid_home/bin/relink

As root again:
# cd Grid_home/rdbms/install/
# ./rootadd_rdbms.sh
# cd Grid_home/crs/install
# perl rootcrs.pl -patch

As per (743649.1) Oralce support rolling upgrade of OS when both OS are certified on the database that's running. This is supported only for the duration of the upgrade. But according to (1391807.1) using multiple versions of ASMLib across the cluster may not be compatible. In this case ASM disks were migrated from ASMLib disk to block devices before the upgrade therefore an ASMLib upgrade was not required. But if ASMLib was in use then ASMLib should be upgraded as part of the OS upgrade.
1. Current kernel version

uname -r
2.6.18-194.el5

2. This upgrade is done across the cluster (not a rolling upgrade) therefore stop all the cluster components

crsctl stop cluster -all

and stop disable start of crs

crsctl stop crs
crsctl disable crs

Also comment spawning of high availability processes

#h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

# init q

3. Upgrade OS. New kernel version

uname -r
2.6.18-308.el5

4. Remove the comment of ohas from inittab and enable crs

h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

crsctl enable crs

5. Relink ORACLE_HOME binaries

 which relink
/opt/app/oracle/product/11.2.0/db_2/bin/relink

cd $ORACLE_HOME/bin
$ relink all
writing relink log to: /opt/app/oracle/product/11.2.0/db_2/install/relink.log

Relink ends with following in the log file

test ! -f /opt/app/oracle/product/11.2.0/db_2/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/db_2/bin/oracle /opt/app/oracle/product/11.2.0/db_2/bin/oracleO
mv /opt/app/oracle/product/11.2.0/db_2/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/db_2/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/db_2/bin/oracle

5. Relink the GI HOME binaries

cd $GI_HOME/crs/install

# perl rootcrs.pl -unlock
Using configuration parameter file: ./crsconfig_params
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully unlock /opt/app/11.2.0/grid3

$ export ORACLE_HOME=$GI_HOME
$ export PATH=$ORACLE_HOME/bin:$PATH
$ which relink
/opt/app/11.2.0/grid3/bin/relink

$GI_HOME/bin/relink

Relink ends with the following in the log file

test ! -f /opt/app/11.2.0/grid3/bin/oracle ||\
           mv -f /opt/app/11.2.0/grid3/bin/oracle /opt/app/11.2.0/grid3/bin/oracleO
mv /opt/app/11.2.0/grid3/rdbms/lib/oracle /opt/app/11.2.0/grid3/bin/oracle
chmod 6751 /opt/app/11.2.0/grid3/bin/oracle

Complete the relink process

# cd $GI_HOME/rdbms/install

# ./rootadd_rdbms.sh

# cd $GI_HOME/crs/install
# perl rootcrs.pl -patch

Using configuration parameter file: ./crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.

Execute

/sbin/init q

to start the cluster stack.

Related Post
Upgrading ASMLib and OS in 11gR1 RAC Environment
Upgrading RHEL 6 OS in a 11gR2 RAC Environment

Useful Metalink notes
Oracle 11gR2 Relink New Feature [ID 883299.1]