Wednesday, May 9, 2012

Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

Oracle Security Alert CVE-2012-1675 mentions two metalink notes.
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC [ID 1340831.1] for RAC systems and Using Class of Secure Transport (COST) to Restrict Instance Registration [ID 1453883.1] for non-RAC systems. Non-RAC systme works fine in Oracle Enterprise and Standard Edition systems. But the solution given in 1340831.1 for "Oracle RAC versions prior to 11.2 (10.2.0.3 - 11.1)" cannot be implemented in Oracle Standard Edition RAC. After the wallet and the patch 12880299 is applied the metalink notes says to add a line to listener.ora for TCPS protocol and restart the listener. (This environment has seperate homes for ASM and DB and listener runs out of ASM home). Oracle database version is 11.1.0.7.11
LISTENER_RAC1 =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCPS)(HOST = rac1-vip)(PORT = 1523)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.0.85)(PORT = 1521)(IP = FIRST))
    )
  )

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/oracle/product/11.1.0/asm_1/network/admin/cost)
    )
  )

#SECURE_REGISTER_LISTENER_RAC1 = (TCP,TCPS)
It then ask to stop and start the listener on that node. During the start of the listener following error is shown on the command line.
$ASM_HOME/bin/srvctl start listener -n `hostname -s`
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:LSNRCTL for Linux: Version 11.1.0.7.0 - Production on 09-MAY-2012 15:01:38
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Copyright (c) 1991, 2008, Oracle.  All rights reserved.
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Starting /opt/app/oracle/product/11.1.0/asm_1/bin/tnslsnr: please wait...
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNSLSNR for Linux: Version 11.1.0.7.0 - Production
rac1:ora.rac1.LISTENER_RAC1.lsnr:System parameter file is /opt/app/oracle/product/11.1.0/asm_1/network/admin/listener.ora
rac1:ora.rac1.LISTENER_RAC1.lsnr:Log messages written to /opt/app/oracle/diag/tnslsnr/rac1/listener_rac1/alert/log.xml
rac1:ora.rac1.LISTENER_RAC1.lsnr:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.89)(PORT=1521)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=rac1-vip)(PORT=1523)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12557: TNS:protocol adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00527: Protocol Adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Listener failed to start. See the error message(s) above...
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:LSNRCTL for Linux: Version 11.1.0.7.0 - Production on 09-MAY-2012 15:01:38
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Copyright (c) 1991, 2008, Oracle.  All rights reserved.
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)(PORT=1521)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12541: TNS:no listener
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00511: No listener
rac1:ora.rac1.LISTENER_RAC1.lsnr:   Linux Error: 111: Connection refused
rac1:ora.rac1.LISTENER_RAC1.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=rac1-vip)(PORT=1523)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12557: TNS:protocol adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00527: Protocol Adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.85)(PORT=1521)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12541: TNS:no listener
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00511: No listener
rac1:ora.rac1.LISTENER_RAC1.lsnr:   Linux Error: 111: Connection refused
CRS-0215: Could not start resource 'ora.rac1.LISTENER_RAC1.lsnr'.
On listener.log
System parameter file is /opt/app/oracle/product/11.1.0/asm_1/network/admin/listener.ora
Log messages written to /opt/app/oracle/diag/tnslsnr/rac1/listener_rac1/alert/log.xml
Trace information written to /opt/app/oracle/diag/tnslsnr/rac1/listener_rac1/trace/ora_15792_47190667617024.trc
Trace level is currently 0

Started with pid=15792
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.89)(PORT=1521)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=rac1-vip)(PORT=1523)(IP=FIRST)))
TNS-12557: TNS:protocol adapter not loadable
 TNS-12560: TNS:protocol adapter error
  TNS-00527: Protocol Adapter not loadable

No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.89)(PORT=1521)))
On crsd.log
2012-05-09 15:01:37.891: [  CRSRES][1510025536] startRunnable: setting CLI values
2012-05-09 15:01:37.895: [  CRSRES][1510025536] Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
2012-05-09 15:01:38.920: [  CRSAPP][1510025536] StartResource error for ora.rac1.LISTENER_RAC1.lsnr error code = 1
2012-05-09 15:01:39.240: [  CRSRES][1510025536] Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` failed.
2012-05-09 15:01:39.255: [  CRSRES][1510025536] rac2 : CRS-1019: Resource ora.rac1.LISTENER_RAC1.lsnr (application) cannot run on rac2
It could be that in SE the Advance Security Option needed to support TCPS is not there.

In the security alert Oracle mentions "Please note that Oracle has added Oracle Advanced Security SSL/TLS to the Oracle Database Standard Edition license when used with the Real Application Clusters and Oracle has added Oracle Advanced Security SSL/TLS to the Enterprise Edition Real Application Clusters (Oracle RAC) and RAC One Node options so that the directions provided in the Support Notes referenced above can be applied by all Oracle customers without additional cost."

The steps works in a 11.1.0.7.11 Enterprise Edition (EE) RAC without any issue. This could be explained using How To Check if Oracle Advanced Security Option is Installed ? [ID 549989.1] in which it is mentioned "Oracle product development has decided that starting with Oracle 11g Oracle Advanced Security Option will be installed by default when installing Enterprise Edition . There is no way to uninstall or to install it."

Update 31 May 2012
For the steps mentioned in 1340831.1 to work on Oracle SE it requires Oracle binaries to be relinked with the tcps. In this example the listener is running out of ASM home therefore oracle binaries in ASM Home will be relinked with tcps. It is not necessary to relink the Oracle Home if listener is not running out of it
1. cd $ASM_HOME/lib (or $ORACLE_HOME/lib if a separate home for ASM is not used).

2. Make a backup of the current tcps binary
$  cp libntcps11.a libntcps11.a.orig
3. Find libntcps11_ee.a.dbl file in the same lib directory and make a copy of it with the file name libntcps11.a
$ cp libntcps11_ee.a.dbl libntcps11.a
4. To make sure the relink is happening in ASM_HOME set the ORACLE_HOME to ASM_HOME and set relink binary in PATH variable
export ORACLE_HOME=$ASM_HOME
which relink
/opt/app/oracle/product/11.1.0/db_1/bin/relink <-- still using the ORACLE_HOME relink

export PATH=$ORACLE_HOME/bin:$PATH
/opt/app/oracle/product/11.1.0/asm_1/bin/relink
5. Stop all processes running out of ASM_HOME. Since database instance is dependent on asm instance this would mean database instance should be stopped as well. Once all running processes are stopped (except clusterware processes) start the relink with
relink all
and the relink process end with following lines (this is on 11.1.0.7 , on other versions it may be different)
..
..
/app/oracle/product/11.1.0/asm_1/lib
mv -f /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc0
mv ctxkbtc /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc
chmod 751 /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc
Once the relink is complete continue with the steps in the metalink document and the listener will be able to start with TCPS.
If the file libntcps11_ee.a.dbl is missing in $ORACLE_HOME/lib then it could be obtained by extracting the
database/stage/Components/oracle.network.rsf/11.1.0.6.0/1/DataFiles/filegroup7.jar
file in the 11.1.0.6 base installation. Even though the system is upgraded to 11.1.0.7 there's no copy of it in 11.1.0.7 patchset files which mean it is not part of the upgraded processes thus only place to get it is on 11.1.0.6 base instllation.