Thursday, May 25, 2017

Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Clusterware

This post is not a step by step guide to installing 12cR2 RAC but list the main differences in the installation process compared to previous versions and things to look out for. One of the main difference in installing 12cR2 clusterware is that, it is no longer done through runInstaller. The downloaded Oracle Grid Infrastructure image files are unzipped into the folder path where GI is to be installed. This only need to be done on the local node only. During installation, the software is copied and installed on all other nodes in the cluster.
If it is planned to ASM filter driver (ASMFD) for managing disks used by ASM then uninstall ASMLib if it is already installed.
Unlike 12cR1, it is possible to specify a separate disk group for GIMR at the installation time. Oracle documentation says "You cannot migrate the GIMR from one disk group to another later". 12cR1 also mentioned the same but there were ways to move GIMR to different disk group after the installation. It has been not verified whether the same techniques could be used for 12.2 as well.
1. Shown below is the OS and kernel versions used for this RAC installation. These are the minimum versions supported for 12.2 clusterware installation. If the OS/Kernel versions are lower than this version then OS need to be upgraded before installation could proceed.
uname -r
2.6.32-358.el6.x86_64
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.4 (Santiago)
2. Create the user groups and users similar to previous 12.1 installation. As said in the previous post 12.2 introduce a new user group for RAC management. Following from install guide "You must designate a group as the OSRACDBA group during database installation. Members of this group are granted the SYSRAC privileges to perform day–to–day administration of Oracle databases on an Oracle RAC cluster".
groupadd dba
groupadd oinstall
groupadd oper
groupadd asmoper
groupadd asmdba
groupadd asmadmin
groupadd backupdba
groupadd dgdba
groupadd kmdba
groupadd racdba

useradd -g oinstall -G dba,oper,asmdba,backupdba,dgdba,kmdba,racdba oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba grid 
3. The other pre-requisite steps are similar to 12.1 RAC installation. Follow oracle installation guide for detail instructions. As said earlier the grid infrastructure software is unzip directly into the install location.
mkdir -p /opt/app/12.2.0/grid
cp ~/linuxx64_12201_grid_home.zip /opt/app/12.2.0/grid
cd /opt/app/12.2.0/grid
unzip linuxx64_12201_grid_home.zip 
4. 12.2 gives the option of using ASM filter driver (ASMFD) for managing disks used by ASM. However installation could be done without the use of AFD as well. Using udev rules for the ASM disks. For this setup UDEV rules have been setup as below. The disks used for OCR were of 3GB (OCR disk group to be of normal redundancy) and GIMR disk group is 40GB (with external redundancy). Minimum storage requirement for standaloen cluster could be found here.
KERNEL=="sd*1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB34650f33-5ecdc029",OWNER="grid",GROUP="asmadmin",MODE="0660",NAME="oracleasm/data1"
KERNEL=="sd*1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBdb057b08-1733da4b",OWNER="grid",GROUP="asmadmin",MODE="0660",NAME="oracleasm/fra1"

KERNEL=="sd*1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a72679c-ab23e15f",OWNER="grid",GROUP="asmadmin",MODE="0660",NAME="oracleasm/ocr1"
KERNEL=="sd*1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB8f43e75c-3e0bec6d",OWNER="grid",GROUP="asmadmin",MODE="0660",NAME="oracleasm/ocr2"
KERNEL=="sd*1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a5c8787-e082e7a7",OWNER="grid",GROUP="asmadmin",MODE="0660",NAME="oracleasm/ocr3"

KERNEL=="sd*1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBf8475ab7-d401ec8b",OWNER="grid",GROUP="asmadmin",MODE="0660",NAME="oracleasm/gimr"


5. The main body of the post shows the installation without using AFD. The end of the post will have additional steps needed to be done before clusteware is install if AFD is desired. 12.2 install documentation states that gcc and gcc+ are not required on OEL and RHEL for GI installations. However the runcluvfy does look for these two packages (see output below). In this case the pre-check was unsuccessful due to lower SWAP size, which is ignorable (unless for production).
 ./runcluvfy.sh stage -pre crsinst -n rhel6m1,rhel6m2 -r 12.2 -osdba asmdba -osoper asmoper -asm -asmgrp asmadmin -presence local -asmdev /dev/oracleasm/data1,/dev/oracleasm/fra1,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3,/dev/oracleasm/gimr  -crshome /opt/app/12.2.0/grid -fixup  -verbose

Verifying Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       7.6865GB (8059836.0KB)    8GB (8388608.0KB)         passed
  rhel6m1       7.6865GB (8059836.0KB)    8GB (8388608.0KB)         passed
Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       7.5462GB (7912808.0KB)    50MB (51200.0KB)          passed
  rhel6m1       7.2342GB (7585564.0KB)    50MB (51200.0KB)          passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       4GB (4194296.0KB)         7.6865GB (8059836.0KB)    failed
  rhel6m1       4GB (4194296.0KB)         7.6865GB (8059836.0KB)    failed
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: rhel6m2:/usr,rhel6m2:/var,rhel6m2:/etc,rhel6m2:/opt/app/12.2.0/grid,rhel6m2:/sbin,rhel6m2:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel6m2       /             31.793GB      25MB          passed
  /var              rhel6m2       /             31.793GB      5MB           passed
  /etc              rhel6m2       /             31.793GB      25MB          passed
  /opt/app/12.2.0/grid  rhel6m2       /             31.793GB      6.9GB         passed
  /sbin             rhel6m2       /             31.793GB      10MB          passed
  /tmp              rhel6m2       /             31.793GB      1GB           passed
Verifying Free Space: rhel6m2:/usr,rhel6m2:/var,rhel6m2:/etc,rhel6m2:/opt/app/12.2.0/grid,rhel6m2:/sbin,rhel6m2:/tmp ...PASSED
Verifying Free Space: rhel6m1:/usr,rhel6m1:/var,rhel6m1:/etc,rhel6m1:/opt/app/12.2.0/grid,rhel6m1:/sbin,rhel6m1:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel6m1       /             21.9208GB     25MB          passed
  /var              rhel6m1       /             21.9208GB     5MB           passed
  /etc              rhel6m1       /             21.9208GB     25MB          passed
  /opt/app/12.2.0/grid  rhel6m1       /             21.9208GB     6.9GB         passed
  /sbin             rhel6m1       /             21.9208GB     10MB          passed
  /tmp              rhel6m1       /             21.9208GB     1GB           passed
Verifying Free Space: rhel6m1:/usr,rhel6m1:/var,rhel6m1:/etc,rhel6m1:/opt/app/12.2.0/grid,rhel6m1:/sbin,rhel6m1:/tmp ...PASSED
Verifying User Existence: grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists(502)
  rhel6m1       passed                    exists(502)

  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmdba ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: oinstall(Primary) ...
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           yes           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           yes           passed
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: asmoper ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3                         3,5                       passed
  rhel6m1       3                         3,5                       passed
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          65536         65536         passed
  rhel6m1           hard          65536         65536         passed
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          1024          1024          passed
  rhel6m1           soft          1024          1024          passed
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          16384         16384         passed
  rhel6m1           hard          16384         16384         passed
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          2047          2047          passed
  rhel6m1           soft          2047          2047          passed
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          10240         10240         passed
  rhel6m1           soft          10240         10240         passed
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       x86_64                    x86_64                    passed
  rhel6m1       x86_64                    x86_64                    passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.6.32-358.el6.x86_64     2.6.32                    passed
  rhel6m1       2.6.32-358.el6.x86_64     2.6.32                    passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           3010          3010          250           passed
  rhel6m2           3010          3010          250           passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           385280        385280        32000         passed
  rhel6m2           385280        385280        32000         passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           3010          3010          100           passed
  rhel6m2           3010          3010          100           passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           128           128           128           passed
  rhel6m2           128           128           128           passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           68719476736   68719476736   4126636032    passed
  rhel6m2           68719476736   68719476736   4126636032    passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4096          4096          4096          passed
  rhel6m2           4096          4096          4096          passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4294967296    4294967296    805983        passed
  rhel6m2           4294967296    4294967296    805983        passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           6815744       6815744       6815744       passed
  rhel6m2           6815744       6815744       6815744       passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel6m2           between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4194304       4194304       262144        passed
  rhel6m2           4194304       4194304       262144        passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4194304       4194304       4194304       passed
  rhel6m2           4194304       4194304       4194304       passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           1048576       1048576       262144        passed
  rhel6m2           1048576       1048576       262144        passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           2097152       2097152       1048576       passed
  rhel6m2           2097152       2097152       1048576       passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           3145728       3145728       1048576       passed
  rhel6m2           3145728       3145728       1048576       passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           1             undefined     1             passed
  rhel6m2           1             undefined     1             passed
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: binutils-2.20.51.0.2 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
  rhel6m1       binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
Verifying Package: binutils-2.20.51.0.2 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libcap1-1.10-1     compat-libcap1-1.10       passed
  rhel6m1       compat-libcap1-1.10-1     compat-libcap1-1.10       passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel6m1       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libgcc-4.4.7 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.7      passed
  rhel6m1       libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.7      passed
Verifying Package: libgcc-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-4.4.7 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.7   passed
  rhel6m1       libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.7   passed
Verifying Package: libstdc++-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.4.7 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.7  passed
  rhel6m1       libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.7  passed
Verifying Package: libstdc++-devel-4.4.7 (x86_64) ...PASSED
Verifying Package: sysstat-9.0.4 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
  rhel6m1       sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
Verifying Package: sysstat-9.0.4 ...PASSED
Verifying Package: gcc-4.4.7 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-4.4.7-3.el6           gcc-4.4.7                 passed
  rhel6m1       gcc-4.4.7-3.el6           gcc-4.4.7                 passed
Verifying Package: gcc-4.4.7 ...PASSED
Verifying Package: gcc-c++-4.4.7 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-c++-4.4.7-3.el6       gcc-c++-4.4.7             passed
  rhel6m1       gcc-c++-4.4.7-3.el6       gcc-c++-4.4.7             passed
Verifying Package: gcc-c++-4.4.7 ...PASSED
Verifying Package: ksh ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       ksh                       ksh                       passed
  rhel6m1       ksh                       ksh                       passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.81 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       make-3.81-20.el6          make-3.81                 passed
  rhel6m1       make-3.81-20.el6          make-3.81                 passed
Verifying Package: make-3.81 ...PASSED
Verifying Package: glibc-2.12 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
  rhel6m1       glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
Verifying Package: glibc-2.12 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.12 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
  rhel6m1       glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
Verifying Package: glibc-devel-2.12 (x86_64) ...PASSED
Verifying Package: libaio-0.3.107 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  rhel6m1       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Verifying Package: libaio-0.3.107 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.107 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
  rhel6m1       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
Verifying Package: libaio-devel-0.3.107 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
  rhel6m1       nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-5.43-1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       smartmontools-5.43-1.el6  smartmontools-5.43-1      passed
  rhel6m1       smartmontools-5.43-1.el6  smartmontools-5.43-1      passed
Verifying Package: smartmontools-5.43-1 ...PASSED
Verifying Package: net-tools-1.60-110 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       net-tools-1.60-110.el6_2  net-tools-1.60-110        passed
  rhel6m1       net-tools-1.60-110.el6_2  net-tools-1.60-110        passed
Verifying Package: net-tools-1.60-110 ...PASSED
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           6200          TCP           yes           successful
  rhel6m1           6200          TCP           yes           successful
  rhel6m2           6100          TCP           yes           successful
  rhel6m1           6100          TCP           yes           successful
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...PASSED
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           42424         TCP           yes           successful
  rhel6m1           42424         TCP           yes           successful
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
  rhel6m1       cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m1                               passed
  rhel6m2                               passed
  Verifying Hosts File ...PASSED

Interface information for node "rhel6m2"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:69:2C:B6 1500

Interface information for node "rhel6m1"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:A3:C4:6F 1500

Check: MTU consistency of the subnet "192.168.1.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           eth1          192.168.1.88  192.168.1.0   1500
  rhel6m1           eth1          192.168.1.87  192.168.1.0   1500

Check: MTU consistency of the subnet "192.168.0.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           eth0          192.168.0.94  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.93  192.168.0.0   1500
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[eth1:192.168.1.87]      rhel6m2[eth1:192.168.1.88]      yes

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.94]      yes
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast check ...
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast check ...PASSED
Verifying Device Checks for ASM ...Disks "/dev/oracleasm/gimr,/dev/oracleasm/fra1,/dev/oracleasm/data1,/dev/oracleasm/ocr3,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2" are managed by ASM.
Verifying Device Checks for ASM ...PASSED
Verifying I/O scheduler ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying I/O scheduler ...PASSED
Verifying Network Time Protocol (NTP) ...
  Verifying '/etc/ntp.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel6m2                               no
  rhel6m1                               no

  Verifying '/etc/ntp.conf' ...PASSED
  Verifying '/var/run/ntpd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel6m2                               no
  rhel6m1                               no

  Verifying '/var/run/ntpd.pid' ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       0022                      0022                      passed
  rhel6m1       0022                      0022                      passed
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    does not exist
  rhel6m1       passed                    does not exist
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying resolv.conf Integrity ...
  Verifying (Linux) resolv.conf Integrity ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m1                               passed
  rhel6m2                               passed

  checking response for name "rhel6m2" from each of the name servers specified
  in "/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       192.168.0.66              IPv4                      passed

  checking response for name "rhel6m1" from each of the name servers specified
  in "/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m1       192.168.0.66              IPv4                      passed
  Verifying (Linux) resolv.conf Integrity ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Domain Sockets ...PASSED
Verifying /boot mount ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying Grid Infrastructure home path: /opt/app/12.2.0/grid ...
  Verifying '/opt/app/12.2.0/grid' ...PASSED
Verifying Grid Infrastructure home path: /opt/app/12.2.0/grid ...PASSED
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying File system mount options for path /var ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Swap Size ...FAILED
rhel6m2: PRVF-7573 : Sufficient swap size is not available on node "rhel6m2"
         [Required = 7.6865GB (8059836.0KB) ; Found = 4GB (4194296.0KB)]

rhel6m1: PRVF-7573 : Sufficient swap size is not available on node "rhel6m1"
         [Required = 7.6865GB (8059836.0KB) ; Found = 4GB (4194296.0KB)]


CVU operation performed:      stage -pre crsinst
Date:                         May 11, 2017 2:54:23 PM
CVU home:                     /opt/app/12.2.0/grid/
User:                         grid
6. From the GI_HOME directory run gridSetup.sh to begin the installation. Select new cluster configuration option.
12.2 has introduced many different types of clusters. This installation creates a standalone cluster.
Set cluster name and scan name. Importantly no GNS is setup or used for this installation.
Add node and node VIP names that will be used in the cluster configuration. Importantly all node roles are set as HUB nodes. If leaf nodes to be configured then this would require setting up and using GNS in previous step. Installation can proceed without a GNS if only HUB nodes are configured.
Select the network interfaces for public and ASM/Private interconnect.
The OCR and vote disks are stored in ASM but ASM could be configured either block devices or using NFS. This setup uses block devices.
As said earlier 12.2 gives the option of storing GIMR in a separate disk group. If a separate disk group is not selected then this would be stored in the same disk group as OCR and vote disks. The backups of the OCR will be stored in the disk group used for GIMR.
The first disk group created is the one for OCR and vote disk. As said earlier this expected to be of normal redundancy. Change the disk discovery path so the candidate disks are displayed. At times the disks may be visible even without changing the discovery path. Even then it's better to change the discovery path to reflect the correct path string. Otherwise during root.sh run the commands will use the default path of /dev/sd* and execution will fail.
2017/05/11 16:26:56 CLSRSC-184: Configuration of ASM failed
2017/05/11 16:27:01 CLSRSC-258: Failed to configure and start ASM
Died at /opt/app/12.2.0/grid/crs/install/crsinstall.pm line 2091.
The command '/opt/app/12.2.0/grid/perl/bin/perl -I/opt/app/12.2.0/grid/perl/lib -I/opt/app/12.2.0/grid/crs/install /opt/app/12.2.0/grid/crs/install/rootcrs.pl ' execution failed
Alert log will show that ASM disk group creation was attempted with default disk path string.
2017-05-11 16:26:24:   Invoking "/opt/app/12.2.0/grid/bin/asmca -silent -diskGroupName clusterdg 
-diskList '/dev/oracleasm/ocr3,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2' -redundancy NORMAL -diskString '/dev/sd*' 
-configureLocalASM -passwordFileLocation +clusterdg/orapwASM -au_size 4 " as user "grid"
If it comes to this it is possible to change the ASM_DISCOVERY_STRING in GI_HOME/crs/install/crsconfig_params to reflect the correct disk path string and run root.sh again. All of this could be avoided by explicitly setting the disk path string.
Select the disk used for OCR disk group (by default redundancy is selected as normal). Since AFD is not used in this setup, uncheck the "Configure Oracle ASM Filter Driver" option.
Select the disk used for GIMR disk group.
Select ORACLE_BASE location if not selected by default. The software location is where the software was extracted to and cannot be changed during the installation.
Running of root scripts could be automated by specify the password for root or sudo user.
The pre-requite check list warnings for ASM sharedness when block devices are used (this is not shown when AFD is used). This appear to be an issue similar to 12.1. However there's no MOS document exists for this at the time of the post. The installation was continued ignoring the warnings.
The installation setup summary (storage information shows Configure AFD : false).
When prompted run root scripts.
root.sh output for first node is shown below.
[root@rhel6m1 ~]# /opt/app/12.2.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m1/crsconfig/rootcrs_rhel6m1_2017-05-15_12-11-53AM.log
2017/05/15 12:11:57 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2017/05/15 12:11:57 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/05/15 12:12:29 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/05/15 12:12:29 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2017/05/15 12:12:38 CLSRSC-363: User ignored prerequisites during installation
2017/05/15 12:12:38 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2017/05/15 12:12:41 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2017/05/15 12:12:44 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2017/05/15 12:12:56 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2017/05/15 12:12:59 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2017/05/15 12:12:59 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2017/05/15 12:13:37 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2017/05/15 12:13:53 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2017/05/15 12:13:53 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2017/05/15 12:14:04 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2017/05/15 12:14:19 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
2017/05/15 12:14:49 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2017/05/15 12:14:59 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/05/15 12:15:57 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2017/05/15 12:16:07 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded

Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-170515PM121648.log for details.


2017/05/15 12:17:45 CLSRSC-482: Running command: '/opt/app/12.2.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m1'
CRS-2676: Start of 'ora.storage' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1'
CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 5dcf6fe02fa14f27bf3597660cd83e26.
Successful addition of voting disk 4cabbda7d3694f32bff3745c4ab89112.
Successful addition of voting disk 3fb1474faee54fa2bfac45159c621cc5.
Successfully replaced voting disk group with +clusterdg.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   5dcf6fe02fa14f27bf3597660cd83e26 (/dev/oracleasm/ocr3) [CLUSTERDG]
 2. ONLINE   4cabbda7d3694f32bff3745c4ab89112 (/dev/oracleasm/ocr1) [CLUSTERDG]
 3. ONLINE   3fb1474faee54fa2bfac45159c621cc5 (/dev/oracleasm/ocr2) [CLUSTERDG]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/05/15 12:19:50 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1'
CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m1'
CRS-2676: Start of 'ora.storage' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m1'
CRS-2676: Start of 'ora.crf' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1'
CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel6m1
CRS-6016: Resource auto-start has completed for server rhel6m1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/05/15 12:22:35 CLSRSC-343: Successfully started Oracle Clusterware stack
2017/05/15 12:22:35 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTERDG.dg' on 'rhel6m1'
CRS-2676: Start of 'ora.CLUSTERDG.dg' on 'rhel6m1' succeeded
2017/05/15 12:25:10 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.

2017/05/15 12:26:08 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
root.sh output for the second node.
[root@rhel6m2 ~]# /opt/app/12.2.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m2/crsconfig/rootcrs_rhel6m2_2017-05-15_12-27-18AM.log
2017/05/15 12:27:22 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2017/05/15 12:27:22 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/05/15 12:27:54 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/05/15 12:27:54 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2017/05/15 12:27:55 CLSRSC-363: User ignored prerequisites during installation
2017/05/15 12:27:55 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2017/05/15 12:27:56 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2017/05/15 12:27:56 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2017/05/15 12:28:01 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2017/05/15 12:28:04 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2017/05/15 12:28:04 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2017/05/15 12:28:07 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2017/05/15 12:28:09 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2017/05/15 12:28:09 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2017/05/15 12:28:12 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2017/05/15 12:28:27 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
2017/05/15 12:28:47 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2017/05/15 12:28:48 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/05/15 12:29:34 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2017/05/15 12:29:36 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/05/15 12:29:51 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m2'
CRS-2676: Start of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m2'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m2'
CRS-2676: Start of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m2'
CRS-2676: Start of 'ora.crf' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m2'
CRS-2676: Start of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m2'
CRS-2676: Start of 'ora.asm' on 'rhel6m2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel6m2
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.net1.network' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel6m2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m2'
CRS-2676: Start of 'ora.ons' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel6m2' succeeded
CRS-6016: Resource auto-start has completed for server rhel6m2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/05/15 12:32:14 CLSRSC-343: Successfully started Oracle Clusterware stack
2017/05/15 12:32:14 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2017/05/15 12:32:36 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2017/05/15 12:32:49 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
At the end of the root.sh run the cluster version and mode could quried.
[root@rhel6m1 bin]#  crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]

[root@rhel6m1 bin]# crsctl get cluster mode status
Cluster is running in "flex" mode

[root@rhel6m1 bin]# crsctl get node role config -all
Node 'rhel6m1' configured role is 'hub'
Node 'rhel6m2' configured role is 'hub'

[root@rhel6m1 bin]#  srvctl config gns
PRKF-1110 : Neither GNS server nor GNS client is configured on this cluster

ASMCMD> showclustermode
ASM cluster : Flex mode enabled
As seen from the output above the cluster mode is flex mode. However unlike in 12.1 flex cluster, GNS was not setup during the installation. It was confirmed through a SR that setting up GNS for standalone cluster is not required even though cluster mode appear as flex.
Once the root scripts are run click OK on the configuration dialogue box to complete the creation of GIMR and other configuration assistants. At the end this clusterware installation would be completed.
With this conclude the installation of 12.2 clusterware.
As said in the beginning the GIMR will be used for storing OCR backup.
ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+MGMTDG]

ocrconfig -showbackup auto

rhel6m1     2017/05/17 06:34:03     +MGMTDG:/ora12c2-cluster/OCRBACKUP/backup00.ocr.287.944202839     0

rhel6m1     2017/05/17 02:33:57     +MGMTDG:/ora12c2-cluster/OCRBACKUP/backup01.ocr.285.944188433     0

rhel6m1     2017/05/16 22:33:52     +MGMTDG:/ora12c2-cluster/OCRBACKUP/backup02.ocr.284.944174029     0

rhel6m1     2017/05/16 18:33:46     +MGMTDG:/ora12c2-cluster/OCRBACKUP/day.ocr.282.944159627     0

rhel6m1     2017/05/16 18:33:46     +MGMTDG:/ora12c2-cluster/OCRBACKUP/week.ocr.283.944159627     0

# ocrconfig -manualbackup

rhel6m1     2017/05/25 15:39:44     +MGMTDG:/ora12c2-cluster/OCRBACKUP/backup_20170525_153944.ocr.287.944926785     0
However, unlike in 12.1 where it gave the option of both file system and ASM for OCR backup, it seems that in 12.2 OCR backup location must be an ASM disk group.
 ocrconfig -help
Name:
        ocrconfig - Configuration tool for Oracle Cluster/Local Registry.

Synopsis:
        ocrconfig [option]
        option:
                -backuploc +<diskgroupname>         - Configure OCR backup location
Trying to set to a file system location will fail as shown below
# ocrconfig -backuploc /opt/app/12.2.0/grid/cdata/ora12c2-cluster
PROT-42: The specified backup location '/opt/app/12.2.0/grid/cdata/ora12c2-cluster' is not an ASM disk group.
At the time of writing Oracle documentation for 12.2 still state that file system is an option for OCR backups. Furthermore the clusterware admin guide states "If the OCR backup file is located in an Oracle ASM disk group, then ensure that the disk group exists and is mounted." Depending on the failure it may not be able to mount the ASM disk containing OCR backups. OCR backups could be copied out of ASM disk group using ASMCMD
asmcmd cp +MGMTDG/ora12c2-cluster/OCRBACKUP/backup_20170515_130825.ocr.281.944053705 /home/grid/
copying +MGMTDG/ora12c2-cluster/OCRBACKUP/backup_20170515_130825.ocr.281.944053705 -> /home/grid//backup_20170515_130825.ocr.281.944053705
ASM configuration list ASM instance count as 3 even though this is a 2 node cluster. This is related to flex cluster configuration where default ASM instance count is 3 even when actual node count is 2.
srvctl config asm -a
ASM home: 
Password file: +CLUSTERDG/orapwASM
Backup of Password file:
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM
The ASMCA interface has also been changed in 12.2. Create the other disk group needed for data and FRA.


Using ASM Filter Driver During Installation
In order to use AFD to mange the disk sued by ASM carry out the following before the installation (before running gridSetup.sh). At this stage is assumed that grid software is extracted to grid home location. set ORACLE_HOME AND ORACLE_BASE variables as below. Oracle base is set to a temp location so that files generated during ASM disk labelling doesn't interfere with actual installation.
# export ORACLE_HOME=/opt/app/12.2.0/grid
# export ORACLE_BASE=/tmp
Run the disk labelling command from the $GI_HOME/bin directory.
# ./asmcmd afd_label DATA1 /dev/oracleasm/data1 --init
# ./asmcmd afd_label FRA1 /dev/oracleasm/fra1 --init

# ./asmcmd afd_label OCR1 /dev/oracleasm/ocr1 --init
# ./asmcmd afd_label OCR2 /dev/oracleasm/ocr2 --init
# ./asmcmd afd_label OCR3 /dev/oracleasm/ocr3 --init

# ./asmcmd afd_label GIMR /dev/oracleasm/gimr --init
Verify the disks are labelled.
# ./asmcmd afd_lslbl '/dev/oracleasm/*'
--------------------------------------------------------------------------------
Label                     Duplicate  Path
================================================================================
DATA1                                 /dev/oracleasm/data1
FRA1                                  /dev/oracleasm/fra1
GIMR                                  /dev/oracleasm/gimr
OCR1                                  /dev/oracleasm/ocr1
OCR2                                  /dev/oracleasm/ocr2
OCR3                                  /dev/oracleasm/ocr3
Afterwards run the gridSetup.sh as before. Give the ASM disk path string as before when it is time to specify the disks for ASM. The listed disks will have the status "provisioned".
With AFD in use ASM sharedness warning is not shown as before.
The rest of the steps are the same. The summary will have Configure Oracle AFD : true.
During the root.sh run the file name for ASM diskgroup would have the AFD: prefix.
Successfully replaced voting disk group with +CLUSTERDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   5525126aef6f4fb4bf9b383115bade73 (AFD:OCR3) [CLUSTERDG]
 2. ONLINE   a5330a1d40594f31bfcc9ee98c6b0034 (AFD:OCR2) [CLUSTERDG]
 3. ONLINE   ad0588fa57fa4f13bfd0800d4e52ea5d (AFD:OCR1) [CLUSTERDG]
Located 3 voting disk(s).
At the end of the installation process asmcmd will show following string as discovery string
asmcmd afd_dsget
AFD discovery string: /dev/oracleasm/*
while querying same from within an ASM instance will show both above disk string and AFD:*
SQL> show parameter string

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------
asm_diskstring                       string      /dev/oracleasm/*, AFD:*
v$asm_disk will have both AFD and /dev/oracleasm/* paths
SQL> select name,path from v$asm_disk order by 2;

NAME                           PATH
------------------------------ ------------------------------
                               /dev/oracleasm/data1
                               /dev/oracleasm/fra1
                               /dev/oracleasm/gimr
                               /dev/oracleasm/ocr1
                               /dev/oracleasm/ocr2
                               /dev/oracleasm/ocr3
                               AFD:DATA1
                               AFD:FRA1
GIMR                           AFD:GIMR
OCR1                           AFD:OCR1
OCR2                           AFD:OCR2
OCR3                           AFD:OCR3
However, when ASMCA is run to create additional diskgroups, the disk paths are shown as AFD paths.

Finally run the post crsisnt to verify cluster status.
$ cluvfy stage -post crsinst -n rhel6m1,rhel6m2 -verbose

Verifying Node Connectivity ...
  Verifying Hosts File ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m1                               passed
  rhel6m2                               passed
  Verifying Hosts File ...PASSED

Interface information for node "rhel6m2"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth0   192.168.0.98    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth0   192.168.0.135   192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth0   192.168.0.145   192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:69:2C:B6 1500

Interface information for node "rhel6m1"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth0   192.168.0.97    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth0   192.168.0.125   192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:A3:C4:6F 1500

Check: MTU consistency on the private interfaces of subnet "192.168.1.0"

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           eth1          192.168.1.88  192.168.1.0   1500
  rhel6m1           eth1          192.168.1.87  192.168.1.0   1500

Check: MTU consistency of the subnet "192.168.0.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           eth0          192.168.0.94  192.168.0.0   1500
  rhel6m2           eth0          192.168.0.98  192.168.0.0   1500
  rhel6m2           eth0          192.168.0.135  192.168.0.0   1500
  rhel6m2           eth0          192.168.0.145  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.93  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.97  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.125  192.168.0.0   1500

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[eth1:192.168.1.87]      rhel6m2[eth1:192.168.1.88]      yes

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.98]      yes
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.135]     yes
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.145]     yes
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.94]      yes
  rhel6m1[eth0:192.168.0.93]      rhel6m1[eth0:192.168.0.97]      yes
  rhel6m1[eth0:192.168.0.93]      rhel6m1[eth0:192.168.0.125]     yes
  rhel6m2[eth0:192.168.0.98]      rhel6m2[eth0:192.168.0.135]     yes
  rhel6m2[eth0:192.168.0.98]      rhel6m2[eth0:192.168.0.145]     yes
  rhel6m2[eth0:192.168.0.98]      rhel6m2[eth0:192.168.0.94]      yes
  rhel6m2[eth0:192.168.0.98]      rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.98]      rhel6m1[eth0:192.168.0.125]     yes
  rhel6m2[eth0:192.168.0.135]     rhel6m2[eth0:192.168.0.145]     yes
  rhel6m2[eth0:192.168.0.135]     rhel6m2[eth0:192.168.0.94]      yes
  rhel6m2[eth0:192.168.0.135]     rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.135]     rhel6m1[eth0:192.168.0.125]     yes
  rhel6m2[eth0:192.168.0.145]     rhel6m2[eth0:192.168.0.94]      yes
  rhel6m2[eth0:192.168.0.145]     rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.145]     rhel6m1[eth0:192.168.0.125]     yes
  rhel6m2[eth0:192.168.0.94]      rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.94]      rhel6m1[eth0:192.168.0.125]     yes
  rhel6m1[eth0:192.168.0.97]      rhel6m1[eth0:192.168.0.125]     yes
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast check ...
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Cluster Manager Integrity ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m1                               running
  rhel6m2                               running
Verifying Cluster Manager Integrity ...PASSED
Verifying User Mask ...
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       0022                      0022                      passed
  rhel6m1       0022                      0022                      passed
Verifying User Mask ...PASSED
Verifying Cluster Integrity ...
  Node Name
  ------------------------------------
  rhel6m1
  rhel6m2
Verifying Cluster Integrity ...PASSED
Verifying OCR Integrity ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...

Checking existence of VIP node application (required)

  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       yes                       yes                       passed
  rhel6m1       yes                       yes                       passed


Checking existence of NETWORK node application (required)

  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       yes                       yes                       passed
  rhel6m1       yes                       yes                       passed


Checking existence of ONS node application (optional)

  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       no                        yes                       passed
  rhel6m1       no                        yes                       passed
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
  SCAN Name         Node          Running?      ListenerName  Port          Running?
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac-scan          rhel6m1       true          LISTENER_SCAN1  1521          true
  rac-scan          rhel6m2       true          LISTENER_SCAN2  1521          true
  rac-scan          rhel6m2       true          LISTENER_SCAN3  1521          true

Checking TCP connectivity to SCAN listeners...

  Node          ListenerName              TCP connectivity?
  ------------  ------------------------  ------------------------
  rhel6m2       LISTENER_SCAN1            yes
  rhel6m2       LISTENER_SCAN2            yes
  rhel6m2       LISTENER_SCAN3            yes

  Verifying DNS/NIS name service 'rac-scan' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED

  SCAN Name     IP Address                Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  rac-scan      192.168.0.145             passed
  rac-scan      192.168.0.125             passed
  rac-scan      192.168.0.135             passed
  Verifying DNS/NIS name service 'rac-scan' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Voting Disk ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
    Verifying Hosts File ...PASSED

Interface information for node "rhel6m2"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth0   192.168.0.98    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth0   192.168.0.135   192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth0   192.168.0.145   192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:69:2C:B6 1500

Interface information for node "rhel6m1"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth0   192.168.0.97    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth0   192.168.0.125   192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:A3:C4:6F 1500

Check: MTU consistency on the private interfaces of subnet "192.168.1.0"

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           eth1          192.168.1.88  192.168.1.0   1500
  rhel6m1           eth1          192.168.1.87  192.168.1.0   1500

Check: MTU consistency of the subnet "192.168.0.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           eth0          192.168.0.94  192.168.0.0   1500
  rhel6m2           eth0          192.168.0.98  192.168.0.0   1500
  rhel6m2           eth0          192.168.0.135  192.168.0.0   1500
  rhel6m2           eth0          192.168.0.145  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.93  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.97  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.125  192.168.0.0   1500

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[eth1:192.168.1.87]      rhel6m2[eth1:192.168.1.88]      yes

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.98]      yes
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.135]     yes
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.145]     yes
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.94]      yes
  rhel6m1[eth0:192.168.0.93]      rhel6m1[eth0:192.168.0.97]      yes
  rhel6m1[eth0:192.168.0.93]      rhel6m1[eth0:192.168.0.125]     yes
  rhel6m2[eth0:192.168.0.98]      rhel6m2[eth0:192.168.0.135]     yes
  rhel6m2[eth0:192.168.0.98]      rhel6m2[eth0:192.168.0.145]     yes
  rhel6m2[eth0:192.168.0.98]      rhel6m2[eth0:192.168.0.94]      yes
  rhel6m2[eth0:192.168.0.98]      rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.98]      rhel6m1[eth0:192.168.0.125]     yes
  rhel6m2[eth0:192.168.0.135]     rhel6m2[eth0:192.168.0.145]     yes
  rhel6m2[eth0:192.168.0.135]     rhel6m2[eth0:192.168.0.94]      yes
  rhel6m2[eth0:192.168.0.135]     rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.135]     rhel6m1[eth0:192.168.0.125]     yes
  rhel6m2[eth0:192.168.0.145]     rhel6m2[eth0:192.168.0.94]      yes
  rhel6m2[eth0:192.168.0.145]     rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.145]     rhel6m1[eth0:192.168.0.125]     yes
  rhel6m2[eth0:192.168.0.94]      rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.94]      rhel6m1[eth0:192.168.0.125]     yes
  rhel6m1[eth0:192.168.0.97]      rhel6m1[eth0:192.168.0.125]     yes
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...
Checking consistency of device owner across all nodes...
Consistency check of device owner for "/dev/oracleasm/gimr" PASSED
Consistency check of device owner for "/dev/oracleasm/fra1" PASSED
Consistency check of device owner for "/dev/oracleasm/data1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr3" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device group across all nodes...
Consistency check of device group for "/dev/oracleasm/gimr" PASSED
Consistency check of device group for "/dev/oracleasm/fra1" PASSED
Consistency check of device group for "/dev/oracleasm/data1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr3" PASSED
Consistency check of device group for "/dev/oracleasm/ocr1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/dev/oracleasm/gimr" PASSED
Consistency check of device permissions for "/dev/oracleasm/fra1" PASSED
Consistency check of device permissions for "/dev/oracleasm/data1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr3" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr2" PASSED
  Verifying Access Control List check ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying I/O scheduler ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying I/O scheduler ...PASSED
Verifying User Not In Group "root": grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    does not exist
  rhel6m1       passed                    does not exist
Verifying User Not In Group "root": grid ...PASSED
Verifying Clock Synchronization ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m1                               passed
  rhel6m2                               passed

  Node Name                             State
  ------------------------------------  ------------------------
  rhel6m2                               Active
  rhel6m1                               Active

  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  rhel6m2       0.0                       passed
  rhel6m1       0.0                       passed
Verifying Clock Synchronization ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED

Post-check for cluster services setup was successful.

CVU operation performed:      stage -post crsinst
Date:                         May 15, 2017 1:18:38 PM
CVU home:                     /opt/app/12.2.0/grid/
User:                         grid
Finally make sure that $ORACLE_BASE/audit and $ORACLE_BASE/admin folders are writeable for both grid and oracle users on all the nodes.
The next step is to install the database software and create the database.

Related Posts
Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)