1. As the grid user login to ASM instance and create a pfile from the ASM spfile
sqlplus / as sysasm SQL> create pfile='/home/grid/asmpfile.ora' from spfile; File created. $ more asmpfile.ora +ASM.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from in memory value +ASM.asm_diskgroups='FLASH','DATA'#Manual Mount *.asm_power_limit=1 *.diagnostic_dest='/opt/app/oracle' *.instance_type='asm' *.large_pool_size=12M *.remote_login_passwordfile='EXCLUSIVE'2. Current ASMLib disks are
SQL> select name,label,path from v$asm_disk; NAME LABEL PATH ------- ------- ----------- DATA DATA ORCL:DATA FLASH FLASH ORCL:FLASH3. Before ASMLib disks could be moved the database must be stopped. This is required as ASM diskgroups will be dismounted later on.
srvctl stop database -d stdalone4. Dismount and mount the flash diskgroup and monitor the ASM alert log which will show the ASM disk(s) used with the diskgroup. This information is useful to verify if the new block device disk assignment is working.
SQL> alter diskgroup flash dismount; Diskgroup altered.From the alert log
NOTE: cache deleting context for group FLASH 2/0x9328489f Thu Jun 14 11:57:49 2012 NOTE: diskgroup resource ora.FLASH.dg is offline SUCCESS: alter diskgroup flash dismount NOTE: diskgroup resource ora.FLASH.dg is updatedWhen mounting
SQL> alter diskgroup flash mountFrom alert log
NOTE: cache registered group FLASH number=2 incarn=0xed0848a6 NOTE: cache began mount (first) of group FLASH number=2 incarn=0xed0848a6 NOTE: Assigning number (2,0) to disk (ORCL:FLASH) Thu Jun 14 11:58:18 2012 NOTE: GMON heartbeating for grp 25. Identify the block device used to create the ASMLib disk by querying the ASMLib disk
# /etc/init.d/oracleasm querydisk -d FLASH Disk "FLASH" is a valid ASM disk on device [8, 33]Block device could be identified with
ls -l /dev/sd* brw-r----- 1 root disk 8, 0 Jun 14 11:47 /dev/sda brw-r----- 1 root disk 8, 1 Jun 14 11:48 /dev/sda1 brw-r----- 1 root disk 8, 2 Jun 14 11:47 /dev/sda2 brw-r----- 1 root disk 8, 3 Jun 14 11:48 /dev/sda3 brw-r----- 1 root disk 8, 16 Jun 14 11:47 /dev/sdb brw-r----- 1 root disk 8, 17 Jun 14 11:48 /dev/sdb1 brw-r----- 1 root disk 8, 32 Jun 14 11:47 /dev/sdc brw-r----- 1 root disk 8, 33 Jun 14 11:48 /dev/sdc1The block device used to create the FLASH ASMLib disk is /dev/sdc1.
6. Set the permission on the block device to grid:asmadmin
# chown grid:asmadmin /dev/sdc1 # ls -l /dev/sdc* brw-r----- 1 root disk 8, 32 Jun 14 11:47 /dev/sdc brw-r----- 1 grid asmadmin 8, 33 Jun 14 11:48 /dev/sdc17. Dismount the FLASH diskgroup again and set the asm_diskstring to include the block device. Still the DATA diskgroup is using ASMLib therefore ORCL* is also set. This mix setting is alright for testing purposes but it's better not to run production environments in the mix mode (both ASMLib and Block devices).
SQL> alter system set asm_diskstring ='ORCL:DATA','/dev/sdc1' SCOPE=MEMORY;Mount the FLASH diskgroup and monitor the ASM alert log and verify block device is used
SQL> ALTER DISKGROUP flash mountOn the alert log
NOTE: cache registered group FLASH number=2 incarn=0x3d4848aa NOTE: cache began mount (first) of group FLASH number=2 incarn=0x3d4848aa NOTE: Assigning number (2,0) to disk (/dev/sdc1) Thu Jun 14 12:01:15 2012 NOTE: GMON heartbeating for grp 2 GMON querying group 2 at 18 for pid 19, osid 3268 NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdc1Further verification from the v$asm_disk view
SQL> select name,label,path from v$asm_disk; NAME LABEL PATH ------- ------- ----------- DATA DATA ORCL:DATA FLASH FLASH /dev/sdc18. Do the same with other diskgroup
# /etc/init.d/oracleasm querydisk -d DATA Disk "DATA" is a valid ASM disk on device [8, 17]Set the block device permissions
# chown grid:asmadmin /dev/sdb1 # ls -l /dev/sdb* brw-r----- 1 root disk 8, 16 Jun 14 11:47 /dev/sdb brw-r----- 1 grid asmadmin 8, 17 Jun 14 11:48 /dev/sdb19. Dismount all diskgroups. Trying to change the asm_diskstring while diskgroups are mounted will throw the following error
SQL> alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=memory; alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=memory * ERROR at line 1: ORA-02097: parameter cannot be modified because specified value is invalid ORA-15014: path 'ORCL:DATA' is not in the discovery setOnce all diskgroups are dismounted no issue.
SQL> alter diskgroup data dismount; Diskgroup altered. SQL> alter diskgroup flash dismount; Diskgroup altered. SQL> alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=both;Mount each diskgroup and verify the block device is used. Following from the alert log
Thu Jun 14 12:04:51 2012 SQL> alter diskgroup data mount NOTE: cache registered group DATA number=1 incarn=0xa8c848b4 NOTE: cache began mount (first) of group DATA number=1 incarn=0xa8c848b4 NOTE: Assigning number (1,0) to disk (/dev/sdb1)Further verification from v$asm_disk
SQL> select name,label,path from v$asm_disk; NAME LABEL PATH ------- ------- ----------- DATA DATA /dev/sdb1 FLASH FLASH /dev/sdc110. Now that it's been verified that block devices are used for ASM create udev rules.
# ASM DATA KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660" # ASM FLASH KERNEL=="sdc[1]", OWNER="grid", GROUP="asmadmin", MODE="660"11. Stop the high availability stack
crsctl stop hasand unload the oracleasm modules
# lsmod | grep oracleasm oracleasm 84136 1 # /etc/init.d/oracleasm stop Dropping Oracle ASMLib disks: [ OK ] Shutting down the Oracle ASMLib driver: [ OK ] # lsmod | grep oracleasmdisable loading of oracleasm when server is rebooted
# chkconfig oracleasm offRestart the high availability service
crsctl start has CRS-4123: Oracle High Availability Services has been started.and check all ASM starting with block devices without an issue. When the restart is complete check the ASM alert log
SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:2} */
NOTE: Diskgroups listed in ASM_DISKGROUPS are
DATA
FLASH
NOTE: cache registered group DATA number=1 incarn=0xa71a3a36
NOTE: cache began mount (first) of group DATA number=1 incarn=0xa71a3a36
NOTE: cache registered group FLASH number=2 incarn=0xa71a3a37
NOTE: cache began mount (first) of group FLASH number=2 incarn=0xa71a3a37
Errors in file /opt/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_rbal_4010.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Assigning number (1,0) to disk (/dev/sdb1)
NOTE: Assigning number (2,0) to disk (/dev/sdc1)
and/or using kfod tool to confirm block devices are being used correctlykfod asm_diskstring='/dev/sd*' disk=all
--------------------------------------------------------------------------------
Disk Size Path User Group
================================================================================
1: 10236 Mb /dev/sdb1 grid asmadmin
2: 10236 Mb /dev/sdc1 grid asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
+ASM /opt/app/oracle/product/11.2.0/grid
12. If possible restart the server and check all changes (udev rules, disk permissions) and are working without issue and as the final step uninstall the ASMLib # rpm -e oracleasmlib-2.0.4-1.el5 # rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5 # rpm -e oracleasm-support-2.1.3-1.el5Related Post
Migrating block devices using ASM instance to ASMLib
Moving ASMLib disk to block devices (Non-ASMLib) in 11gR1 RAC
Useful Metalink Notes
How To Migrate ASMLIB Devices To Block Devices (Non-ASMLIB)? [ID 567508.1]
