A Multipath disk provide failover by using redundant physical path components, eliminating a single point of failure with the Storage Area Network (SAN). Multipathing is a software technology implemented at the operating system device driver level.
With ASM, you can ensure the discovery of a multipath disk by setting the value of the initialization parameter ASM_DISKSTRING equal to the name of the pseudo device that represents the multipath disk.
Below steps can be followed to add a new disk to ASM Disk Group.
STEP 1) CHECK CURRENT DISKS
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
SQL> select GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,VOTING_FILE,name,path from v$asm_disk;
GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE V NAME PATH
———— ———– ——- ———— ——- ——– – ————– ———————————
2 0 CACHED MEMBER ONLINE NORMAL N ARC_LOG_0000 /dev/oracleasm/disks/ARC_LOG01
1 0 CACHED MEMBER ONLINE NORMAL Y OCR_VOTE_0000 /dev/oracleasm/disks/OCR_VOTE01
3 4 CACHED MEMBER ONLINE NORMAL N DB_DATA_0004 /dev/oracleasm/disks/DB_DATA05
3 3 CACHED MEMBER ONLINE NORMAL N DB_DATA_0003 /dev/oracleasm/disks/DB_DATA04
3 2 CACHED MEMBER ONLINE NORMAL N DB_DATA_0002 /dev/oracleasm/disks/DB_DATA03
3 1 CACHED MEMBER ONLINE NORMAL N DB_DATA_0001 /dev/oracleasm/disks/DB_DATA02
3 0 CACHED MEMBER ONLINE NORMAL N DB_DATA_0000 /dev/oracleasm/disks/DB_DATA01
3 5 CACHED MEMBER ONLINE NORMAL N DB_DATA_0005 /dev/oracleasm/disks/DB_DATA06
8 rows selected.
|
STEP 2) CHECK ASM TO PHYSICAL DISK MAPPING
If you have not documented earlier which ASM disk is mapped to which physical disk and what all free disks you have in your server, your need t firs find out this information.
Use asm querydisk command first with every ASM disk name:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
# /etc/init.d/oracleasm querydisk -d DB_DATA01
Disk “DB_DATA01” is a valid ASM disk on device /dev/sdl[8,176]
# /etc/init.d/oracleasm querydisk -d DB_DATA02
Disk “DB_DATA02” is a valid ASM disk on device /dev/sdm[8,192]
# /etc/init.d/oracleasm querydisk -d DB_DATA03
Disk “DB_DATA03” is a valid ASM disk on device /dev/sdn[8,208]
# /etc/init.d/oracleasm querydisk -d DB_DATA04
Disk “DB_DATA04” is a valid ASM disk on device /dev/sdo[8,224]
# /etc/init.d/oracleasm querydisk -d DB_DATA05
Disk “DB_DATA05” is a valid ASM disk on device /dev/sdp[8,240]
# /etc/init.d/oracleasm querydisk -d DB_DATA06
Disk “DB_DATA06” is a valid ASM disk on device /dev/sdq[65,0]
# /etc/init.d/oracleasm querydisk -d ARC_LOG01
Disk “ARC_LOG01” is a valid ASM disk on device /dev/sdr[65,16]
# /etc/init.d/oracleasm querydisk -d OCR_VOTE01
Disk “OCR_VOTE01” is a valid ASM disk on device /dev/sdb1[8,17]
|
Now use multipath -ll to check corresponding multipath disk and what all disks are free in your server.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
|
# multipath -ll
mpath7 (360060e8006d8e7000000d8e700001000) dm–17 HP,OPEN–V
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round–robin 0 [prio=0][active]
\_ 3:0:0:0 sdb 8:16 [active][ready]
\_ 4:0:0:0 sdv 65:80 [active][ready]
mpath18 (360060e8006d8e7000000d8e700000001) dm–8 HP,OPEN–V
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round–robin 0 [prio=0][active]
\_ 4:0:0:11 sdag 66:0 [active][ready]
\_ 3:0:0:11 sdm 8:192 [active][ready]
mpath19 (360060e8006d8e7000000d8e700000002) dm–9 HP,OPEN–V
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round–robin 0 [prio=0][active]
\_ 4:0:0:12 sdah 66:16 [active][ready]
\_ 3:0:0:12 sdn 8:208 [active][ready]
mpath20 (360060e8006d8e7000000d8e700000003) dm–10 HP,OPEN–V
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round–robin 0 [prio=0][active]
\_ 4:0:0:13 sdai 66:32 [active][ready]
\_ 3:0:0:13 sdo 8:224 [active][ready]
mpath21 (360060e8006d8e7000000d8e700000004) dm–11 HP,OPEN–V
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round–robin 0 [prio=0][active]
\_ 4:0:0:14 sdaj 66:48 [active][ready]
\_ 3:0:0:14 sdp 8:240 [active][ready]
mpath22 (360060e8006d8e7000000d8e700000005) dm–12 HP,OPEN–V
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round–robin 0 [prio=0][active]
\_ 4:0:0:15 sdak 66:64 [active][ready]
\_ 3:0:0:15 sdq 65:0 [active][ready]
mpath23 (360060e8006d8e7000000d8e700000006) dm–13 HP,OPEN–V
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round–robin 0 [prio=0][active]
\_ 4:0:0:16 sdal 66:80 [active][ready]
\_ 3:0:0:16 sdr 65:16 [active][ready]
mpath24 (360060e8006d8e7000000d8e700000007) dm–14 HP,OPEN–V
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round–robin 0 [prio=0][active]
\_ 4:0:0:17 sdam 66:96 [active][ready]
\_ 3:0:0:17 sds 65:32 [active][ready]
|
Looking at the above two outputs we can find out which ASM disk is attached to what physical disk
For example DB_DATA02 (on /dev/sdm) is attached to Physical multi disk path mpath18.
mpath24 can be seen as free physical disk which is not linked to any ASM disk yet.
STEP 3) PARTITION THE DISK
We will use the free disk /dev/mapper/mpath24
|
1
2
3
4
5
6
7
|
# fdisk -l /dev/mapper/mpath24
Disk /dev/mapper/mpath24: 107.3 GB, 107374510080 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mapper/mpath24 doesn’t contain a valid partition table
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
# fdisk /dev/mapper/mpath24
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won’t be recoverable.
The number of cylinders for this disk is set to 13054.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): u
Changing display/entry units to sectors
Command (m for help): n
Command action
e extended
p primary partition (1–4)
p
Partition number (1–4): 1
First sector (63–209715839, default 63): 2048
Last sector or +size or +sizeM or +sizeK (2048–209715839, default 209715839):
Using default value 209715839
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re–read partition table.
WARNING: Re–reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
|
|
1
|
# /sbin/partprobe
|
|
1
2
3
4
5
6
7
8
|
# fdisk -l /dev/mapper/mpath24
Disk /dev/mapper/mpath24: 107.3 GB, 107374510080 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/mapper/mpath24p1 1 13055 104856896 83 Linux
|
|
1
|
# kpartx -a /dev/mapper/mpath24
|
STEP 4) CREATE ASM DISK
|
1
2
|
# /etc/init.d/oracleasm createdisk ARC_LOG02 /dev/mapper/mpath24p1
Marking disk “ARC_LOG02” as an ASM disk: [ OK ]
|
Run below command on all database nodes (if using RAC):
|
1
2
|
# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
|
|
1
2
3
4
5
6
7
8
9
10
|
# /etc/init.d/oracleasm listdisks
ARC_LOG01
ARC_LOG02
DB_DATA01
DB_DATA02
DB_DATA03
DB_DATA04
DB_DATA05
DB_DATA06
OCR_VOTE01
|
The new disk should be visible from all database nodes now.
Now we have ASM disk ready to be added to ASM Disk Group
STEP 5) ADD ASM DISK TO DISKGROUP
Check Current Status >
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
SQL> set lines 132
SQL> col name format a14
SQL> col PATH format a33
SQL> select GROUP_NUMBER,NAME,STATE,TYPE,TOTAL_MB,FREE_MB,VOTING_FILES from v$asm_diskgroup;
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB V
———— ————– ———– —— ———- ———- –
1 ARC_LOG MOUNTED EXTERN 102400 101705 N
2 DB_DATA MOUNTED EXTERN 614400 167345 N
3 OCR_VOTE MOUNTED EXTERN 51199 50803 Y
SQL> select GROUP_NUMBER,DISK_NUMBER,STATE,name,path,MOUNT_STATUS,HEADER_STATUS from v$asm_disk;
GROUP_NUMBER DISK_NUMBER STATE NAME PATH MOUNT_S HEADER_STATU
———— ———– ——– ————– ——————————— ——- ————
0 0 NORMAL /dev/oracleasm/disks/ARC_LOG02 CLOSED PROVISIONED
2 0 NORMAL ARC_LOG_0000 /dev/oracleasm/disks/ARC_LOG01 CACHED MEMBER
1 0 NORMAL OCR_VOTE_0000 /dev/oracleasm/disks/OCR_VOTE01 CACHED MEMBER
3 4 NORMAL DB_DATA_0004 /dev/oracleasm/disks/DB_DATA05 CACHED MEMBER
3 3 NORMAL DB_DATA_0003 /dev/oracleasm/disks/DB_DATA04 CACHED MEMBER
3 2 NORMAL DB_DATA_0002 /dev/oracleasm/disks/DB_DATA03 CACHED MEMBER
3 1 NORMAL DB_DATA_0001 /dev/oracleasm/disks/DB_DATA02 CACHED MEMBER
3 0 NORMAL DB_DATA_0000 /dev/oracleasm/disks/DB_DATA01 CACHED MEMBER
3 5 NORMAL DB_DATA_0005 /dev/oracleasm/disks/DB_DATA06 CACHED MEMBER
9 rows selected.
|
As you can see that the new disk’s status is showing as ‘PROVISIONED’.
NOW ADD NEW DISK :
|
1
|
SQL> ALTER DISKGROUP ARC_LOG ADD DISK ‘/dev/oracleasm/disks/ARC_LOG02’ NOFORCE ;
|
You can verify the rebalancing operation by below SQL command.
|
1
2
3
4
5
|
SQL> select * from v$asm_operation ;
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
———— —– —- ———- ———- ———- ———- ———————————-
1 REBAL RUN 1 1 5046 5631 4780 0
|
STEP 6) CHECK NEW STATUS
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
SQL> select GROUP_NUMBER,NAME,STATE,TYPE,TOTAL_MB,FREE_MB,VOTING_FILES from v$asm_diskgroup;
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB V
———— ————– ———– —— ———- ———- –
1 ARC_LOG MOUNTED EXTERN 204799 192963 N
2 DB_DATA MOUNTED EXTERN 614400 145525 N
3 OCR_VOTE MOUNTED EXTERN 51199 50803 Y
SQL> select GROUP_NUMBER,DISK_NUMBER,STATE,name,path,MOUNT_STATUS,HEADER_STATUS from v$asm_disk;
GROUP_NUMBER DISK_NUMBER STATE NAME PATH MOUNT_S HEADER_STATU
———— ———– ——– ————– ——————————— ——- ————
2 0 NORMAL ARC_LOG_0000 /dev/oracleasm/disks/ARC_LOG01 CACHED MEMBER
1 0 NORMAL OCR_VOTE_0000 /dev/oracleasm/disks/OCR_VOTE01 CACHED MEMBER
3 4 NORMAL DB_DATA_0004 /dev/oracleasm/disks/DB_DATA05 CACHED MEMBER
3 3 NORMAL DB_DATA_0003 /dev/oracleasm/disks/DB_DATA04 CACHED MEMBER
3 2 NORMAL DB_DATA_0002 /dev/oracleasm/disks/DB_DATA03 CACHED MEMBER
3 1 NORMAL DB_DATA_0001 /dev/oracleasm/disks/DB_DATA02 CACHED MEMBER
3 0 NORMAL DB_DATA_0000 /dev/oracleasm/disks/DB_DATA01 CACHED MEMBER
3 5 NORMAL DB_DATA_0005 /dev/oracleasm/disks/DB_DATA06 CACHED MEMBER
2 1 NORMAL ARC_LOG_0001 /dev/oracleasm/disks/ARC_LOG02 CACHED MEMBER
9 rows selected.
|
I’m an experienced Cloud/Oracle Applications/DBA Architect with more than 15 years of full-time DBA/Architect experience. I have gained wide knowledge on Oracle and Non-Oracle software stack running on-prem and on Cloud and have worked on several big projects for multi-national companies. I enjoy working with leading-edge technology and have a passion for Cloud architecture, automation, database performance, and stability. Thankfully my work allows me time for researching new technologies (and to write about them).
Latest posts by Brijesh Gogia (see all)
- Oracle Multitenant DB 4 : Parameters/SGA/PGA management in CDB-PDB - July 18, 2020
- Details of running concurrent requests in Oracle EBS - May 26, 2020
- Oracle Multitenant DB 3 : Data Dictionary Architecture in CDB-PDB - March 20, 2020
Thanks Brijesh……thank you for sharing this post its really helpful
Its a wonderful post, I ever read.
Thanku so much.
Thanks Brijesh… your post was so helpful for me.