Database Sync

[증 상]
-Oracle Linux 9.4 + Oracle DB 19c 설치 과정에서 아래와 같은 메시지 출력

[원 인]
Oracle Linux 9 / Redhat Linux 9인 경우 Bug 33794281 - OL9: LIBPTHREAD_NONSHARED.A IS NEEDED FOR ORACLE DATABASE 23C AND 19C 임
해당 Bug는 DB RU 19.21에서 Fix 됨

그외 OS는 Oralce MOS예서 2760289.1 문서 확인 필요

[처리방법]
※주의 : Oracle Linux 9 / Redhat Linux 9인 경우에만 해당, 그외 OS 및 OS버전은 Oralce MOS예서 2760289.1 문서 확인 필요

-최신 OPatch로 교체

[oracle@ol9 OPatch]$
[oracle@ol9 OPatch]$ pwd
/u01/app/oracle/product/19.0.0/dbhome_1/OPatch
[oracle@ol9 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.17

OPatch succeeded.
[oracle@ol9 OPatch]$
[oracle@ol9 OPatch]$ cd ..
[oracle@ol9 dbhome_1]$ pwd
/u01/app/oracle/product/19.0.0/dbhome_1
[oracle@ol9 dbhome_1]$ mv OPatch OPatch_OLD
[oracle@ol9 dbhome_1]$
[oracle@ol9 dbhome_1]$
[oracle@ol9 dbhome_1]$ cd /u01/STAGE/
[oracle@ol9 STAGE]$ unzip -d $ORACLE_HOME p6880880_190000_Linux-x86-64.zip
Archive:  p6880880_190000_Linux-x86-64.zip
   creating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/
  inflating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatchauto
   creating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/ocm/
   creating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/ocm/doc/
   creating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/ocm/bin/
   creating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/ocm/lib/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  inflating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/modules/features/com.oracle.glcm.patch.opatch-common-api-classpath.jar
  inflating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/modules/com.sun.org.apache.xml.internal.resolver.jar
  inflating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/modules/com.oracle.glcm.patch.opatchauto-wallet_12.2.1.42.0.jar
  inflating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/modules/com.sun.xml.bind.jaxb-jxc.jar
  inflating: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/modules/javax.activation.javax.activation.jar
[oracle@ol9 STAGE]$
[oracle@ol9 STAGE]$ cd $ORACLE_HOME/OPatch
[oracle@ol9 OPatch]$ pwd
/u01/app/oracle/product/19.0.0/dbhome_1/OPatch
[oracle@ol9 OPatch]$
[oracle@ol9 OPatch]$ cd $ORACLE_HOME/OPatch
[oracle@ol9 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.42

OPatch succeeded.
[oracle@ol9 OPatch]$

-최신 RU를 적용 하고 설치 작업

ex)

$ export CV_ASSUME_DISTID=OL8                          <<<<<   If performing Installation on Oracle Linux 9
$ export CV_ASSUME_DISTID=RHEL8                        <<<<<   If performing Installation on Red Hat Linux 9
$ ./runInstaller -applyRU <19.23 DBRU Patch 36233263 unzip Location> -applyOneOffs <19.23 OCW RU 36240578 unzip location>

[oracle@ol9 dbhome_1]$
[oracle@ol9 dbhome_1]$ cd /u01/STAGE/RU/36233126
[oracle@ol9 36233126]$ ls -al
total 144
drwxr-x---. 8 oracle dba    159 Apr 12 10:03 .
drwxr-xr-x. 3 oracle dba     45 Jun 25 13:34 ..
drwxr-x---. 5 oracle dba     81 Apr 12 10:04 36233263
drwxr-x---. 5 oracle dba     62 Apr 12 10:03 36233343
drwxr-x---. 5 oracle dba     62 Apr 12 10:03 36240578
drwxr-x---. 4 oracle dba     48 Apr 12 10:04 36383196
drwxr-x---. 4 oracle dba     48 Apr 12 10:04 36460248
drwxr-x---. 2 oracle dba   4096 Apr 12 10:08 automation
-rw-rw-r--. 1 oracle dba   5824 Apr 12 19:44 bundle.xml
-rw-r--r--. 1 oracle dba 132988 Apr 16 21:15 README.html
-rw-r--r--. 1 oracle dba      0 Apr 12 10:08 README.txt
[oracle@ol9 36233126]$
[oracle@ol9 36233126]$ cd $ORACLE_HOME
[oracle@ol9 dbhome_1]$ ls
addnode     cfgtoollogs  ctx     dbs          dmu      has            inventory  jlib  mgw      olap        oracore  oui    precomp  rdbms        root.sh.old.1  slax          sqlplus   usm
apex        clone        cv      deinstall    drdaas   hs             javavm     ldap  network  OPatch      ord      owm    QOpatch  relnotes     runInstaller   sqldeveloper  srvm      utl
assistants  crs          data    demo         dv       install        jdbc       lib   nls      OPatch_OLD  ords     perl   R        root.sh      schagent.conf  sqlj          suptools  wwg
bin         css          dbjava  diagnostics  env.ora  instantclient  jdk        md    odbc     opmn        oss      plsql  racg     root.sh.old  sdk            sqlpatch      ucp       xdk
[oracle@ol9 dbhome_1]$
[oracle@ol9 dbhome_1]$ export CV_ASSUME_DISTID=OL8
[oracle@ol9 dbhome_1]$ ./runInstaller -applyRU /u01/STAGE/RU/36233126/36233263 -applyOneOffs /u01/STAGE/RU/36233126/36240578
Preparing the home to patch...
Applying the patch /u01/STAGE/RU/36233126/36233263...
Successfully applied the patch.
Applying the patch /u01/STAGE/RU/36233126/36240578...
Successfully applied the patch.
The log can be found at: /tmp/InstallActions2024-06-25_02-14-43PM/installerPatchActions_2024-06-25_02-14-43PM.log
Launching Oracle Database Setup Wizard...

The response file for this session can be found at:
 /u01/app/oracle/product/19.0.0/dbhome_1/install/response/db_2024-06-25_02-14-43PM.rsp

You can find the log of this install session at:
 /tmp/InstallActions2024-06-25_02-14-43PM/installActions2024-06-25_02-14-43PM.log

[참고문서]

19c Database Installation/relink fails with :"Error in invoking target 'libasmclntsh19.ohso libasmperl19.ohso client_sharedlib' of makefile ins_rdbms.mk " (문서 ID 2760289.1)
Requirements for Installing Oracle Database/Client 19c (19.22 or higher) on OL9 or RHEL9 64-bit (x86-64) (문서 ID 2982833.1)

[상태 확인]

[oracle@datarac1 ~]$ sqlplus "/as sysdba"
SQL*Plus: Release 19.0.0.0.0 - Production on Wed May 31 09:51:29 2023
Version 19.19.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
SQL> col USERNAME for a15
SQL> select USER_ID, USERNAME, ACCOUNT_STATUS from dba_users where USERNAME in ('SYS','SYSTEM');
   USER_ID USERNAME        ACCOUNT_STATUS
---------- --------------- --------------------------------
         0 SYS             OPEN
         9 SYSTEM          OPEN
SQL>

[SYSTEM User 삭제]

SQL> show user
USER is "SYS"
SQL>
SQL> drop user system;
drop user system
*
ERROR at line 1:
ORA-28050: specified user or role cannot be dropped
SQL> drop user system CASCADE;
drop user system CASCADE
*
ERROR at line 1:
ORA-28050: specified user or role cannot be dropped
SQL>

[SYS User 삭제]

SQL> show user
USER is "SYSTEM"
SQL>
SQL> drop user sys;
drop user sys
*
ERROR at line 1:
ORA-28050: specified user or role cannot be dropped
SQL> drop user sys CASCADE;
drop user sys CASCADE
*
ERROR at line 1:
ORA-28050: specified user or role cannot be dropped
SQL>

[결론]
안된다.
그러니 이상한 소리 하지 말아라..

현 상태 확인

[root@datarac1 bin]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@datarac1 bin]# pwd
/u01/app/19.0.0/grid/bin
[root@datarac1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      84516
         Available space (kbytes) :     407168
         ID                       : 1930868031
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded
[root@datarac1 bin]#
[root@datarac1 bin]# ./ocrcheck -details
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      84516
         Available space (kbytes) :     407168
         ID                       : 1930868031
         Device/File Name         : +OCRVOTE/datarac/OCRFILE/registry.255.1137239017
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded
[root@datarac1 bin]#
[root@datarac1 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   8e99de0aed5d4f45bff8818da83cea68 (/dev/oracleasm/asm-disk01) [OCRVOTE]
 2. ONLINE   1c4f3d989f684fbebf542f9bfb363318 (/dev/oracleasm/asm-disk02) [OCRVOTE]
 3. ONLINE   124978c179a24f20bf37f7db95407eed (/dev/oracleasm/asm-disk03) [OCRVOTE]
Located 3 voting disk(s).
[root@datarac1 bin]#

--OCR/VOTE가 사용하는 ASM Disk 확인
[oracle@datarac1 ~]$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@datarac1 ~]$ env | grep SID
ORACLE_SID=+ASM1
[oracle@datarac1 ~]$ sqlplus "/as sysdba"
SQL*Plus: Release 19.0.0.0.0 - Production on Tue May 23 09:21:38 2023
Version 19.19.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
SQL> set linesize 150
SQL> col name for a20
SQL> col state for a10
SQL> col path for a40
SQL> select name, group_number "g_no", disk_number "d_no", state, redundancy, path from v$asm_disk order by 6;
NAME                       g_no       d_no STATE      REDUNDA PATH
-------------------- ---------- ---------- ---------- ------- ----------------------------------------
OCRVOTE_0000                  2          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk01
OCRVOTE_0001                  2          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk02
OCRVOTE_0002                  2          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk03
DATA1_0000                    1          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk04
DATA1_0001                    1          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk05
DATA1_0002                    1          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk06
                              0          3 NORMAL     UNKNOWN /dev/oracleasm/asm-disk07
                              0          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk08
                              0          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk09
                              0          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk10
10 rows selected.
SQL>

OCR/VOTE용 ASM Disk Group에 ASM DISK 추가 삭제

--OCR/VOTE용 ASM Disk Group에 ASM DISK 추가
[oracle@datarac1 ~]$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@datarac1 ~]$ env | grep SID
ORACLE_SID=+ASM1
[oracle@datarac1 ~]$
[oracle@datarac1 ~]$
[oracle@datarac1 ~]$
[oracle@datarac1 ~]$ sqlplus "/as sysasm"
SQL*Plus: Release 19.0.0.0.0 - Production on Tue May 23 10:39:17 2023
Version 19.19.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
SQL> select * from v$asm_operation;
no rows selected
SQL> ALTER DISKGROUP OCRVOTE ADD DISK '/dev/oracleasm/asm-disk07' REBALANCE POWER 5;
Diskgroup altered.
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA PASS      STAT      POWER     ACTUAL      SOFAR   EST_WORK
------------ ----- --------- ---- ---------- ---------- ---------- ----------
  EST_RATE EST_MINUTES ERROR_CODE                                       CON_ID
---------- ----------- -------------------------------------------- ----------
           2 REBAL COMPACT   WAIT          5          5          0          0
         0           0                                                       0

           2 REBAL REBALANCE RUN           5          5         62         62
      4241           0                                                       0

           2 REBAL REBUILD   DONE          5          5          0          0
         0           0                                                       0


GROUP_NUMBER OPERA PASS      STAT      POWER     ACTUAL      SOFAR   EST_WORK
------------ ----- --------- ---- ---------- ---------- ---------- ----------
  EST_RATE EST_MINUTES ERROR_CODE                                       CON_ID
---------- ----------- -------------------------------------------- ----------
           2 REBAL RESYNC    DONE          5          5          0          0
         0           0                                                       0
SQL> select * from v$asm_operation;
no rows selected
SQL>
SQL> set linesize 150
SQL> col name for a20
SQL> col state for a10
SQL> col path for a40
SQL> select name, group_number "g_no", disk_number "d_no", state, redundancy, path from v$asm_disk order by 6;

NAME                       g_no       d_no STATE      REDUNDA PATH
-------------------- ---------- ---------- ---------- ------- ----------------------------------------
OCRVOTE_0000                  2          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk01
OCRVOTE_0001                  2          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk02
OCRVOTE_0002                  2          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk03
DATA1_0000                    1          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk04
DATA1_0001                    1          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk05
DATA1_0002                    1          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk06
OCRVOTE_0003                  2          3 NORMAL     UNKNOWN /dev/oracleasm/asm-disk07
                              0          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk08
                              0          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk09
                              0          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk10

10 rows selected.

SQL>exit
--OCR/VOTE 확인
[root@datarac1 bin]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@datarac1 bin]# pwd
/u01/app/19.0.0/grid/bin
[root@datarac1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      84516
         Available space (kbytes) :     407168
         ID                       : 1930868031
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded
[root@datarac1 bin]#
[root@datarac1 bin]#
[root@datarac1 bin]# ./ocrcheck -details
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      84516
         Available space (kbytes) :     407168
         ID                       : 1930868031
         Device/File Name         : +OCRVOTE/datarac/OCRFILE/registry.255.1137239017
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded
[root@datarac1 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   8e99de0aed5d4f45bff8818da83cea68 (/dev/oracleasm/asm-disk01) [OCRVOTE]
 2. ONLINE   1c4f3d989f684fbebf542f9bfb363318 (/dev/oracleasm/asm-disk02) [OCRVOTE]
 3. ONLINE   124978c179a24f20bf37f7db95407eed (/dev/oracleasm/asm-disk03) [OCRVOTE]
Located 3 voting disk(s).
[root@datarac1 bin]#
--OCR/VOTE용 ASM Disk Group에 ASM DISK 삭제
[oracle@datarac1 ~]$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@datarac1 ~]$ env | grep SID
ORACLE_SID=+ASM1
[oracle@datarac1 ~]$ sqlplus "/as sysasm"
SQL*Plus: Release 19.0.0.0.0 - Production on Tue May 23 10:46:20 2023
Version 19.19.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
SQL> set linesize 150
SQL> col name for a20
SQL> col state for a10
SQL> col path for a40
SQL> select name, group_number "g_no", disk_number "d_no", state, redundancy, path from v$asm_disk order by 6;
NAME                       g_no       d_no STATE      REDUNDA PATH
-------------------- ---------- ---------- ---------- ------- ----------------------------------------
OCRVOTE_0000                  2          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk01
OCRVOTE_0001                  2          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk02
OCRVOTE_0002                  2          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk03
DATA1_0000                    1          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk04
DATA1_0001                    1          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk05
DATA1_0002                    1          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk06
OCRVOTE_0003                  2          3 NORMAL     UNKNOWN /dev/oracleasm/asm-disk07
                              0          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk08
                              0          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk09
                              0          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk10

10 rows selected.
QL>
SQL> ALTER DISKGROUP OCRVOTE DROP DISK OCRVOTE_0000;
Diskgroup altered.
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA PASS      STATE           POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
------------ ----- --------- ---------- ---------- ---------- ---------- ---------- ---------- -----------
ERROR_CODE                                       CON_ID
-------------------------------------------- ----------
           2 REBAL COMPACT   WAIT                1          1          0          0          0           0
                                                      0

           2 REBAL REBALANCE RUN                 1          1         53         94       2148           0
                                                      0

           2 REBAL REBUILD   DONE                1          1          0          0          0           0
                                                      0


GROUP_NUMBER OPERA PASS      STATE           POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
------------ ----- --------- ---------- ---------- ---------- ---------- ---------- ---------- -----------
ERROR_CODE                                       CON_ID
-------------------------------------------- ----------
           2 REBAL RESYNC    DONE                1          1          0          0          0           0
                                                      0


SQL> select * from v$asm_operation;
no rows selected
SQL> select name, group_number "g_no", disk_number "d_no", state, redundancy, path from v$asm_disk order by 6;
NAME                       g_no       d_no STATE      REDUNDA PATH
-------------------- ---------- ---------- ---------- ------- ----------------------------------------
                              0          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk01
OCRVOTE_0001                  2          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk02
OCRVOTE_0002                  2          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk03
DATA1_0000                    1          0 NORMAL     UNKNOWN /dev/oracleasm/asm-disk04
DATA1_0001                    1          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk05
DATA1_0002                    1          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk06
OCRVOTE_0003                  2          3 NORMAL     UNKNOWN /dev/oracleasm/asm-disk07
                              0          2 NORMAL     UNKNOWN /dev/oracleasm/asm-disk08
                              0          1 NORMAL     UNKNOWN /dev/oracleasm/asm-disk09
                              0          3 NORMAL     UNKNOWN /dev/oracleasm/asm-disk10

10 rows selected.
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@datarac1 ~]$
--OCR/VOTE 확인
[root@datarac1 bin]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@datarac1 bin]# pwd
/u01/app/19.0.0/grid/bin
[root@datarac1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      84516
         Available space (kbytes) :     407168
         ID                       : 1930868031
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded
[root@datarac1 bin]# ./ocrcheck -details
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      84516
         Available space (kbytes) :     407168
         ID                       : 1930868031
         Device/File Name         : +OCRVOTE/datarac/OCRFILE/registry.255.1137239017
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded
[root@datarac1 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1c4f3d989f684fbebf542f9bfb363318 (/dev/oracleasm/asm-disk02) [OCRVOTE]
 2. ONLINE   124978c179a24f20bf37f7db95407eed (/dev/oracleasm/asm-disk03) [OCRVOTE]
 3. ONLINE   626f4793a1794fc9bfeec52aa161bbcf (/dev/oracleasm/asm-disk07) [OCRVOTE]
Located 3 voting disk(s).
[root@datarac1 bin]# exit

Oracle DB S/W addnode

[oracle@datarac1 addnode]$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@datarac1 addnode]$ env | grep SID
ORACLE_SID=DATARAC1
[oracle@datarac1 addnode]$ pwd
/u01/app/oracle/product/19.0.0/db_1/addnode
[oracle@datarac1 addnode]$ ls
addnode_oraparam.ini  addnode_oraparam.ini.sbs  addnode.pl  addnode.sh
[oracle@datarac1 addnode]$
[oracle@datarac1 addnode]$
[oracle@datarac1 addnode]$ ./addnode.sh -silent -ignoreSysPrereqs -ignorePrereqFailure \
> -waitForCompletion CLUSTER_NEW_NODES=datarac3
[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
   CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2023-05-22_05-51-48PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2023-05-22_05-51-48PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   7% Done.

Copy Files to Remote Nodes in progress.
..................................................   12% Done.
..................................................   18% Done.
..............................
Copy Files to Remote Nodes successful.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/addNodeActions2023-05-22_05-51-48PM.log

Instantiate files in progress.

Instantiate files successful.
..................................................   52% Done.

Saving cluster inventory in progress.
..................................................   89% Done.

Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/oracle/product/19.0.0/db_1 was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2023-05-22_05-51-48PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   96% Done.

As a root user, execute the following script(s):
        1. /u01/app/oracle/product/19.0.0/db_1/root.sh

Execute /u01/app/oracle/product/19.0.0/db_1/root.sh on the following nodes:
[datarac3]


Successfully Setup Software with warning(s).
..................................................   100% Done.
[oracle@datarac1 addnode]$

추가 Node에서 root.sh 실행

[root@datarac3 ~]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@datarac3 ~]#
[root@datarac3 ~]# /u01/app/oracle/product/19.0.0/db_1/root.sh
Check /u01/app/oracle/product/19.0.0/db_1/install/root_datarac3.localdomain_2023-05-22_18-00-18-763394494.log for the output of root script
[root@datarac3 ~]#

DB Instance 추가

[oracle@datarac1 ~]$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@datarac1 ~]$ env | grep SID
ORACLE_SID=DATARAC1
[oracle@datarac1 ~]$
[oracle@datarac1 ~]$ dbca -silent -addInstance -gdbName DATARAC -nodeName datarac3 -instanceName DATARAC3
Prepare for db operation
40% complete
Adding instance
48% complete
52% complete
56% complete
60% complete
64% complete
68% complete
72% complete
80% complete
Completing instance management.
86% complete
100% complete
Instance "DATARAC" added successfully on node "datarac3".
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/DATARAC/DATARAC.log" for further details.
[oracle@datarac1 ~]$

최종 상태 확인

[oracle@datarac1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       datarac1               STABLE
               ONLINE  ONLINE       datarac2               STABLE
               ONLINE  ONLINE       datarac3               STABLE
ora.chad
               ONLINE  ONLINE       datarac1               STABLE
               ONLINE  ONLINE       datarac2               STABLE
               ONLINE  ONLINE       datarac3               STABLE
ora.net1.network
               ONLINE  ONLINE       datarac1               STABLE
               ONLINE  ONLINE       datarac2               STABLE
               ONLINE  ONLINE       datarac3               STABLE
ora.ons
               ONLINE  ONLINE       datarac1               STABLE
               ONLINE  ONLINE       datarac2               STABLE
               ONLINE  ONLINE       datarac3               STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               STABLE
      2        ONLINE  ONLINE       datarac2               STABLE
      3        ONLINE  ONLINE       datarac3               STABLE
ora.DATA1.dg(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               STABLE
      2        ONLINE  ONLINE       datarac2               STABLE
      3        ONLINE  ONLINE       datarac3               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       datarac1               STABLE
ora.OCRVOTE.dg(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               STABLE
      2        ONLINE  ONLINE       datarac2               STABLE
      3        ONLINE  ONLINE       datarac3               STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               Started,STABLE
      2        ONLINE  ONLINE       datarac2               Started,STABLE
      3        ONLINE  ONLINE       datarac3               Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               STABLE
      2        ONLINE  ONLINE       datarac2               STABLE
      3        ONLINE  ONLINE       datarac3               STABLE
ora.cvu
      1        ONLINE  ONLINE       datarac1               STABLE
ora.datarac.db
      1        ONLINE  ONLINE       datarac1               Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       datarac2               Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             /db_1,STABLE
      3        ONLINE  ONLINE       datarac3               Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             /db_1,STABLE
ora.datarac1.vip
      1        ONLINE  ONLINE       datarac1               STABLE
ora.datarac2.vip
      1        ONLINE  ONLINE       datarac2               STABLE
ora.datarac3.vip
      1        ONLINE  ONLINE       datarac3               STABLE
ora.qosmserver
      1        ONLINE  ONLINE       datarac1               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       datarac1               STABLE
--------------------------------------------------------------------------------
[oracle@datarac1 ~]$

Oracle GI S/W addnode

[oracle@datarac1 addnode]$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@datarac1 addnode]$
[oracle@datarac1 addnode]$ env | grep SID
ORACLE_SID=+ASM1
[oracle@datarac1 addnode]$
[oracle@datarac1 addnode]$ ls
addnode_oraparam.ini  addnode_oraparam.ini.sbs  addnode.pl  addnode.sh
[oracle@datarac1 addnode]$
[oracle@datarac1 addnode]$ ./addnode.sh -silent -ignoreSysPrereqs -ignorePrereqFailure \
> -waitForCompletion CLUSTER_NEW_NODES=datarac3 CLUSTER_NEW_VIRTUAL_HOSTNAMES=datarac3-vip
[WARNING] [INS-40111] The specified Oracle Base location is not empty on following nodes: [datarac3].
   ACTION: Specify an empty location for Oracle Base.

[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
   CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2023-05-22_05-03-32PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2023-05-22_05-03-32PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.

Copy Files to Remote Nodes in progress.
..................................................   6% Done.
..................................................   11% Done.
....................
Copy Files to Remote Nodes successful.

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   21% Done.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/addNodeActions2023-05-22_05-03-32PM.log

Instantiate files in progress.

Instantiate files successful.
..................................................   49% Done.

Saving cluster inventory in progress.
..................................................   83% Done.

Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/19.0.0/grid was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2023-05-22_05-03-32PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   90% Done.

Update Inventory in progress.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/addNodeActions2023-05-22_05-03-32PM.log

Update Inventory successful.
..................................................   97% Done.

As a root user, execute the following script(s):
        1. /u01/app/oraInventory/orainstRoot.sh
        2. /u01/app/19.0.0/grid/root.sh

Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes:
[datarac3]
Execute /u01/app/19.0.0/grid/root.sh on the following nodes:
[datarac3]

The scripts can be executed in parallel on all the nodes.

Successfully Setup Software with warning(s).
..................................................   100% Done.
[oracle@datarac1 addnode]$

추가 Node에서 root.sh 실행

[root@datarac3 ~]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@datarac3 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to dba.
The execution of the script is complete.
[root@datarac3 ~]#
[root@datarac3 ~]#
[root@datarac3 ~]# /u01/app/19.0.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19.0.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/datarac3/crsconfig/rootcrs_datarac3_2023-05-22_05-36-54PM.log
2023/05/22 17:37:01 CLSRSC-594: Executing installation step 1 of 19: 'ValidateEnv'.
2023/05/22 17:37:01 CLSRSC-363: User ignored prerequisites during installation
2023/05/22 17:37:01 CLSRSC-594: Executing installation step 2 of 19: 'CheckFirstNode'.
2023/05/22 17:37:02 CLSRSC-594: Executing installation step 3 of 19: 'GenSiteGUIDs'.
2023/05/22 17:37:09 CLSRSC-594: Executing installation step 4 of 19: 'SetupOSD'.
Redirecting to /bin/systemctl restart rsyslog.service
2023/05/22 17:37:10 CLSRSC-594: Executing installation step 5 of 19: 'CheckCRSConfig'.
2023/05/22 17:37:13 CLSRSC-594: Executing installation step 6 of 19: 'SetupLocalGPNP'.
2023/05/22 17:37:16 CLSRSC-594: Executing installation step 7 of 19: 'CreateRootCert'.
2023/05/22 17:37:16 CLSRSC-594: Executing installation step 8 of 19: 'ConfigOLR'.
2023/05/22 17:37:32 CLSRSC-594: Executing installation step 9 of 19: 'ConfigCHMOS'.
2023/05/22 17:37:32 CLSRSC-594: Executing installation step 10 of 19: 'CreateOHASD'.
2023/05/22 17:37:35 CLSRSC-594: Executing installation step 11 of 19: 'ConfigOHASD'.
2023/05/22 17:37:48 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2023/05/22 17:38:19 CLSRSC-594: Executing installation step 12 of 19: 'SetupTFA'.
2023/05/22 17:38:19 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2023/05/22 17:38:20 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2023/05/22 17:39:12 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2023/05/22 17:39:14 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2023/05/22 17:39:26 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2023/05/22 17:41:34 CLSRSC-343: Successfully started Oracle Clusterware stack
2023/05/22 17:41:34 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
clscfg: EXISTING configuration version 19 detected.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2023/05/22 17:42:21 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2023/05/22 17:42:39 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@datarac3 ~]#

Node 추가 상태 확인

[oracle@datarac1 ~]$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@datarac1 ~]$ env | grep SID
ORACLE_SID=+ASM1
[oracle@datarac1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       datarac1               STABLE
               ONLINE  ONLINE       datarac2               STABLE
               ONLINE  ONLINE       datarac3               STABLE
ora.chad
               ONLINE  ONLINE       datarac1               STABLE
               ONLINE  ONLINE       datarac2               STABLE
               ONLINE  ONLINE       datarac3               STABLE
ora.net1.network
               ONLINE  ONLINE       datarac1               STABLE
               ONLINE  ONLINE       datarac2               STABLE
               ONLINE  ONLINE       datarac3               STABLE
ora.ons
               ONLINE  ONLINE       datarac1               STABLE
               ONLINE  ONLINE       datarac2               STABLE
               ONLINE  ONLINE       datarac3               STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               STABLE
      2        ONLINE  ONLINE       datarac2               STABLE
      3        ONLINE  ONLINE       datarac3               STABLE
ora.DATA1.dg(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               STABLE
      2        ONLINE  ONLINE       datarac2               STABLE
      3        ONLINE  ONLINE       datarac3               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       datarac1               STABLE
ora.OCRVOTE.dg(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               STABLE
      2        ONLINE  ONLINE       datarac2               STABLE
      3        ONLINE  ONLINE       datarac3               STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               Started,STABLE
      2        ONLINE  ONLINE       datarac2               Started,STABLE
      3        ONLINE  ONLINE       datarac3               Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       datarac1               STABLE
      2        ONLINE  ONLINE       datarac2               STABLE
      3        ONLINE  ONLINE       datarac3               STABLE
ora.cvu
      1        ONLINE  ONLINE       datarac1               STABLE
ora.datarac.db
      1        ONLINE  ONLINE       datarac1               Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       datarac2               Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             /db_1,STABLE
ora.datarac1.vip
      1        ONLINE  ONLINE       datarac1               STABLE
ora.datarac2.vip
      1        ONLINE  ONLINE       datarac2               STABLE
ora.datarac3.vip
      1        ONLINE  ONLINE       datarac3               STABLE
ora.qosmserver
      1        ONLINE  ONLINE       datarac1               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       datarac1               STABLE
--------------------------------------------------------------------------------
[oracle@datarac1 ~]$