Monday, October 5, 2020

19c Failed Upgrade restart Process

19c Failed Upgrade restart Process 


Step 1 . Set the env to Oracle 19C

. ./dbenv.csh_19

Env | grep ORA

Check catupgrd.sql

cd $ORACLE_HOME/rdbms/admin

ls -l catupgrd.sql


Step 2 . Startup database in Upgrade mode .

Sqlplus / as sysdba

Startup upgrade;

Exit


Step 3. cd $ORACLE_HOME/rdbms/admin

$ORACLE_HOME/perl/bin/perl catctl.pl -n 4 -R -l $ORACLE_HOME/cfgtoollogs catupgrd.sql


After this Dbs will go down some time so depends upon situation you have to action.

So to start this upgraded database in 19c please ensure that 19c env is  set, properly.

 

Step 4 .

Run utlusts.sql, the Post-Upgrade Status Tool, which provides a summary of the status of the upgrade in the spool log.

You can run utlusts.sql any time before or after you complete the upgrade, but not during the upgrade.


SQL> @$ORACLE_HOME/rdbms/admin/utlusts.sql


Step 5 .


Check for invalid objects , if they exists run $ORACLE_HOME/rdbms/admin/utlrp.sql


Step 6 .


Now go back to 12c env and complete the post upgrade activity ----> No need


Note . Please note the DST sometimes is not upgraded from 26 to 32 when we follow the above process . You need to manually upgrade the DST in case it is not upgrade from 26 to 32.


Steps to check and upgrade DST-


Step 1 . Check the DST version


SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value FROM DATABASE_PROPERTIES WHERE PROPERTY_NAME LIKE 'DST_%' ORDER BY PROPERTY_NAME;


PROPERTY_NAME              VALUE

----------------------     ---------------------

DST_PRIMARY_TT_VERSION     26

DST_SECONDARY_TT_VERSION   0

DST_UPGRADE_STATE          None


Oracle 19c should have DST as 32 , and if you see 26 , that means it has not been upgraded. This will also be reported in post check script output.


Check the latest timezone


SELECT DBMS_DST.GET_LATEST_TIMEZONE_VERSION FROM DUAL;


GET_LATEST_TIMEZONE_VERSION

--------------------------------------------

32


Step 2 . Upgrade of DST TMZ is done in 2 part ( prepare and upgrade window)


Prepare .

During the prepare window, you can run any of the following scripts present in the $ORACLE_HOME/rdbms/admin directory to check how much data will need to be updated in the database during the time zone data upgrade:

.       utltz_countstats.sql

.       This script shows the optimizer statistics of num_rows of all the tables having TIMESTAMP WITH TIME ZONE (TSTZ) data.

.       Note:

.       Run the utltz_countstats.sql script only when the database optimizer statistics are up to date, else run the utltz_countstar.sql script. If you run the utltz_countstats.sql script, then you need not run the utltz_countstar.sql script.


Step 3 .

Upgrade Window

During the upgrade window, you can run the following scripts present in the $ORACLE_HOME/rdbms/admin directory to upgrade the time zone data in the database:

1.      Run the utltz_upg_check.sql script from the $ORACLE_HOME directory:

2.      spool utltz_upg_check.log

3.      @utltz_upg_check.sql

spool off

The following information is displayed on the screen after successful execution of the script:

CopyINFO: A newer RDBMS DST version than the one currently used is found.

INFO: Note that NO DST update was yet done.

INFO: Now run utltz_upg_apply.sql to do the actual RDBMS DST update.

INFO: Note that the utltz_upg_apply.sql script will

INFO: restart the database 2 times WITHOUT any confirmation or prompt.

The script also writes the following information in the alert.log file:

utltz_upg_check sucessfully found newer RDBMS DSTv new_time_zone_version and took number_of_minutes minutes to run.

If the utltz_upg_check.sql script displays the following error, check the previous message displayed on the screen and proceed accordingly.

ORA-20xxx: Stopping script - see previous message...

4.      Run the utltz_upg_apply.sql script from the $ORACLE_HOME directory after the utltz_upg_check.sql script is executed successfully:

5.      spool utltz_upg_apply.log

6.      @utltz_upg_apply.sql

spool off

Note:

The following are the prerequisites for running the utltz_upg_apply.sql script:

.       No application should query or insert time zone data in the database during the time zone upgrade process.

.       In an RAC environment, the RAC database must be started as a single database instance.

.       In a multitenant environment, all the PDBs must be shut down before running the utltz_upg_apply.sql script on the CDB.

Also, note the following:

.       The utltz_upg_apply.sql script automatically restarts the database multiple times during its execution.

.       The utltz_upg_apply.sql script generally takes less time to execute than the utltz_upg_check.sql script.

The following information is displayed on the screen after successful execution of the utltz_upg_apply.sql script:

INFO: The RDBMS DST update is successfully finished.

INFO: Make sure to exit this sqlplus session.

INFO: Do not use it for timezone related selects.

The TZ_VERSION column in the Registry$database table now gets updated with the new time zone version.

If the script displays the following error message, then check the previous message displayed on the screen and proceed accordingly.

ORA-20xxx: Stopping script - see previous message...


Thursday, October 1, 2020

SCAN Series PART :- 2

         HOW  SCAN VIP  & SCAN Listener are WORK together ???

 

Let see the SCAN-VIP information.

[grid@rac1 ~]$ srvctl config scan
SCAN name: mycluster-scan, Network: 1
Subnet IPv4: 192.168.56.0/255.255.255.0/enp0s3, static
Subnet IPv6:
SCAN 0 IPv4 VIP: 192.168.56.13
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 1 IPv4 VIP: 192.168.56.15
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 2 IPv4 VIP: 192.168.56.14
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
[grid@rac1 ~]$


[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_vip.type"
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac1

NAME=ora.scan2.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.scan3.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2

Here SCAN-VIP is up and running. 


Now lets try to stop  SCAN_VIP resources.

[grid@rac1 ~]$ srvctl stop scan
PRCR-1065 : Failed to stop resource ora.scan1.vip
CRS-2529: Unable to act on 'ora.scan1.vip' because that would require stopping or relocating 'ora.LISTENER_SCAN1.lsnr', but the force option was not specified
PRCR-1065 : Failed to stop resource ora.scan2.vip
CRS-2529: Unable to act on 'ora.scan2.vip' because that would require stopping or relocating 'ora.LISTENER_SCAN2.lsnr', but the force option was not specified
PRCR-1065 : Failed to stop resource ora.scan3.vip
CRS-2529: Unable to act on 'ora.scan3.vip' because that would require stopping or relocating 'ora.LISTENER_SCAN3.lsnr', but the force option was not specified

Not able to stop. 

Lets use force option .

[grid@rac1 ~]$ srvctl stop scan -f
[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_vip.type"
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
TARGET=OFFLINE====> offline 
STATE=OFFLINE====> offline 

NAME=ora.scan2.vip
TYPE=ora.scan_vip.type
TARGET=OFFLINE====> offline 
STATE=OFFLINE====> offline 

NAME=ora.scan3.vip
TYPE=ora.scan_vip.type
TARGET=OFFLINE    ====> offline 
STATE=OFFLINE ====> offline 


[grid@rac1 ~]$ ps -ef|grep -i scan
grid     21700 16262  0 09:37 pts/0    00:00:00 grep --color=auto -i scan


[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_listener.type"
NAME=ora.LISTENER_SCAN1.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=OFFLINE    ====> offline 

NAME=ora.LISTENER_SCAN2.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=OFFLINE    ====> offline 

NAME=ora.LISTENER_SCAN3.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=OFFLINE    ====> offline 


Look the scan-vip & Scan-listener also getting down in pair in cluster. and its indicate its working in Pair.

Lets Do another try.

First down the SCAN LISTENER :======>

lets verify the status of SCAN Listeners and VIP Listeners

[grid@rac1 ~]$ srvctl start scan
[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_vip.type"
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac1

NAME=ora.scan2.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.scan3.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac1

[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_listener.type"
NAME=ora.LISTENER_SCAN1.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=ONLINE on rac1

NAME=ora.LISTENER_SCAN2.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.LISTENER_SCAN3.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=ONLINE on rac1

[grid@rac1 ~]$ ps -ef|grep -i scan
grid     26330     1  0 09:49 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid     26403     1  0 09:49 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid     26618 16262  0 09:50 pts/0    00:00:00 grep --color=auto -i scan
[grid@rac1 ~]$



Let down the SCAN Listeners:-

[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_listener.type"
NAME=ora.LISTENER_SCAN1.lsnr
TYPE=ora.scan_listener.type
TARGET=OFFLINE
STATE=OFFLINE=====> its showing down

NAME=ora.LISTENER_SCAN2.lsnr
TYPE=ora.scan_listener.type
TARGET=OFFLINE
STATE=OFFLINE=====> its showing down

NAME=ora.LISTENER_SCAN3.lsnr
TYPE=ora.scan_listener.type
TARGET=OFFLINE
STATE=OFFLINE=====> its showing down

[grid@rac1 ~]$ ps -ef|grep -i scan
grid     27693 16262  0 09:51 pts/0    00:00:00 grep --color=auto -i scan
[grid@rac1 ~]$

But SCAN-VIP is up and running so SCAN-VIP resources  is parent process 
and SCAN-LISTENERS process is child process which depend upon SCAN_VIP process.

[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_vip.type"
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac1

NAME=ora.scan2.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.scan3.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac1

[grid@rac1 ~]$


[grid@rac1 ~]$ srvctl start scan_listener
[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_vip.type"
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac1

NAME=ora.scan2.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.scan3.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac1

[grid@rac1 ~]$ ps -ef|grep -i scan
grid     29759     1  0 09:56 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid     29846     1  0 09:56 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit


Now Let See how the SCAN-VIP resources and  SCAN_LISTENERS Resource  are move to another node in Cluster ??

On node1 : Two Scan Listeners are running 
[grid@rac1 ~]$ ps -ef|grep -i scan
grid     29759     1  0 09:56 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid     29846     1  0 09:56 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid     31263 16262  0 10:01 pts/0    00:00:00 grep --color=auto -i scan
[grid@rac1 ~]$


On node2 : One Scan Listeners is running 
[oracle@rac2 ~]$ ps -ef|grep -i scan
grid     28111     1  0 00:59 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
oracle   31379 30882  0 01:04 pts/0    00:00:00 grep --color=auto -i scan
[oracle@rac2 ~]$

Lets shut down the node 

grid     31263 16262  0 10:01 pts/0    00:00:00 grep --color=auto -i scan
[grid@rac1 ~]$ su - root
Password:
Last login: Wed Sep 30 08:05:05 IST 2020 on pts/0

[root@rac1 ~]#
[root@rac1 ~]#
[root@rac1 ~]# init 0

Now Check the SCAN_LISTENER Status on Node2

[grid@rac2 ~]$ ps -ef|grep -i scan
grid     28111     1  0 00:59 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
grid     32266     1  0 01:07 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid     32280     1  0 01:07 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid     32434 32294  0 01:07 pts/0    00:00:00 grep --color=auto -i scan
[grid@rac2 ~]$

here we can see three scan listeners are running ..... Good

Lets see scan-vip resources status :-

[grid@rac2 ~]$  crsctl stat res -w "TYPE = ora.scan_vip.type"
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2  =====on Node2

NAME=ora.scan2.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2 ====on Node2

NAME=ora.scan3.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2====on Node2

[grid@rac2 ~]$


[grid@rac2 ~]$ crsctl stat res -w "TYPE = ora.scan_listener.type"
NAME=ora.LISTENER_SCAN1.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.LISTENER_SCAN2.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.LISTENER_SCAN3.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=ONLINE on rac2

[grid@rac2 ~]$


 ****This indicate how the scan-vip and scan-listeners work together******

IF SCAN VIP & SCAN LISETNERS are DOWN still User can connect ????

Lets try :- 


[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_listener.type"
NAME=ora.LISTENER_SCAN1.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=OFFLINE

NAME=ora.LISTENER_SCAN2.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=OFFLINE

NAME=ora.LISTENER_SCAN3.lsnr
TYPE=ora.scan_listener.type
TARGET=ONLINE
STATE=OFFLINE

[grid@rac1 ~]$ crsctl stat res -w "TYPE = ora.scan_vip.type"
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.scan2.vip
TYPE=ora.scan_vip.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.scan3.vip
TYPE=ora.scan_vip.type
TARGET=OFFLINE
STATE=OFFLINE

[grid@rac1 ~]$


[oracle@rac1 admin]$ tnsping TEST.databasexpert.com

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 30-SEP-2020 03:39:14

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:
/oracle/DEA/12102/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = mycluster-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TEST.databasexpert.com)))


^C ===========Taking Huge time so i have to cancel.

[oracle@rac1 admin]$ tnsping mycluster-scan

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 30-SEP-2020 03:39:34

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:
/oracle/DEA/12102/network/admin/sqlnet.ora

Used HOSTNAME adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.14)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.15)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.13)(PORT=1521)))
^C ======> Same again , i have to cancel
[oracle@rac1 admin]$



[oracle@rac1 admin]$ sqlplus scott/tiger@TEST.databasexpert.com

SQL*Plus: Release 12.1.0.2.0 Production on Wed Sep 30 03:42:29 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

ERROR:
ORA-12543: TNS:destination host unreachable  ======> Same issue 


Enter user-name:



So its Indicate the SCAN VIP & SCAN Listener plays very critical role for connection . if they down no connection will happened.


Oracle Grid also gave the different option to shut down the scan-listener.


Lets Check status of Scan_listeners.

 

[grid@rac2 ~]$ crsctl stat res -w "TYPE = ora.scan_vip.type"
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.scan2.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.scan3.vip
TYPE=ora.scan_vip.type
TARGET=ONLINE
STATE=ONLINE on rac2

[grid@rac2 ~]$ ps -ef|grep -i scan
grid     10332  8593  0 01:30 pts/1    00:00:00 grep --color=auto -i scan
grid     28111     1  0 00:59 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
grid     32266     1  0 01:07 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid     32280     1  0 01:07 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit



11gR2 Syntax

As grid user stop scan resources

[grid@rac2 admin]$ srvctl stop scan -i 1 -f

[grid@rac2 ~]$ srvctl stop scan -i 1 -f =======>
[grid@rac2 ~]$  ps -ef|grep -i scan
grid     10992  8593  0 01:32 pts/1    00:00:00 grep --color=auto -i scan
grid     28111     1  0 00:59 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
grid     32280     1  0 01:07 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
[grid@rac2 ~]$

LISTENER_SCAN1 is down successfully.


lets use 12CR1 & R2 Syntax:- 

As grid user stop scan resources

[grid@rac2 ~]$ srvctl stop scan -scannumber 2 -f   =======>
[grid@rac2 ~]$ ps -ef|grep -i scan
grid     11425  8593  0 01:33 pts/1    00:00:00 grep --color=auto -i scan
grid     32280     1  0 01:07 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
[grid@rac2 ~]$


Now Lest start stop Scan Listener individually per node.

Lets check status of scan_listener:====>

[grid@rac2 ~]$ ps -ef|grep -i scan
grid     12500  8593  0 01:35 pts/1    00:00:00 grep --color=auto -i scan
grid     32280     1  0 01:07 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
[grid@rac2 ~]$


11gR2 Syntax  ===> 

As grid user stop scan resources

[grid@rac2 ~]$ srvctl start scan -i 1


[grid@rac2 ~]$  ps -ef|grep -i scan
grid     12802     1  0 01:35 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid     12832  8593  0 01:36 pts/1    00:00:00 grep --color=auto -i scan
grid     32280     1  0 01:07 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
[grid@rac2 ~]$



12CR1 &R2 syntax =====>

As grid user stop scan resources

[grid@rac2 ~]$ srvctl start scan -scannumber 2


[grid@rac2 ~]$  ps -ef|grep -i scan
grid     12802     1  0 01:35 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid     13272     1  0 01:37 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
grid     13319  8593  0 01:37 pts/1    00:00:00 grep --color=auto -i scan
grid     32280     1  0 01:07 ?        00:00:00 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
[grid@rac2 ~]$



           All Three Scan_Listener resources are up and running.



                                    *******EOD********

SCAN Series PART :- 1

 

Introduction to SCAN & SCAN Listener.


There  are different ways to connect the  Database in cluster Configuration.


[oracle@rac1]$ sqlplus scott/tiger@rac2-  vip.databasexpert.com:1521/TEST.databasexpert.com 

===> Here Instead of SCAN IP ,use VIP IP 

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 29 19:11:43 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Last Successful login time: Tue Sep 29 2020 00:07:39 -05:00

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Advanced Analytics and Real Application Testing options

SQL>exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
[oracle@rac1 admin]$


Again Here use the VIP IP ==============>

sqlplus scott/tiger@rac2-vip.databasexpert.com:1521/TEST.databasexpert.com

Here We are able to connect to the Database with this VIP address. 

But Here we can use this straight forward way  to connect to the database  But What happen if we have more than thousands application 

connection  then What happened ?????


Even you can use this  IP to connect =================================>

sqlplus scott/tiger@192.168.56.15:1521/TEST.databasexpert.com

sqlplus scott/tiger@192.168.56.13:1521/TEST.databasexpert.com

sqlplus scott/tiger@192.168.56.14:1521/TEST.databasexpert.com


But still its useful?  to keep address of the each IP in mind????

or other ways ========>

test.databasexpert.com

(DESCRIPTION= (ADDRESS_LIST= (ADDRESS=(PROTOCOL=tcp)

(LOAD_BALANCE=YES)

(FAILOVER=YES)

(HOST=rac1-vip.databasexpert.com)(PORT=1521))  =====> VIP Address 

(ADDRESS=(PROTOCOL=tcp)

(HOST=rac2-vip.databasexpert.com)(PORT=1521))) =======> VIP Address

 (CONNECT_DATA=(SERVICE_NAME=TEST.databasexpert.com)))=====> Service Name


Clients can continue to access the cluster in the same way as with previous releases, but Oracle recommends that clients must access  cluster using SCAN. EZCONNECT method can be used to access the cluster database.

In 11g R1 ,10g and earlier oracle version  preceding Connect String or connect method used  to connect to application  &  if we try to used this  connect string in new version then it will work  but its has  some drawbacks.

*****************************************

VIP Listeners =======> 

1) If we add a new node to the Cluster then ====> Then we have to update preceding connect string.

and share with multiple application users . if  this tns entry used by thousands users ?? then what happened ??? more work right  & more co-ordination.

2) If node is removed  from the topology then what happened ========>

Again More work & Co-ordination with thousands of users. 

3) If  one of the node in cluster is shut down for prolonged period of time =======>  

  Then connect string would try connect to all VIP listeners , and this would take time if that first VIP Listeners itself is down  and then after some time it would move to available VIP-Listener, this Process would  take some time (brief TCP timeouts)   and this  will induce delay in application connection .

4) The VIP Listeners is not enough  smart to do proper load balance of the session request in the Cluster. for eg .In Some case one node has more than 100 connection and one node has less than 50 connection .

5) VIP listeners session connection management is not centrally  & optimal in cluster.


To Resolve this Issue or to overcome above drawbacks , oracle from 11g Release 2  introduced the new Feature SCAN Listeners.

SCAN Listeners  =====

SCAN Listeners resolve the connect time issue & Improve the load balancing accuracy.

SCAN Listeners know about the cluster topology changes . This changes are instantly propagated  to scan listeners.  

SCAN Listener is responsible to accept the connection and forward this connection to least loaded instance(node).

SCAN Listeners always use the service metrics which is shared by cluster DB process to identify the Least loaded Node in cluster. Each scan listener keeps updated cluster load statistics

SCAN Listeners is lightweight process and simply redirected the connection.

SCAN Listeners Can managed spurious Connection request without any delay.

Application always used scan listeners for connection. 

Below connect string is use the SCAN Listeners Feature.


TEST.databasexpert.com =

  (DESCRIPTION =

    (ADDRESS = (PROTOCOL = TCP)(HOST = mycluster-scan)(PORT = 1521))

    (CONNECT_DATA =

      (SERVER = DEDICATED)

      (SERVICE_NAME = TEST.databasexpert.com)

    )

  )

mycluster-scan =======> 

This connect string use DNS alias (SCAN NAME)  and it has the three IP address  (SCAN IP) and its resolve to three IP address, 

What is DNS? ====================>

Domain Name Server handle name resolution (IP to NAME & vice Versa).Generally Its Created BY network Admin  Team.

In the DNS create single Name to resolve the three IP address using Round Robin Method.

Using DNS Configuration  we used & defined SCAN name & its IP in Cluster.

This Three IP address don't assign to network Interface.

The IP addresses must be on the same subnet as your default public network in the cluster. 

Name must be mycluster Not mycluster.expert.com

Name must be resolved with Mycluster or mycluster.expert.com

During Cluster ware installation cluster process will take care of this three IP.

Snippets from DNS server.

rac1     IN      A       192.168.56.10

rac1            IN      A       192.168.56.10

rac2            IN      A       192.168.56.20

rac1-vip        IN      A       192.168.56.11

rac2-vip        IN      A       192.168.56.12

mycluster-scan  IN      A       192.168.56.13 ====> Single Name assign to resolve three IP address.

                         IN      A       192.168.56.14

                        IN      A       192.168.56.15


[root@rac1 named]# nslookup mycluster-scan

Server:         192.168.56.10

Address:        192.168.56.10#53

Name:   mycluster-scan.databasexpert.com

Address: 192.168.56.14

Name:   mycluster-scan.databasexpert.com

Address: 192.168.56.15

Name:   mycluster-scan.databasexpert.com

Address: 192.168.56.13

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@rac1 named]# nslookup mycluster-scan.databasexpert.com

Server:         192.168.56.10

Address:        192.168.56.10#53

Name:   mycluster-scan.databasexpert.com

Address: 192.168.56.13

Name:   mycluster-scan.databasexpert.com

Address: 192.168.56.14

Name:   mycluster-scan.databasexpert.com

Address: 192.168.56.13

Note:-

Please always Enable the round robin on DNS level to resolve the IP address.

If no round robin setup then please contact Network Administrator.

Its balancing the connection load in the cluster.

If no Round robin Setup then =================>

Now days Client typically handle the failover connection. 

Older version not supporting such client side failover.

minimum version of the client should be 11g release 2 or higher. 

Still failover of connection request to another SCAN LISETNER , in case first Listener in list is down.

If client side DNS round robin setup is disabled. Oracle Call Interface (OCI) based database access drivers will apply an internal round-robin.

Do not configure SCAN VIP addresses in the hosts file. But if you use the hosts file to resolve SCAN name, you can have only one SCAN IP address.

If hosts file is used to resolve SCAN hostname, you will receive Cluster Verification Utility failure at end of installation


Now how  to configure ******SCAN & SCAN Listeners  *************

SCAN======>

Single Client Access Name (SCAN) is a  New feature used in Oracle Real Application Clusters.

SCAN is  a single name for clients to access any Oracle Database running in a cluster.

You can think of SCAN as a alias for cluster to access databases

We can called SCAN as Domain Name which is register with one and up to three IP address. And to make SCAN successful this IP or one IP should be resolved.

To used this Feature we have to Use DNS or GNS. Most of the time DNS has been used by most of DBA.SCAN provides a single name for clients to access any Oracle Database running in a cluster.

The benefit is that the client’s connect information does not need to change if you add or remove nodes or databases in the cluster.

EZconnet:-  sqlplus scott/tiger@mycluster-scan:1521/sale

JDBC connect jdbc:oracle:thin:@mycluster-scan:1521/sale

Having a single name to access the cluster  or alias of cluster to connect to a database in this cluster. 

Allows clients to use EZConnect and the simple JDBC thin URL to access any database running in the cluster, independently of the number of databases or servers running in the cluster and regardless on which server(s) in the cluster the requested database is actually active. 

The VIPs and SCAN VIPs must be on the same subnet as the public interface. 

The default gateway must be on the same subnet as the VIPs (including SCAN VIPs) to prevent VIP start/stop/failover issues. 

With 11gR2 this is detected and reported by the OUI, if the check is ignored this will result in the failure to start the VIPs resulting in failure of the installation itself.

Number of SCAN VIP’s you notice it will be same as the Number of SCAN LISTENERS in cluster environment

This Three IP address don't assign to network Interface.

The IP addresses must be on the same subnet as your default public network in the cluster. 

SCAN Name must be mycluster Not mycluster.expert.com

Name must be resolved with Mycluster or mycluster.expert.com

During Cluster ware installation cluster process will take care of this three IP.


How , When SCAN Configured ? ???????? And How SCAN LISTENERS created ??????

During oracle Grid Infrastructure installation  ====> Consist of Oracle Cluster ware and Oracle Automatic Storage Management 

The SCAN configuration is defined by default & Its Mandatory.

We have to provide the SCAN -NAME. And this SCAN name must be defined using DNS OR GNS.

If the SCAN name resolves to one  or up to  three IP address ====> 

root script (root.sh or rootupgrade.sh) will create the  ========>

SCAN VIP resources(ora.scan1.vip,ora.scan2.vip,ora.scan3.vip)

corresponding SCAN listener resource(ora.LISTENER_SCAN1.lsnr,2,3)

This depends upon how many IP address SCAN name resolve. (up to three).

If its resolve two IP address then its create two scan listeners resource & two scan vip resources.

During Cluster Configuration two resources were created  


SCAN Configuration With Grid Infrastructure.========>


For Each three IP address which is resolved by Single Name (SCAN). ========>

192.168.56.13=====>

1)SCAN VIP resource 

2)SCAN Listener resources


192.168.56.14======>

 1)SCAN VIP resource 

2)SCAN Listener resources

 

192.168.56.15======>

1)SCAN VIP resource 

2)SCAN Listener resources.


This SCAN LISETNER dependent on SCAN VIP. So this SCAN LISETNER and SCAN VIP Dispersed across the Cluster.

its means this scan resource pair started across the clustered assuming the cluster consists of three or more nodes


In case, a 2-node-cluster is used ======>

one server in the cluster will host two sets of SCAN Listener & scan vip  resources. & other Node host one set of SCAN Resources.

[grid@rac2 ~]$ ps -ef| grep SCAN

grid     21545     1  0 Sep28 ?        00:00:06 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit

[root@rac1 named]# ps -ef|grep -i scan

grid     25556     1  0 Sep29 ?        00:00:06 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit

grid     25571     1  0 Sep29 ?        00:00:05 /u002/app/oracle/product/12.1.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit


In three Node Cluster :=====> 

If the node on which a SCAN VIP is running fails, the SCAN VIP and its associated listener will fail over to another

node in the cluster. If by means of such a failure the number of available servers in the cluster becomes

less than three, one server would again host two sets of SCAN resources. 

If a node becomes available in the cluster again,dispersion will take effect and relocate one set accordingly. 


To see the SCAN VIP Resource Information =============> srvctl config scan

[grid@rac2 ~]$ srvctl config scan

SCAN name: mycluster-scan, Network: 1 =====================> SCAN NAME 

Subnet IPv4: 192.168.56.0/255.255.255.0/enp0s3, static

Subnet IPv6:

SCAN 0 IPv4 VIP: 192.168.56.13   =====================> SCAN IP

=======

SCAN VIP is enabled.

SCAN VIP is individually enabled on nodes:

SCAN VIP is individually disabled on nodes:

SCAN 1 IPv4 VIP: 192.168.56.15   =====================> SCAN IP

=======

SCAN VIP is enabled.

SCAN VIP is individually enabled on nodes:

SCAN VIP is individually disabled on nodes:

SCAN 2 IPv4 VIP: 192.168.56.14   =====================> SCAN IP

=======

SCAN VIP is enabled.

SCAN VIP is individually enabled on nodes:

SCAN VIP is individually disabled on nodes:


To see the SCAN  Listeners Resource Information ========>

[grid@rac2 ~]$  srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521 

Registration invited nodes:

Registration invited subnets:

SCAN Listener is enabled.

SCAN Listener is individually enabled on nodes:

SCAN Listener is individually disabled on nodes:

SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521

Registration invited nodes:

Registration invited subnets:

SCAN Listener is enabled.

SCAN Listener is individually enabled on nodes:

SCAN Listener is individually disabled on nodes:

SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

Registration invited nodes:

Registration invited subnets:

SCAN Listener is enabled.

SCAN Listener is individually enabled on nodes:

SCAN Listener is individually disabled on nodes:


SCAN is by default restricted to only accept service registration from nodes in the cluster.

this is only work of SCAN listener is assigned by the cluster.

SCAN and Oracle Cluster ware managed VIPs now support IPv6 based IP addresses

SCAN supports multiple subnets in the cluster (one SCAN per subnet) 


 Always keep in mind when ever you had issue with SCAN LISETNER please take a look on following Component.

SCAN VIP

SCAN LISTENER

NODE VIP

Grid Infrastructure ====> start SCAN listener LISTENER_SCAN1,2,3  to listen on SCAN VIP(s);

Grid Infrastructure ====>  default set remote_listener to SCAN listener.

GRID_HOME ====>  local listener + SCAN listener run by default 

Grid Infrastructure ====>  start local listener LISTENER on all nodes to listen on local VIP. 



Oracle Database Configuration Using SCAN ================>

In  RAC DB this below two parameter play very important role while accessing the local and remote instance.


Local_listener 

Remote_listener.


Local_listener    ==============>

In Database Configuration  LOCAL_LISTENER play critical role, its use the node-vip IP address.

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

local_listener                       string       (ADDRESS=(PROTOCOL=TCP)(HOST=

                                                 192.168.56.12)(PORT=1521))

SQL>

cat /etc/hosts | grep -i vip 

192.168.56.11             rac1-vip.databasexpert.com                     rac1-vip

192.168.56.12             rac2-vip.databasexpert.com                     rac2-vip

  

Remote_listener   ===========>

Ever since Oracle Database 11g Release 2, SCAN is an essential part of the Oracle RAC database

configuration and therefore the REMOTE_LISTENER parameter is set to the SCAN per default,

assuming that the database is created using standard Oracle tools (e.g. the formerly mentioned DBCA).This allows the instances to register with the SCAN Listeners as remote listeners to

provide information on what services are being provided by the instance, the current load and a

recommendation on how many incoming connections should be directed to the instance.

[oracle@rac2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 29 14:10:50 202

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Advanced Analytics and Real Application Testing options

SQL> show parameter remote_lis

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

remote_listener                      string       mycluster-scan:1521

SQL>

[oracle@rac1 ~]$ tnsping mycluster-scan

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 29-SEP-2020 17:19:43

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:

/oracle/REA/12102/network/admin/sqlnet.ora

Used HOSTNAME adapter to resolve the alias

Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.15)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.13)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.14)(PORT=1521)))

OK (0 msec)

[oracle@rac1 ~]$ tnsping mycluster-scan

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 29-SEP-2020 17:19:44

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:

/oracle/REA/12102/network/admin/sqlnet.ora

Used HOSTNAME adapter to resolve the alias

Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.13)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.15)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.14)(PORT=1521)))

OK (10 msec)


Client Load Balancing using SCAN   =======>

1) User Connected to Database in cluster using SQLNET. tnsname.ora file.

TEST.databasexpert.com =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = mycluster-scan)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TEST.databasexpert.com)
    )
  )

2)Then mycluster-scan name is resolved through DNS server and user get list of three IP Address by applying Round robin algorithm.
try connecting through one of the IPs received.

3)If the user receives an error, it will try the other addresses before returning an error to the user or application.

4)When SCAN Listener receives a connection request, the SCAN Listener will check for the least loaded instance providing the requested service.

5)  SCAN listener replies with the address of the local listener on the least-loaded node .It will then re-direct the connection request to the local listener on the node where the least loaded instance is running.


6)Then Local listener will create the process & user get connect to the database in cluster.


                       ******************EOD*********************