1/20/2012
1
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Understanding Oracle RAC Internals Markus Michalewicz (
[email protected]) Senior Principal Product Manager Oracle RAC and Oracle RAC One Node 2
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
1
1/20/2012
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Agenda • The Oracle RAC Architecture • VIPs and Networks • Listeners and SCAN – and Services
• • • • 4
Client Connectivity Node Membership The Interconnect Installation and Upgrade Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
2
1/20/2012
The Oracle RAC Architecture Typical hardware architecture Network Client
Public Lan Node 1
Node 2
Node N-1
Node N
Interconnect with switch SAN switch
Shared Storage
5
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
The Oracle RAC Architecture Typical software deployment 11g Release 2
Oracle Real Application Clusters (RAC) Oracle Grid Infrastructure OS
OS
OS
OS
Shared Storage
6
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
3
1/20/2012
The Oracle RAC Architecture Typical software deployment 11g Release 2
Oracle Real Application Clusters (RAC) Oracle Automatic Storage Management (ASM) Oracle Clusterware OS
OS
OS
OS
Oracle ASM managed Shared Storage 7
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
The Oracle RAC Architecture Oracle RAC 11g Release 2 per node layer • Cloud File System (aka ASM Cluster File File System - ACFS) DB Instance
Oracle RAC ASM Instance
Oracle Grid Infrastructure OS
OS
HA Framework Node Membership OS
• For all other files • 11.2.0.3: For Backup Sets • Oracle Clusterware files • Voting Files (disks) • Cluster Registry (OCR) • Database Files • Redo / Archive Logs
Oracle ASM managed Shared Storage 8
• Control Files • etc.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
4
1/20/2012
The Oracle RAC Architecture Oracle Grid Infrastructure 11g Release 2 process overview
ASM Instance
Oracle Grid Infrastructure OS
OS
HA Framework
Node Membership • My Oracle Support (MOS) OS • Note 1053147.1 - 11gR2 Clusterware and Grid Home - What You Need to Know • Note 1050908.1 - How to Troubleshoot Grid Infrastructure Startup Issues 9
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
The Oracle RAC Architecture Local and Cluster Resources (11.2.0.2 list) LISTENER_SCAN1
LISTENER_SCAN2
LISTENER_SCAN3
ora.oc4j
If database is deployed:
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.SCAN3.VIP
ora.cvu
• ora.Dbname.db
ora.node1.vip
ora.node2.vip
ora.node3.vip
ora.node4.vip
ora.asm
ora.asm
ora.asm
ora.asm
ora.dgname.dg
ora.dgname.dg
ora.dgname.dg
ora.dgname.dg
ora.listener.lsnr
ora.listener.lsnr
ora.listener.lsnr
ora.listener.lsnr
ora.ons
ora.ons
ora.ons
ora.ons
ora.gsd
ora.gsd
ora.gsd
ora.gsd
ora.net1.network
ora.net1.network
ora.net1.network
ora.net1.network
ora.registry.acfs
ora.registry.acfs
ora.registry.acfs
ora.registry.acfs
• ora.Dbname.Srvcname.svc Command to list: • crsctl stat res -t
Oracle Grid Infrastructure OS
10
OS
OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
5
1/20/2012
VIPs and Networks
11
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
VIPs and Networks Oracle VIPs • SCAN and node VIPs are cluster resources • SCAN VIPs can float in the cluster rac1
• Node VIPs per default run on specific nodes • All VIPs on the public network (SCAN, node, user or application VIPs) depend on the Oracle network resource. • The network resource monitors the public network interface(s). • IOCTL based checks vs. default gateway ping • Default gateway checks as backup
rac2
LISTENER_SCAN1
LISTENER_SCAN2
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.rac1.vip
ora.rac2.vip
…
…
ora.listener.lsnr
ora.listener.lsnr
…
…
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
OS
• racgvip for backward compatibility only
12
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
6
1/20/2012
VIPs and Networks Node VIP Details
1 2 3
[GRID]> crsctl status res ora.rac1.vip -p NAME=ora.rac1.vip TYPE=ora.cluster_vip_net1.type ... AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX% ... rac1 CHECK_INTERVAL=1 … CHECK_TIMEOUT=30 … ... ora.rac1.vip START_DEPENDENCIES=hard(ora.net1.network) … pullup(ora.net1.network) ora.listener.lsnr ... STOP_DEPENDENCIES=hard(ora.net1.network) … ... ora.net1.network USR_ORA_VIP=rac1-vip VERSION=11.2.0.2.0 Oracle Grid
rac2 … … ora.rac2.vip
… ora.listener.lsnr … ora.net1.network
Infrastructure
OS
13
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
VIPs and Networks The network resource • The Oracle network resource monitors the public network interface(s) (e.g. eth0). • The default is network number 1 (net1) • There is one network resource per subnet:
rac2 …
[GRID]> srvctl config network
…
…
Network exists: 1/192.168.0.0/255.255.255.0/eth0, type static
ora.rac1.vip
ora.rac2.vip
…
…
ora.listener.lsnr
ora.listener.lsnr
…
…
ora.net1.network
ora.net1.network
• New in 11.2.0.2: • Direct management of the network resource • srvctl verb network • Network type support (static, dhcp, mixed)
14
rac1 …
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
7
1/20/2012
VIPs and Networks How to enable multiple networks in the cluster? 1
[GRID]> srvctl add network –h Adds a network configuration to the Oracle Clusterware. Usage: srvctl add network -k
network number (default number is 1)
-S //[if1[|if2...]] NET address spec for network -w
The network type (static, dhcp, mixed)
-h
Print usage
-v
Verbose output
…
…
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
…
…
Oracle Grid Infrastructure
• For example:
OS
srvctl add network -k 2 -S 10.1.1.0/255.255.255.0
15
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
VIPs and Networks How to enable multiple networks in the cluster? 2
[GRID]> srvctl add vip -h Adds a VIP to the Oracle Clusterware. Usage: srvctl add vip -n
Node name
-A //[if1[|if2...]] VIP address spec for node applications -k -h -v
network number (default number is 1)
ora.rac2.vip ora.rac2-vip2.vip
…
…
Print usage
ora.net1.network
ora.net1.network
Verbose output
ora.net2.network
ora.net2.network
…
…
• For example: srvctl add vip -n rac1 -k 2 -A rac1-vip2/255.255.255.0 srvctl add vip -n rac2 -k 2 -A rac2-vip2/255.255.255.0 16
ora.rac1.vip ora.rac1-vip2.vip
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
8
1/20/2012
VIPs and Networks Application / user and node VIPs 1
[GRID]> crsctl stat res ora.rac1-vip2.vip -p NAME=ora.rac1-vip2.vip TYPE=ora.cluster_vip_net2.type ... HOSTING_MEMBERS=rac1
This is a node vip!
... START_DEPENDENCIES=hard(ora.net2.network) pullup(ora.net2.network) ... STOP_DEPENDENCIES=hard(ora.net2.network) ...
ora.rac1.vip ora.rac1-vip2.vip …
…
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
…
…
Oracle Grid Infrastructure
USR_ORA_VIP=rac1-vip2
OS
VERSION=11.2.0.2.0
17
ora.rac2.vip ora.rac2-vip2.vip
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
VIPs and Networks Application / user and node VIPs 2
[GRID]> appvipcfg create -network=1 -ip 192.168.0.66 -vipname=myVIP -user=root
This is an application vip! [GRID]> appvipcfg create -network=2 -ip 10.1.1.88 -vipname=myNewVIP -user=root
ora.rac1.vip ora.rac1-vip2.vip myVIP
ora.rac2.vip ora.rac2-vip2.vip myNewVIP
…
…
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
…
…
Oracle Grid Infrastructure OS
18
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
9
1/20/2012
VIPs and Networks Application / user and node VIPs 3
[GRID]> crsctl stat res myVIP -p NAME=myVIP TYPE=app.appvip_net1.type ... HOSTING_MEMBERS=rac1 ...
This is an application vip!
ora.rac1.vip ora.rac1-vip2.vip myVIP
RESTART_ATTEMPTS=0
ora.rac2.vip ora.rac2-vip2.vip myNewVIP
...
…
…
USR_ORA_VIP=192.168.0.66
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
…
…
VERSION=11.2.0.2.0
Oracle Grid Infrastructure • You can modify it as needed!
19
OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Listeners and SCAN
20
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
10
1/20/2012
Listeners Listeners and dependencies • In Oracle RAC 11g Release 2 there have been three major changes regarding listeners: 1. 2. 3.
rac2 LISTENER_SCAN2
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.rac1.vip
ora.rac2.vip
Listeners are managed by an Oracle agent
…
…
SCAN Listeners were introduced
ora.listener.lsnr
ora.listener.lsnr
…
…
ora.net1.network
ora.net1.network
Listeners per default now run from the Oracle Grid Infrastructure (GRID) home
• A listener depends on a VIP • A listener depends indirectly on a network
21
rac1 LISTENER_SCAN1
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Listeners The default LISTENER [GRID]> srvctl config listener Name: LISTENER Network: 1, Owner: oracle Grid Software Owner Home:
rac1
End points: TCP:1521
rac2
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
22
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
11
1/20/2012
Listeners The default LISTENER FAQ • Can you define another port? - YES rac1
• Can you define another owner? – YES • Can you define another home? – YES, if really needed. • Can I add another listener on another port? - YES
rac2
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
23
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
The default LISTENER Can you define another port? - YES 1
[GRID]> srvctl modify listener -l LISTENER -p TCP:1545 [GRID]> srvctl config listener Name: LISTENER
rac1
Network: 1, Owner: oracle
rac2
Home: End points: TCP:1545
•
Check: /network/admin/endpoints_listener.ora before:
LISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip) (PORT=1521))(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.51)(PORT=1521)(IP=FIRST)))) # line added by Agent
24
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
12
1/20/2012
The default LISTENER Can you define another port? - YES 2
[GRID]> srvctl modify listener -l LISTENER -p TCP:1545 [GRID]> srvctl config listener Name: LISTENER
rac1
Network: 1, Owner: oracle
rac2
Home: End points: TCP:1545
•
Check: /network/admin/endpoints_listener.ora after restart:
LISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip) (PORT=1545))(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.51)(PORT=1545)(IP=FIRST)))) # line added by Agent
25
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
The default LISTENER Can you define another owner? - YES [GRID]>
srvctl modify listener -l LISTENER -u root
[GRID]>
srvctl config listener
You need to be root… Name: LISTENER
rac1
Network: 1, Owner: root For demonstration purposes only…
rac2
Home: End points: TCP:1545
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
26
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
13
1/20/2012
The default LISTENER Can you define another home? - YES [GRID]> srvctl modify listener -l LISTENER -o /u01/app/oracle/product/11.2.0/dbhome_1
[GRID]> srvctl config listener Name: LISTENER
rac1
rac2
Network: 1, Owner: oracle Home: /u01/app/oracle/product/11.2.0/dbhome_1 End points: TCP:1545
• This may not be required.
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure
• Instead, create a new listener from a separate home, if needed and use the SRVCTL command as shown, but try using the GI home.
27
OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Listeners Can I add another listener on another port? - YES [GRID]> srvctl config listener Name: LISTENER Network: 1, Owner: oracle Home:
rac1
End points: TCP:1521
rac2
Name: LISTENER2011 Network: 1, Owner: oracle Home: End points: TCP:2011
• MOS Note 220970.1 - RAC: Frequently Asked Questions •
How to use SCAN and node listeners with different ports? •
28
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.LISTENERK2.lsnr
ora.LISTENERK2.lsnr
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
OS
Add additional node listeners to the system using different ports
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
14
1/20/2012
Listeners Can I add another listener on another network? - YES [GRID]> srvctl add listener -l ListenerK2 -p 1544 -k 2 [GRID]> srvctl config listener Name: LISTENER Network: 1, Owner: oracle
rac1
Home: End points: TCP:1545
…
…
Name: LISTENERK2
ora.LISTENER.lsnr
ora.LISTENER.lsnr
Network: 2, Owner: oracle
Ora.LISTENERK2.lsnr
ora.LISTENERK2.lsnr
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
…
…
Home: End points: TCP:1544
• MOS Note 1063571.1 - How to Configure A Second Listener on a Separate Network in 11.2 Grid Infrastructure 29
rac2
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Listeners Remember – it’s just another listener … 1
[GRID]> srvctl config listener Name: JUSTALISTENER Network: 1, Owner: root Home:
rac2
LISTENER_SCAN1
LISTENER_SCAN2
...
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.rac1.vip
ora.rac2.vip
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
[GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora JUSTALISTENER=(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL=IPC)(KEY=JUSTALISTENER)))) # line added by Agent [GRID]> vi /u01/app/11.2.0/grid/network/admin/endpoints_listener.ora JUSTALISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)(PORT=1522)) (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.51)(PORT=1522)(IP=FIRST))))
30
rac1
End points: TCP:1522
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
15
1/20/2012
Listeners Remember – it’s just another listener … 2
[GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora JUSTALISTENER=(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL=IPC)(RATE_LIMIT=10) For demonstration purposes only… (KEY=JUSTALISTENER)))) # line added by Agent ...
rac1
[GRID]> srvctl stop listener -l JustAListener
LISTENER_SCAN2
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.rac1.vip
ora.rac2.vip
…
…
...
ora.LISTENER.lsnr
ora.LISTENER.lsnr
[GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora
…
…
ora.net1.network
ora.net1.network
[GRID]> srvctl start listener -l JustAListener [GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora
JUSTALISTENER=(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL=IPC)(RATE_LIMIT=10) (KEY=JUSTALISTENER)))) # line added by Agent
31
rac2
LISTENER_SCAN1
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
SCAN The basics and overview • In Oracle RAC 11g Release 2 SCAN Listeners were introduced •
SCAN
There is one SCAN setup per cluster
• Read about SCAN: • •
http://www.oracle.com/technetwork/database/ clustering/overview/scan-129069.pdf MOS Note 887522.1 – 11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained
• SCAN consists of a bundle containing:
32
1.
SCAN IP
2.
Associated (dependency) SCAN Listener
rac2
rac1
rac3
LISTENER_SCAN1
LISTENER_SCAN2
LISTENER_SCAN3
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.SCAN3.VIP
…
…
…
…
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
…
ora.net1.network
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
16
1/20/2012
SCAN The SCAN bundle 1 • 3 SCANs are the default for HA and LB SCAN
• Regardless of number of nodes • You can define less or more, if really needed. • SCAN listener resources run with an active dispersion dependency • If you use more nodes in the cluster than SCAN listeners are defined, no node should run more than one SCAN bundle at a time • If you use less nodes in the cluster than SCAN listeners are defined, there will be nodes running more than one SCAN bundle at a time.
• SCAN VIP moves with the listener, if possible.
33
rac2
rac1
rac3
LISTENER_SCAN1
LISTENER_SCAN2
LISTENER_SCAN3
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.SCAN3.VIP
…
…
…
…
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
…
ora.net1.network
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
SCAN The SCAN bundle 2
[GRID]> srvctl config scan SCAN name: cluster1, Network: 1/192.168.0.0/255.255.255.0/eth0 SCAN VIP name: scan1, IP: /cluster1.us.oracle.com/192.168.0.41 rac1
rac2
[GRID]> srvctl modify scan
LISTENER_SCAN1
…
Modifies the SCAN name.
ora.SCAN1.VIP
…
…
…
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Usage: srvctl modify scan -n
Domain name qualified SCAN name
-h
Print usage
• http://www.oracle.com/technetwork/database/ clustering/overview/scan-129069.pdf page 2
34
SCAN
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
17
1/20/2012
SCAN The SCAN bundle 3
[GRID]> srvctl add scan -h Adds a SCAN VIP to the Oracle Clusterware. Usage: srvctl add scan -n -n
Domain name qualified SCAN name
-k
network number (default number is 1)
SCAN rac1
rac2
LISTENER_SCAN1
…
-S //[if1[|if2...]] NET address spec for network
ora.SCAN1.VIP
…
…
…
-h
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Print usage
• Note: SCAN can only operate on one network / in one subnet.
Oracle Grid Infrastructure OS
35
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
SCAN The SCAN FAQ 1 • From MOS Note 220970.1 –
RAC: Frequently Asked Questions
•
How to use SCAN and node listeners with different ports?
SCAN
• With Oracle RAC 11g Release 2 using SCAN is the default. • As with other listeners, there is no direct communication between the node (listeners) and the SCAN listeners. • Listeners are only aware of the instances and services served, since the instances (PMON) register themselves and the services they host with the listeners. • The instances use the LOCAL and REMOTE Listener parameters to know with which listeners to register. • Listeners used for a client connection to Oracle RAC should be managed by Oracle Clusterware and should be listening on an Oracle managed VIP. 36
rac1
rac2
LISTENER_SCAN1
…
ora.SCAN1.VIP
…
…
…
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
18
1/20/2012
SCAN The SCAN FAQ 2 • Can you define another port? - YES • See MOS Note 220970.1 – RAC: Frequently Asked Questions •
How to use SCAN and node listeners with different ports?
•
Use srvctl modify scan_listener –p
• Can you define another owner? – NO • Can you define another home? – NO • Can you have more than one node listener with SCAN? – YES • Can the SCAN and the node listener ports differ? - YES • See also MOS Note 972500.1 – How to Modify SCAN Setting or SCAN Listener Port after Installation
37
SCAN rac1
rac2
LISTENER_SCAN1
…
ora.SCAN1.VIP
…
…
…
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
ora.net1.network
ora.net1.network
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Listeners and Instance Registration Listener changes may require instance changes • See MOS Note 220970.1 – RAC: Frequently Asked Questions •
How to use SCAN and node listeners with different ports?
• Whenever you change an instance parameter configuration a change to the LOCAL_LISTENER or REMOTE_LISTENER per database (instance) may be required. • Note that using server-side TNSANMES aliases to resolve the listener configuration is not recommended. • The use of the TNSNAMES connector string is the default: local_listener = ‘(DESCRIPTION= (ADDRESS_LIST= (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.61)(PORT=2011)) ))'
38
SCAN rac1
rac2
DB Instance
DB Instance
LISTENER_SCAN1
…
ora.SCAN1.VIP
…
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
19
1/20/2012
Listeners and Instance Registration EXAMPLE 1 • Taken from MOS Note 220970.1 – RAC: FAQ •
How to use SCAN and node listeners with different ports? •
Add additional node listeners to the system using different ports •
srvctl add listener -l LISTENER2011 -p 2011
•
srvctl start listener -l LISTENER2011
•
srvctl status listener -l LISTENER2011
•
lsnrctl status LISTENER2011
SCAN rac1
rac2
DB Instance
DB Instance
LISTENER_SCAN1
…
ora.SCAN1.VIP
…
…
…
[GRID]> srvctl config listener
ora.LISTENER.lsnr
ora.LISTENER.lsnr
Name: LISTENER2011
…
…
Network: 1, Owner: oracle
Oracle Grid Infrastructure
Home:
OS
OS
End points: TCP:2011 39
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Listeners and Instance Registration EXAMPLE 2 • Taken from MOS Note 220970.1 – RAC: FAQ •
How to use SCAN and node listeners with different ports? •
Add additional node listeners to the system using different ports
SCAN rac1
rac2
• New node listener does not serve any database instance • You must alter the LOCAL_LISTENER parameters for each instance accordingly:
DB Instance
DB Instance
LISTENER_SCAN1
…
ora.SCAN1.VIP
…
… … alter system set local_listener =‘ (DESCRIPTION= (ADDRESS_LIST= ora.LISTENER.lsnr ora.LISTENER.lsnr (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.61)(PORT=2011)))) … … ' scope=BOTH SID='OCRL1'; alter system register; Oracle Grid Infrastructure
OS
40
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
20
1/20/2012
Listeners and Instance Registration EXAMPLE 3 • Taken from MOS Note 220970.1 – RAC: FAQ •
How to use SCAN and node listeners with different ports? •
Add additional node listeners to the system using different ports
SCAN rac1
rac2
• Note: Use spaces between the various segments of the command • Reason: the database agent in Oracle Clusterware currently determines whether the local_listener or remote_listener have been manually set by a string comparison operation. • If the string looks unaltered it will reset these parameters with the default values that it determines on instance startup. • Slight modifications are therefore required. ER 11772838 has been filed to allow for a more convenient mechanism.
41
DB Instance
DB Instance
LISTENER_SCAN1
…
ora.SCAN1.VIP
…
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Listeners and Instance Registration Using listeners in different subnets / networks • SCAN can only operate in one network / subnet • If listeners in different subnets are used, • use LISTENER_NETWORKS: http://download.oracle.com/docs/cd/E11882_01 /server.112/e17110/initparams115.htm#REFRN10313 •
Note: Listeners specified by the LISTENER_NETWORKS parameter should not be used in the LOCAL_LISTENER and REMOTE_LISTENER parameters. Otherwise, cross registration will happen and connections will be redirected cross networks.
• For more information, see also MOS Note 1063571.1 – How to Configure A Second Listener on a Separate Network in 11.2 Grid Infrastructure 42
rac1
rac2
DB Instance
DB Instance
LISTENER_SCAN1
…
ora.SCAN1.VIP
…
…
…
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
Ora.LISTENERK2.lsnr
ora.LISTENERK2.lsnr
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
…
…
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
21
1/20/2012
Listeners, SCAN and Services
43
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Listeners and Service Registration A summary • Services depend on a database • Per default, the database instance will register all services it hosts with all listeners it registers with.
SCAN rac1
• Services operate within the following boundaries: • Network (indirectly via dependency) • Server Pools (in policy-managed database environments) • Currently there is no support for service failover
• Between Server Pools • Between networks
44
rac2
DB Instance
DB Instance
ora.Dbname.db
ora.Dbname.db
ora.Dbname.Srvcname.svc
ora.Dbname.Srvcname.svc
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
Oracle Grid Infrastructure OS
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
22
1/20/2012
Services and Dependencies Dependency on DB and Server Pool 1
[GRID]> srvctl config service -d ORCL -s MyService Service name: MyService Service is enabled
SCAN
Server pool: ORCL rac1
Cardinality: 1 Disconnect: false Service role: PRIMARY Management policy: AUTOMATIC ... Preferred instances: ORCL_1 Available instances:
rac2
DB Instance
DB Instance
ora.ORCL.db
ora.ORCL.db
ora.ORCL.MyService.svc
ora.Dbname.Srvcname.svc
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
Oracle Grid Infrastructure OS
45
OS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Services and Dependencies Start / Stop dependencies 2
[GRID]> crsctl stat res ora.orcl.myservice.svc -p ... DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=service) PROPERTY(SERVICE_NAME=%GEN_SERVICE_NAME%) PROPERTY(DB_UNIQUE_NAME=CONCAT(PARSE(%NAME%, ., 2), STAT(ora.orcl.db, USR_ORA_DOMAIN), .)) ELEMENT(INSTANCE_NAME=STAT(ora.orcl.db, GEN_USR_ORA_INST_NAME)) ... START_DEPENDENCIES=hard(ora.orcl.db,type:ora.cluster _vip_net1.type) weak(type:ora.listener.type) pullup(type:ora.cluster_vip_net1.type) pullup:always(ora.orcl.db) dispersion(type:ora.service.type) STOP_DEPENDENCIES=hard(intermediate:ora.orcl.db,inte rmediate:type:ora.cluster_vip_net1.type)
SCAN rac1
rac2
DB Instance
DB Instance
ora.ORCL.db
ora.Dbname.db
ora.ORCL.MyService.svc
ora.Dbname.Srvcname.svc
…
…
ora.LISTENER.lsnr
ora.LISTENER.lsnr
…
…
Oracle Grid Infrastructure OS
OS
... 46
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
23
1/20/2012
Client Connectivity
47
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Client Connectivity Direct or indirect connect • Connect Time Load Balancing (CTLB) • Connect Time Connection Failover (CTCF) • Runtime Connection Load Balancing (RTLB) • Runtime Connection Failover (RTCF) BATCH Production
SCAN
Email
48
Connection Pool
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
24
1/20/2012
Client Connectivity Connect Time Connection Failover jdbc:oracle:thin:@MySCAN:1521/Email PMRAC = (DESCRIPTION = (FAILOVER=ON) (ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)))
Email
49
MySCAN
BATCH Production
Connection Pool
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Client Connectivity Runtime Time Connection Failover PMRAC = (DESCRIPTION = (FAILOVER=ON) (ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email) ...))
Email
50
MySCAN
BATCH Production
Connection Pool
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
25
1/20/2012
Client Connectivity Runtime Time Connection Failover PMRAC = (DESCRIPTION = (FAILOVER=ON) (ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email) (FAILOVER_MODE= (TYPE=select)(METHOD=basic)(RETRIES=180)(DELAY=5))))
BATCH Production
51
MySCAN
Email
?
Connection Pool
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Client Connectivity More information • If problems occur, see: • Note 975457.1 – How to Troubleshoot Connectivity Issues with 11gR2 SCAN Name • For more advanced configurations, see: • Note 1306927.1 – Using the TNS_ADMIN variable and changing the default port
number of all Listeners in an 11.2 RAC for an 11.2, 11.1, and 10.2 Database
Email
52
?? ? MySCAN
BATCH Production
Connection Pool
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
26
1/20/2012
Client Connectivity Two ways to protect the client 1.
Transparent Application Failover (TAF) • • • • •
2.
Fast Application Notification (FAN) •
Tries to make the client unaware of a failure Provides means of CTCF and RTCF Allows for pure ‘selects’ (reads) to continue Write transactions need to be re-issued The Application needs to be TAF aware
• • •
FAN wants to inform clients ASAP Client can react to failure asap Expects clients to re-connect on failure (FCF) Sends messages about changes in the cluster
Email
53
MySCAN
BATCH Production
?
Connection Pool
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Client Connectivity and Service Definition Define settings on the server • HA (and LB) settings
[GRID]> srvctl config service -d ORCL -s MyService
can be defined per service
Service name: MyService
• Clients connecting to the service will
...
adhere to the settings considering the client used.
DTP transaction: false AQ HA notifications: false Failover type: NONE
BATCH Production
MySCAN
Failover method: NONE Email
TAF failover retries: 0 TAF failover delay: 0 Connection Load Balancing Goal: LONG Runtime Load Balancing Goal: NONE TAF policy specification: BASIC
54
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
27
1/20/2012
Client Connectivity Use a FAN aware connection pool 1•
If a connection pool is used • •
•
The clients (users) get a physical connection to the connection pool The connection pool creates a physical connection to the database • It is a direct client to the database
Internally the pool maintains logical connections
BATCH Production
Connection Pool
MySCAN
Email
55
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Client Connectivity Use a FAN aware connection pool 2•
The connection pool • • •
•
Invalidates connections to one instance Re-establishes new logical connections • May create new physical connections Prevent new clients to be misrouted
The application needs to handle the transaction failure that might have occurred. BATCH Production
Connection Pool
MySCAN
Email
56
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
28
1/20/2012
Client Connectivity The Load Balancing (LB) cases • Connect Time Load Balancing (CTLB) • Runtime Connection Load Balancing (RTLB) • On the Client Side • On the Server Side Email
57
MySCAN
BATCH Production
Connection Pool
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Client Connectivity Connect Time Load Balancing (CTLB) – on the client side PMRAC = (DESCRIPTION = (FAILOVER=ON)(LOAD_BALANCE=ON) (ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)))
Email
58
MySCAN
BATCH Production
Connection Pool
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
29
1/20/2012
Client Connectivity Connect Time Load Balancing (CTLB) – on the server side • Traditionally, PMON dynamically registers the services to the specified listeners with: • •
Service names for each running instance of the database and instance names for the DB The listener is updated with the load information for every instance and node as follows: • 1-Minute OS Node Load Average all 30 secs. • Number of Connections to Each Instance •
Number of Connections to Each Dispatcher BATCH Production
MySCAN
Email
Connection Pool
? 59
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Client Connectivity Use FAN for the Load Balancing cases • Connect Time Load Balancing (CTLB) • Connect Time Connection Failover (CTCF) • Runtime Connection Load Balancing (RTLB) • Runtime Connection Failover (RTCF) 30% connections
RAC Database
I’m busy Instance1 10% connections I’m very busy I’m idle
60% connections
60
Instance2
Instance3
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
30
1/20/2012
Client Connectivity Use FAN for the Load Balancing cases • Connect Time Load Balancing (CTLB) • Runtime Connection Load Balancing (RTLB) • Also via AQ (Advanced Queuing) based notifications • Background is always the Load Balancing Advisory
RAC Database 30% connections
I’m busy Instance1
MySCAN
10% connections
• For more information, see: • Oracle® Real Application Clusters Administration and Deployment Guide 11g Release 2: 5 Introduction to Automatic Workload Management 61
I’m very busy I’m idle
Instance2
60% connections Instance3
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Node Membership
62
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
31
1/20/2012
Oracle Clusterware Architecture Node Membership Processes and Basics Public Lan
Public Lan
Private Lan / Interconnect
Main processes involved: • CSSD (ora.cssd)
Oracle Clusterware
• CSSDMONITOR • was: oprocd
• now: ora.cssdmonitor
CSSD
CSSD
SAN Network
63
CSSD
Voting Disk
SAN Network
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle Clusterware Architecture What does CSSD do? • Monitors nodes using 2 communication channels: –
Private Interconnect Network Heartbeat
–
Voting Disk based communication Disk Heartbeat
• Evicts (forcibly removes nodes from a
cluster) nodes dependent on heartbeat feedback (failures)
Oracle Clusterware
CSSD
“Ping”
CSSD
“Ping” 64
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
32
1/20/2012
Oracle Clusterware Architecture Interconnect basics – network heartbeat • Each node in the cluster is “pinged” every second • Nodes must respond in css_misscount time (defaults to 30 secs.) –
Reducing the css_misscount time is generally not supported
• Network heartbeat failures
“Ping”
will lead to node evictions –
CSSD-log: [date / time] [CSSD][1111902528] clssnmPollingThread: node mynodename (5) at 75% heartbeat fatal, removal in 6.770 seconds
65
CSSD
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle Clusterware Architecture Voting Disk basics – disk heartbeat 1 • Each node in the cluster “pings” (r/w) the Voting Disk(s) every second • Nodes must receive a response in (long / short) diskTimeout time –
IF I/O errors indicate clear accessibility problems timeout is irrelevant
• Disk heartbeat failures
will lead to node evictions –
CSSD-log: … [CSSD] [1115699552] >TRACE: clssnmReadDskHeartbeat: node(2) is down. rcfg(1) wrtcnt(1) LATS(63436584) Disk lastSeqNo(1)
CSSD
CSSD
“Ping” 66
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
33
1/20/2012
Oracle Clusterware Architecture Voting Disk basics – Structure 2 • Voting Disks contain dynamic and static data: –
Dynamic data: disk heartbeat logging
–
Static data: information about the nodes in the cluster
• With 11.2.0.1 Voting Disks got an “identity”: –
E.g. Voting Disk serial number: [GRID]> crsctl query css votedisk 1.
2 1212f9d6e85c4ff7bf80cc9e3f533cc1 (/dev/sdd5) [DATA]
Node information
Disk Heartbeat Logging
• Voting Disks must therefore not be copied using “dd” or “cp” anymore 67
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle Clusterware Architecture Voting Disk basics – Simple Majority rule 3 • Oracle supports redundant Voting Disks for disk failure protection • “Simple Majority Rule” applies: – Each node must “see” the simple majority of configured Voting Disks
at all times in order not to be evicted (to remain in the cluster) trunc(n/2+1) with n=number of voting disks
configured and n>=1 CSSD
CSSD
“Ping”
68
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
34
1/20/2012
Oracle Clusterware Architecture Simple Majority rule – in extended clusters •
http://www.oracle.com/goto/rac – Using standard NFS to support
a third voting file for extended cluster configurations (PDF) CSSD
CSSD
• Same principles apply • Voting Disks are just
geographically dispersed 69
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle Clusterware Architecture Voting Disks in Oracle ASM does not change their usage [GRID]> crsctl query css votedisk 1.
2 1212f9d6e85c4ff7bf80cc9e3f533cc1 (/dev/sdd5) [DATA]
2.
2 aafab95f9ef84f03bf6e26adc2a3b0e8 (/dev/sde5) [DATA]
3.
2 28dd4128f4a74f73bf8653dabd88c737 (/dev/sdd6) [DATA]
Located 3 voting disk(s).
• Oracle ASM auto creates 1/3/5 Voting Files – Voting Disks reside in one diskgroup only – Based on Ext/Normal/High redundancy and on Failure Groups in the Disk Group – Per default there is one failure group per disk – ASM will enforce the required number of disks – New failure group type: Quorum Failgroup
70
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
35
1/20/2012
Oracle Clusterware Architecture Oracle Cluster Registry (OCR) placement in Oracle ASM •
The OCR is managed like a datafile in ASM (new type) • •
71
It adheres completely to the redundancy settings for the diskgroup (DG) There can be more than one OCR location in more than one DG (DG:OCR 1:1) • Recommendation is 2 OCR locations, 1 in DATA, 1 in FRA for example
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle Clusterware Architecture Backup of Clusteware files is fully automatic (11.2+) • Clusterware Files (managed in ASM) enables fully Automatic Backups: • The Voting Disks are backed up into the OCR • Any configuration change in the cluster (e.g. node addition) triggers a new backup of the Voting Files. • A single, failed Voting Disks is restored by ASM automatically within a Disk Group – no action required • Note: Do not use DD to back up the Voting Disks anymore! • The OCR is backed up automatically every 4 hours • Manual Backups can be taken as required • ONLY IF all Voting Disks are corrupted or failed AND (all copies of) the OCR are also corrupted or unavailable THEN manual interference would be required – the rest is automatic.
72
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
36
1/20/2012
Fencing Basics Why are nodes evicted? • Evicting (fencing) nodes is a preventive measure (it’s a good thing)! • Nodes are evicted to prevent consequences of a split brain: –
Shared data must not be written by independently operating nodes
–
The easiest way to prevent this is to forcibly remove a node from the cluster
1
CSSD
73
2
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Fencing Basics How are nodes evicted? – STONITH •
•
Once it is determined that a node needs to be evicted, –
A “kill request” is sent to the respective node(s)
–
Using all (remaining) communication channels
A node (CSSD) is requested to “kill itself” “STONITH like” –
“STONITH” foresees that a remote node kills the node to be evicted
1
CSSD
74
2
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
37
1/20/2012
Fencing Basics EXAMPLE: Network heartbeat failure •
The network heartbeat between nodes has failed –
It is determined which nodes can still talk to each other
–
A “kill request” is sent to the node(s) to be evicted
Using all (remaining) communication channels Voting Disk(s) A node is requested to “kill itself”; executer: typically CSSD
1
2
CSSD
CSSD 2
75
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Fencing Basics What happens, if CSSD is stuck? •
A node is requested to “kill itself”
•
BUT CSSD is “stuck” or “sick” (does not execute) – e.g.: –
CSSD failed for some reason
–
CSSD is not scheduled within a certain margin
• See also: MOS note 1050693.1 Troubleshooting 11.2 Clusterware Node Evictions (Reboots)
OCSSDMONITOR (was: oprocd) will take over and execute
1
CSSD
2 CSSDmonitor CSSD
2
CSSD 76
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
38
1/20/2012
Fencing Basics How can nodes be evicted? • Oracle Clusterware 11.2.0.1 and later supports IPMI (optional) – Intelligent Platform Management Interface (IPMI) drivers required • IPMI allows remote-shutdown of nodes using additional hardware – A Baseboard Management Controller (BMC) per cluster node is required
1
CSSD
77
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Fencing Basics EXAMPLE: IPMI based eviction on heartbeat failure • The network heartbeat between the nodes has failed – It is determined which nodes can still talk to each other – IPMI is used to remotely shutdown the node to be evicted
1
CSSD
78
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
39
1/20/2012
Fencing Basics Which node gets evicted? • Voting Disks and heartbeat communication is used to determine the node • In a 2 node cluster, the node with the lowest node number should survive • In a n-node cluster, the biggest sub-cluster should survive (votes based)
1
2
CSSD
79
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Fencing Basics Cluster members can escalate a kill request • Cluster members (e.g Oracle RAC instances) can request
Oracle Clusterware to kill a specific member of the cluster • Oracle Clusterware will then attempt to kill the requested member
Oracle RAC DB Inst. 1
Oracle RAC DB Inst. 2
Oracle Clusterware
Inst. 1: kill inst. 2 80
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
40
1/20/2012
Fencing Basics Cluster members can escalate a kill request • Oracle Clusterware will then attempt to kill the requested member • If the requested member kill is unsuccessful, a node eviction escalation can be issued,
which leads to the eviction of the node, on which the particular member currently resides
Oracle RAC DB Inst. 1
Oracle RAC DB Inst. 2
Oracle Clusterware
Inst. 1: kill inst. 2 81
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Fencing Basics Cluster members can escalate a kill request • Oracle Clusterware will then attempt to kill the requested member • If the requested member kill is unsuccessful, a node eviction escalation can be issued,
which leads to the eviction of the node, on which the particular member currently resides
Oracle RAC DB Inst. 1
Oracle RAC DB Inst. 2
Oracle Clusterware
Inst. 1: kill inst. 2 82
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
41
1/20/2012
Fencing Basics Cluster members can escalate a kill request • Oracle Clusterware will then attempt to kill the requested member • If the requested member kill is unsuccessful, a node eviction escalation can be issued,
which leads to the eviction of the node, on which the particular member currently resides
Oracle RAC DB Inst. 1 Oracle Clusterware
83
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Re-Bootless Node Fencing With 11.2.0.2 onwards, fencing may not mean re-boot • Until Oracle Clusterware 11.2.0.2, fencing meant “re-boot” • With Oracle Clusterware 11.2.0.2, re-boots will be seen less, because:
– Re-boots affect applications that might run an a node, but are not protected – Customer requirement: prevent a reboot, just stop the cluster – implemented...
App X RAC DB Inst. 1
CSSD
84
App Y RAC DB Inst. 2
Oracle Clusterware
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
42
1/20/2012
Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less:
– Instead of fast re-booting the node, a graceful shutdown of the stack is attempted • It starts with a failure – e.g. network heartbeat or interconnect failure
App X RAC DB Inst. 2
RAC DB Inst. 1
CSSD
85
App Y
Oracle Clusterware
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less:
– Instead of fast re-booting the node, a graceful shutdown of the stack is attempted
• Then IO issuing processes are killed; it is made sure that no IO process remains – For a RAC DB mainly the log writer and the database writer are of concern App X
App Y
RAC DB Inst. 1
CSSD
86
Oracle Clusterware
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
43
1/20/2012
Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less:
– Instead of fast re-booting the node, a graceful shutdown of the stack is attempted
• Once all IO issuing processes are killed, remaining processes are stopped – IF the check for a successful kill of the IO processes, fails → reboot App X
App Y
RAC DB Inst. 1
CSSD
87
Oracle Clusterware
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less:
– Instead of fast re-booting the node, a graceful shutdown of the stack is attempted
• Once all remaining processes are stopped, the stack stops itself with a “restart flag”
App X
App Y
RAC DB Inst. 1 Oracle Clusterware CSSD
88
OHASD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
44
1/20/2012
Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less:
– Instead of fast re-booting the node, a graceful shutdown of the stack is attempted
• OHASD will finally attempt to restart the stack after the graceful shutdown
App X
App Y
RAC DB Inst. 1
OHASD
Oracle Clusterware CSSD
89
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Re-Bootless Node Fencing EXCEPTIONS • With Oracle Clusterware 11.2.0.2, re-boots will be seen less, unless…:
– – – –
IF the check for a successful kill of the IO processes fails → reboot IF CSSD gets killed during the operation → reboot IF cssdmonitor (oprocd replacement) is not scheduled → reboot IF the stack cannot be shutdown in “short_disk_timeout”-seconds → reboot App X RAC DB Inst. 1
CSSD
90
App Y RAC DB Inst. 2
Oracle Clusterware
CSSD
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
45
1/20/2012
The Interconnect
91
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
The Interconnect Heartbeat and “memory channel” between instances Network Public Lan Node 1
Node 2
Node N-1
Node N
Client
Interconnect with switch SAN switch
92
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
46
1/20/2012
The Interconnect Redundant Interconnect Usage 1 • Redundant Interconnect Usage can be used as a bonding alternative – It works for “private networks” only; the nodeVIPs use a different approach – It enables HA and Load Balancing for up to 4 NICs per server (on Linux / Unix) – It can be used by Oracle Databases 11.2.0.2 and Oracle Clusterware 11.2.0.2 – It uses so called HAIPs that are assigned to the private networks on the server – The HAIPs will be used by the database and ASM instances and processes
Node 1
93
Node 2
HAIP1
HAIP3
HAIP2
HAIP4
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
The Interconnect Redundant Interconnect Usage 2 • A multiple listening endpoint approach is used – The HAIPs are taken from the “link-local” (Linux / Unix) IP range (169.254.0.0) – To find the communication partners, multicasting on the interconnect is required – With 11.2.0.3 Broadcast is a fallback alternative (BUG 10411721) – Multicasting is still required on the public lan for MDNS for example. – Details in My Oracle Support (MOS) Note with Doc ID 1212703.1: 11.2.0.2 Grid Infrastructure Install or Upgrade may fail due to Multicasting Node 1
94
Node 2
HAIP1
HAIP3
HAIP2
HAIP4
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
47
1/20/2012
The Interconnect Redundant Interconnect Usage and the HAIPs • If a network interface fails, the assigned HAIP is failed over to a remaining one. • Redundant Interconnect Usage allows having networks in different subnet • •
You can either have one subnet for all networks or a different one for each You can also use VLANs with the interconnect. For more information see: •
Note 1210883.1 - 11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip
•
Note 220970.1 - RAC: Frequently Asked Questions - How to use VLANs in Oracle RAC? AND Are there any issues for the interconnect when sharing the same switch as the public network by using VLAN to separate the network?
Node 1
HAIP1
95
Node 2
HAIP1
HAIP3
HAIP2
HAIP4
HAIP3
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Installation and Upgrade
96
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
48
1/20/2012
Installation Patch Sets are also more than BUG fixe with 11.2.0.x • Upgrade + New Features • Upgrade + Full Installation • And (always) “Out-of-place” • Patch uses new Oracle Home
• First patch set: 11.2.0.2 • 11.2.0.3 works the same • Less new features
97
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Installation Installation is meant to be simpler with 11g Release 2 • Oracle Universal Installer (OUI) is the main installation instrument. • Installation tips and tricks in: • MOS note 810394.1 (generic) • See: RAC Platform Specific Starter Kits and Best Practices
•
Step by Step for various versions available.
• See also: RAC Platform Generic Load Testing and System Test Plan Outline
• For an Oracle VM based deployment see:
98
•
Oracle RAC OVM Templates on OTN
•
Details in MOS note 1185244.1 – Pre-Built Grid Infrastructure and Oracle RAC Templates For Oracle VM
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
49
1/20/2012
Upgrade and Update Upgrade to 11g Release 2 (11.2.0.3) recommended • The Upgrade to 11g Release 2 from previous versions is generally recommended • Things to consider: •
MOS note 1312225.1 - Things to Consider Before Upgrading to 11.2.0.2 Grid Infrastructure
•
MOS note 1320966.1 - Things to Consider Before Upgrade to 11.2.0.2 Database Performance
•
MOS note 1363369.1 – Things to Consider Before Upgrading to 11.2.0.3 Grid Infrastructure/ASM
99
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
10 0
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
50
