Adding Node 11gRAC

                                                                                                          Adding Node to RAC  

ADDING NODE TO 11g RAC

NAME PLATFORM VERSION DATE

: : : :

SURESH.V ORACLE ENTERPRISE LINUX 5.x 64bit 11.1.0.6.0 16.09.2010

1

ORAFACT

                                                                                                          Adding Node to RAC  

Table of Contents Adding Node to 11g RAC...............................................................................................................................3 1.1 Configuring OS and hardware for new node.......................................................................................3 1.2 Adding Oracle Clusterware to the new node.......................................................................................7 1.3 Configuring ONS for the new node...................................................................................................10 1.4 Adding RAC home to the new node..................................................................................................10 1.5 Adding a listener to the new node......................................................................................................13 1.6 Adding a ASM instance to the new node...........................................................................................18 1.7 Adding a database instance to the new node......................................................................................20

2

ORAFACT

                                                                                                          Adding Node to RAC  

Adding Node to 11g RAC 1.1 Configuring OS and hardware for new node Server Configuration Name : rac3.orafact.com Hardware : Intel Core i3 Processor RAM : 2GB Swap : 4GB Platform : Oracle Enterprise Linux 5 – 64bit Network Card : Two (eth0, eth1) Network Setting host name : IP Address eth0 : Default Subnet Mask : Default Gateway eth0 : Primary DNS : IP Address eth1 : Default Subnet Mask : Default Gateway eth1 :

rac3.orafact.com 192.168.1.85 (public address) 255.255.255.0 192.168.1.1 (public address) 192.168.1.1 192.168.2.85 (private address) 255.255.255.0 none

Editing Hosts file (For all the Nodes) [root@rac1 ~]# vi /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost # Public 192.168.1.81 rac1.orafact.com rac1 192.168.1.82 rac2.orafact.com rac2 192.168.1.85 rac3.orafact.com rac3 # Private 192.168.2.81 rac1-priv.orafact.com rac1-priv 192.168.2.82 rac2-priv.orafact.com rac2-priv 192.168.2.85 rac3-priv.orafact.com rac3-priv # Virtual 192.168.1.83 rac1-vip.orafact.com rac1-vip 192.168.1.84 rac2-vip.orafact.com rac2-vip 192.168.1.86 rac3-vip.orafact.com rac3-vip

3

ORAFACT

                                                                                                          Adding Node to RAC   Checking Network Connection (From All the Node) [root@rac1 ~]# ping rac2 [root@rac1 ~]# ping rac3 [root@rac1 ~]# ping rac2-priv [root@rac1 ~]# ping rac3-priv [root@rac2 ~]# ping rac1 [root@rac2 ~]# ping rac3 [root@rac2 ~]# ping rac1-priv [root@rac2 ~]# ping rac3-priv [root@rac3 ~]# ping rac1 [root@rac3 ~]# ping rac2 [root@rac3 ~]# ping rac1-priv [root@rac3 ~]# ping rac2-priv Installing Packages (Node RAC3) Node RAC3 needs the below packages: Below Packages are already available [root@rac3 ~] rpm -qa binutils-2.17.50.0.6 [root@rac3 ~] rpm -qa gcc-4.1.1 [root@rac3 ~] rpm -qa gcc-c++-4.1.1 [root@rac3 ~] rpm -qa glibc-common-2.5 [root@rac3 ~] rpm -qa make-3.81 [root@rac3 ~] rpm -qa elfutils-libelf-0.125 [root@rac3 ~] rpm -qa elfutils-libelf-devel-0.125 [root@rac3 ~] rpm -qa glibc-2.5-12 [root@rac3 ~] rpm -qa glibc-devel-2.5 [root@rac3 ~] rpm -qa libaio-0.3.106 [root@rac3 ~] rpm -qa libgcc-4.1.1 [root@rac3 ~] rpm -qa libstdc++-4.1.1 [root@rac3 ~] rpm -qa libstdc++-devel 4.1.1 packages are not available. Install the packages from both the nodes [root@rac3 ~] rpm -ivh compat-libstdc++-33-3.2.3-61.i386.rpm [root@rac3 ~] rpm -ivh compat-libstdc++-33-3.2.3-61.x86_64.rpm [root@rac3 ~] rpm -ivh libaio-devel-0.3.106-3.2.x86_64.rpm [root@rac3 ~] rpm -ivh numactl-devel-0.9.8-7.el5.x86_64.rpm [root@rac3 ~] rpm -ivh sysstat-7.0.2-3.el5.x86_64.rpm [root@rac3 ~] rpm -ivh unixODBC-2.2.11-7.1.x86_64.rpm [root@rac3 ~] rpm -ivh unixODBC-devel-2.2.11-7.1.x86_64.rpm

4

ORAFACT

                                                                                                          Adding Node to RAC   Setting Kernel Parameters (Node RAC3) [root@rac3 ~]# cat /etc/sysctl.conf fs.file-max = 6553600 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 1024 65000 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 kernel.hostname=rac3 kernel.domainname=orafact.com Configuring limitations and others (Node RAC3) [root@rac3 ~]# cat /etc/security/limits.conf #For RAC oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 [root@rac3 ~]# cat /etc/pam.d/login #for RAC session required pam_limits.so [root@rac3 ~]# cat /etc/selinux/config SELINUX=disabled [root@rac3 ~]# vi /etc/rc.local Add the following to /etc/rc.local file in RAC3 chown oracle:oinstall /dev/sdb1;chmod 600 /dev/sdb1 chown oracle:oinstall /dev/sdc1;chmod 600 /dev/sdc1 chown oracle:oinstall /dev/sdd1;chmod 600 /dev/sdd1 chown oracle:oinstall /dev/sde1;chmod 600 /dev/sde1 chown oracle:oinstall /dev/sdf1;chmod 600 /dev/sdf1 Creating Groups and Users (Node RAC3) [root@rac3 ~]# groupadd oinstall [root@rac3 ~]# groupadd dba [root@rac3 ~]# groupadd oper [root@rac3 ~]# groupadd asmadmin [root@rac3 ~]# useradd -u 501 -g oinstall -G dba,oper,asmadmin oracle [root@rac3 ~]# passwd oracle 5

ORAFACT

                                                                                                          Adding Node to RAC   Making Directories (Node RAC3) [root@rac3 ~]# mkdir /u01/crs/11.1.0.6 [root@rac3 ~]# mkdir /u01/db/11.1.0.6 [root@rac3 ~]# chown -R oracle:oinstall /u01 [root@rac3 ~]# chmod -R 775 /u01 Editing bash profile (Node RAC3) [oracle@rac3 ~]# vi .bash_profile export TMP=/tmp export TMPDIR=/tmp export ORACLE_HOSTNAME=rac3.orafact.com if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi Creating Environment files (From Node RAC3 for Cluster, ASM and Database) [oracle@rac3 ~]# vi crs.env export ORACLE_BASE=/u01/crs export ORACLE_HOME=$ORACLE_BASE/11.1.0.6 export ORACLE_SID=racdb export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib export PATH=$ORACLE_HOME/bin:$PATH [oracle@rac3 ~]# vi asm.env export ORACLE_BASE=/u01/db export ORACLE_HOME=$ORACLE_BASE/11.1.0.6 export ORACLE_SID=+ASM3 export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib export PATH=$ORACLE_HOME/bin:$PATH [oracle@rac3 ~]# vi racdb.env export ORACLE_BASE=/u01/db export ORACLE_HOME=$ORACLE_BASE/11.1.0.6 export ORACLE_SID=racdb3 export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib export PATH=$ORACLE_HOME/bin:$PATH 6

ORAFACT

                                                                                                          Adding Node to RAC   Verifying Configuration information [oracle@rac1 ~]$ /stage/database/runcluvfy.sh stage -pre crsinst -n rac1,rac2,rac3 -verbose > preclust_veri.txt

1.2 Adding Oracle Clusterware to the new node [oracle@rac1 ~]$ cd /u01/crs/11.1.0.6/oui/bin/ [oracle@rac1 bin]$ ./addNode.sh Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 3872 MB Passed Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed Oracle Universal Installer, Version 11.1.0.6.0 Production Copyright (C) 1999, 2007, Oracle. All rights reserved. Welcome Screen. Click Next

7

ORAFACT

                                                                                                          Adding Node to RAC   Specify Cluster node to add to install and Click Next

Click Install

8

ORAFACT

                                                                                                          Adding Node to RAC   Clusterware Installation ongoing on Node-3....

As per the message run the scripts as root user respectively [root@rac3 ~]# /u01/oraInventory/orainstRoot.sh Changing permissions of /u01/oraInventory to 770. Changing groupname of /u01/oraInventory to oinstall. The execution of the script is complete [root@rac1 ~]# /u01/crs/11.1.0.6/install/rootaddnode.sh clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Attempting to add 1 new nodes to the configuration Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node : node 3: rac3 rac3-priv rac3 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. /u01/crs/11.1.0.6/bin/srvctl add nodeapps -n rac3 -A rac3-vip/255.255.255.0/eth0 [root@rac3 ~]# /u01/crs/11.1.0.6/root.sh WARNING: directory '/u01/crs' is not owned by root WARNING: directory '/u01' is not owned by root Checking to see if Oracle CRS stack is already configured 9

ORAFACT

                                                                                                          Adding Node to RAC   /etc/oracle does not exist. Creating it now. OCR LOCATIONS = /dev/sdb1 OCR backup directory '/u01/crs/11.1.0.6/cdata/rac_cluster' does not exist. Creating now Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/u01/crs' is not owned by root. Changing owner to root The directory '/u01' is not owned by root. Changing owner to root clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node : node 1: rac1 rac1-priv rac1 node 2: rac2 rac2-priv rac2 clscfg: Arguments check out successfully. NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. Cluster Synchronization Services is active on these nodes. rac1 rac2 rac3 Cluster Synchronization Services is active on all the nodes. Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(1M)

1.3 Configuring ONS for the new node [oracle@rac1 bin]$ cat /u01/crs/11.1.0.6/opmn/conf/ons.config localport=6150 useocr=on allowgroup=true usesharedinstall=true [oracle@rac1 bin]$ ./racgons add_config rac3:6250

1.4 Adding RAC home to the new node [oracle@rac1 ~]$ cd /u01/db/11.1.0.6/oui/bin/ [oracle@rac1 bin]$ ./addNode.sh 10

ORAFACT

                                                                                                          Adding Node to RAC   Welcome Screen. Click Next

Specify Cluster Nodes to add to Install and Click Next

11

ORAFACT

                                                                                                          Adding Node to RAC   Click Install

Database Server software Installation ongoing on Node-3

12

ORAFACT

                                                                                                          Adding Node to RAC   Run the script as root user as per the message and Click OK

[root@rac3 ~]# /u01/db/11.1.0.6/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/db/11.1.0.6 Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions. You have new mail in /var/spool/mail/root

1.5 Adding a listener to the new node [oracle@rac3 ~]$ . racdb.env [oracle@rac3 ~]$ netca Oracle Net Services Configuration:

13

ORAFACT

                                                                                                          Adding Node to RAC  

Select Cluster Configuration and Click Next

Select Node-3(rac3) and Click Next

14

ORAFACT

                                                                                                          Adding Node to RAC  

Select Listener Configuration and Click Next

Select Add and Click Next

15

ORAFACT

                                                                                                          Adding Node to RAC  

Specify Listener name and Click Next

Select necessary PROTOCOL and Click Next

16

ORAFACT

                                                                                                          Adding Node to RAC   Select Port number for Listener and Click Next

Select No then Click Next and Finish

[oracle@rac2 ~]$ srvctl config listener -n rac3 rac3 LISTENER_RAC3

17

ORAFACT

                                                                                                          Adding Node to RAC  

1.6 Adding a ASM instance to the new node [oracle@rac3 ~]$ dbca Select Oracle Real Application Clusters Database and Click Next

Select Configure Automatic Storage Management and Click Next

18

ORAFACT

                                                                                                          Adding Node to RAC   Select Node rac3 and Click Next

To extend ASM on rac3 node Click Yes

Specify ASM password and Click OK

19

ORAFACT

                                                                                                          Adding Node to RAC   Click Finish and Click Yes to do another task.

1.7 Adding a database instance to the new node Select Instance Management and Click Next

20

ORAFACT

                                                                                                          Adding Node to RAC   Select Add an instance and Click Next

Select Database name and Specify Username, password and Click Next

21

ORAFACT

                                                                                                          Adding Node to RAC   Click Next to add a new instance

Specify instance name for rac3 and Click Next

22

ORAFACT

                                                                                                          Adding Node to RAC   Verify Database Storage information and Click Finish

Click OK to confirm

23

ORAFACT

                                                                                                          Adding Node to RAC  

Adding instance to rac3 node

Verification [oracle@rac2 ~]$ crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1 ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1 ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1 ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1 ora.rac1.vip application 0/0 0/0 ONLINE ONLINE rac1 ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2 ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2 ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2 ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2 ora.rac2.vip application 0/0 0/0 ONLINE ONLINE rac2 ora....SM3.asm application 0/5 0/0 ONLINE ONLINE rac3 ora....C3.lsnr application 0/5 0/0 ONLINE ONLINE rac3 ora.rac3.gsd application 0/5 0/0 ONLINE ONLINE rac3 ora.rac3.ons application 0/3 0/0 ONLINE ONLINE rac3 ora.rac3.vip application 0/0 0/0 ONLINE ONLINE rac3 ora.racdb.db application 0/0 0/1 ONLINE ONLINE rac1 24

ORAFACT

                                                                                                          Adding Node to RAC   ora....b1.inst application 0/5 0/0 ONLINE ONLINE rac1 ora....b2.inst application 0/5 0/0 ONLINE ONLINE rac2 ora....b3.inst application 0/5 0/0 ONLINE ONLINE rac3 SQL> show parameter db_name NAME TYPE VALUE ------------------------------------ ----------- -----------------------------db_name string racdb SQL> show parameter instance_name NAME TYPE VALUE ------------------------------------ ----------- -----------------------------instance_name string racdb3 Adding a node to 11g RAC completed successfully. ******END******

25

ORAFACT