Transcript Configure HACMP and Install Oracle 10g RAC on the HACMP

IBM eServer pSeries
Configuration of HACMP and
Installation of Oracle 10g RAC
on IBM pSeries
Technical Sales Support, IBM Korea
김 주 홍([email protected])
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
AIX Environments
 AIX Version
– AIX 5L Version 5.2 ML01(ML03 Recommended) or latest
• IY43980, IY44810, IY45462, IY,45707, IY46214, IY46605, IY48525, IY51801, IY42783,
IY43602, IY45695, U496124
• IY 45695, IY42783, IY44237, IY49116 for 2-Port Gigabit Ethernet-SX PCI-X Adapter(#5707)
• IY44237 for p670/p690+
 Required filesets for Oracle 10g RAC
2
Filesets
Version
bos.adt.base
5.2.0.30
bos.adt.lib
5.2.0.10
bos.adt.libm
5.2.0.30
bos.perf.libperfstat
5.2.0.30
bos.perf.perstat
5.2.0.30
bos.perf.proctools
5.2.0.30
rsct.basic.rte
2.3.3.2
rsct.compat.basic.hacmp
2.3.3.0
rsct.compat.clients.hacmp
2.3.3.0
bos.clvm
5.2.0.30
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
HACMP Environments and Creation of Oracle User
 HACMP Version
– HACMP V5.1 with lastest patch
 Required fileset for Oracle10g RAC
Filesets
Version
Cluster.es
5.1 with latest patch
Cluster.es.clvm
5.1 with latest patch
Cluster.es.cspoc
5.2 with latest patch
 Create Oracle User on the each node
– User
: oracle(uid:901)
– Primary Group : oinstall(gid:902)
– Group Set
3
: dba(gid:901), hagsuser(gid:903)
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Oracle User Environments
 Example of ~oracle/.profile
umask 022
export ORACLE_BASE=/oracle
export ORACLE_HOME=/oracle/product/10.1.0/db_1
export ORA_CRS_HOME=/oracle/product/10.1.0/crs
export ORACLE_SID=rac1(or rac2)
export PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin
 Edit to enable rlogin between each nodes
– ~oracle/.rhosts
 Tuning Network Parameters
– tcp_recvspace = 65536, tcp_sendspace = 65536
– udp_recvspace = 655360, udp_sendspace = 65536
– iqpmaxlen = 512, rfc1323 = 1, sb_max = 2 * udp_sendspace
4
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
AIX Environments - others
 Paging space, /tmp filesystem
– paging space
: min 1GB
– /tmp filesystem : min 400MB
 Create /oracle filesystem
– min 5GB
– # chown oracle:oinstall /oracle
– # chmod –R 775 /oracle
5
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Creation of Volume Group
 Create shared VG for Oracle Datafile and CRS
– must create separate VG for Oracle Datafile and CRS
– oravg : Oracle Datafile
smitty cl_conlvm
Concurrent Volume Groups
Create a Concurrent Volume Group
– crsvg : Oracle CRS
mkvg –y’crsvg’ –s’32’ ‘–n’ –V’46’ ‘–C’ hdisk8
varyonvg crsvg
6
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Creation of Logical Volume
 Create Logical Volume for Oracle Datafiles and CRS files
mklv –y’rac_system’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_sysaux’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_undotbs1’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_undotbs2’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_example’ –T O ‘-S64K’ oravg 8 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_users’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_redo1’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_redo2’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_redo3’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_redo4’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_control1’ –T O ‘-S64K’ oravg 8 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_control2’ –T O ‘-S64K’ oravg 8 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_temp’ –T O ‘-S64K’ oravg 32 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_spfile’ –T O ‘-S64K’ oravg 1 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_pwdfile’ –T O ‘-S64K’ oravg 1 hdisk4 hdisk5 hdisk6 hdisk7
mklv –y’rac_ocr’ –T O ‘-S64K’ oravg 5 hdisk4 hdisk5 hdisk6 hdisk7(minimum 100MB)
mklv –y’rac_vote’ –T O ‘-S64K’ oravg 2 hdisk4 hdisk5 hdisk6 hdisk7
7
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Creation of Logical Volume
 Change the owner, permission for LV on the each node
# chown oracle:dba /dev/rrac_*
# chmod 660 /dev/rrac_*
 Create the DBCA raw device mapping file
# cat /oracle/rac_raw.conf
control1=/dev/rrac_control1
control2=/dev/rrac_control2
example=/dev/rrac_example
pwdfile=/dev/rrac_pwdfile
redo1_1=/dev/rrac_redo1
redo1_2=/dev/rrac_redo2
redo2_1=/dev/rrac_redo3
redo2_2=/dev/rrac_redo4
spfile=/dev/rrac_spfile
sysaux=/dev/rrac_sysaux
system=/dev/rrac_system
temp=/dev/rrac_temp
users=/dev/rrac_users
undotbs1=/dev/rrac_undotbs1
undotbs2=/dev/rrac_undotbs2
8
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Hardware Environments
Machine Room Public LAN : 192.1.10/24
100Base-T
pvl31 : 192.1.10.31
7028-6C4 pSeries 630 model 6C4
LPAR1
POWER4+ 1.45GHz 2/4way
7/16GB memory
Inter Connect : 192.1.11/24
1000Base-TX
pvl31_int :
192.1.11.31
pvl41_int :
192.1.11.41
1000Base-TX
pvl41 : 192.1.10.41
7028-6C4 pSeries 630 model 6C4
LPAR2
POWER4+ 1.45GHz 2/4way
7/16GB memory
2Gbps Fibre Channel
Controller A
Controller B
1742-1RU FAStT 700
2GB Cache
]
1740-1RU EXP 700
1740-1RU EXP 700
RAID5 27+P
15krpm
36GB
28disk
9
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Add an HACMP Cluster
Smitty hacmp
Extended Configuration
Extended Topology Configuration
Configure an HACMP Cluster
Add/Change/Show an HACMP Cluster
10
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Configure HACMP Nodes
Smitty hacmp
Extended Configuration
Extended Topology Configuration
Configure an HACMP Nodes
Add a Node to the HACMP Cluster
11
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Configure HACMP Networks
Smitty hacmp
Extended Configuration
Extended Topology Configuration
Configure an HACMP Networks
Add a Networks to the HACMP Cluster
12
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Configure HACMP Communication Interfaces/Devices
Smitty hacmp
Extended Configuration
Extended Topology Configuration
Configure an HACMP Communication Interfaces/Devices
Add Communication Interfaces/Devices
13
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Add a Resource Group
Smitty hacmp
Extended Configuration
Extended Resource Configuration
HACMP Extended Resource Group Configuration
Add a Resource Group
14
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Change Resources and Attributes for a Resource Group
Smitty hacmp
Extended Configuration
Extended Resource Configuration
HACMP Extended Resource Group Configuration
Change/Show Resource and Attributes for a Resource Group
15
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Verification and Synchronization
Smitty hacmp
Extended Configuration
Extended Verification and Synchronization
16
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Startup HACMP
Smitty hacmp
Extended Configuration
Extended Verification and Synchronization
17
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Import Volume Group
 Import Volume Group for Oracle CRS file
# lsvg –o
rootvg
oravg
crsvg
# varyoffvg crsvg
# importvg –y crsvg –V 46 hdisk8(on the pvl41)
# chvg –a n crsvg
# varyonvg –c crsvg
 Must be not include crsvg in the HACMP Resource Group
OK !! Let’s Install Oracle 10g RAC
18
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle Cluster Ready Service(CRS)
 What is CRS
– S/W stack that provides inter-node communications and cluster failover
capabilities
 Daemon Processes
– crsd
• provides high availability services
• manages application services
• manages data in OCR
– cssd
• provides access to node membership
• provides cluster locking
• failure of the daemon intentionally causes the node to reboot
– evmd
• event manager daemon
19
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle Cluster Ready Service(CRS)
 Mount the /work directory that contain Oracle CRS Software
# mount pvl2:/work /mnt
 Run rootpre.sh script as root user on the each node
# cd /mnt/crs/Disk1/rootpre
#./rootpre.sh
 Check Oracle Environments
# xhost +
# su – oracle
# set | grep ORA
ORACLE_BASE=/oracle
ORACLE_HOME=/oracle/product/10.1.0/db_1
ORA_CRS_HOME=/oracle/product/10.1.0/crs
ORACLE_SID=rac1
# export DISPLAY=pvl320.0
20
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install ORACLE CRS
 Run Oracle Installer as oracle user
# cd /mnt/crs/Disk1
# ./runInstaller
When the following message is displayed, enter y
Has ‘rootpre.sh’ been run by root?[y/n](n) y
Click Next
21
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS – Specify Inventory directory
2. Run /oracle/oraInventory/
orainstRoot.sh as root user
and Click Continue
1. Click Next
22
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS – Specify File Locations
1. Input OUICRS on Name
2. Input /oracle/product/
10.1.0/crs
3. Click Next
23
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS – Language Selection
Select Languages and
Click Nexth
24
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS – Cluster Configuration
Input Node Name for
Interconnect Networks
25
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS – Private Interconnect Enforcement
Select public and
Private for Interface Type
26
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS – Oracle Cluster Registry
Specify OCR Location
27
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS – Voting Disk
1. Input voting file name and
Click Next
2. Run /oracle/oraInventory/
orainstRoot.sh on all nodes
3. Click Continue
28
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS - Summary
1. Check Cluster Nodes
and Remote Nodes
2. Click Install
29
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS
1. Install on the local node(pvl31)
2. Copy oracle files on the remote node(pvl41)
30
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS – run roor.sh on each nodes
1. After Installation of Oracle CRS
finished, run /oracle/product/10.1.0/crs/
root.sh on the pvl31, plv41
2. Click OK
31
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS – Configuration Assistants
Check Status and
Click Next
32
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle CRS
33
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Check the Status of Oracle CRS
Check CRS Status on each nodes using
olsnodes, oifcfg commands
34
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle 10g RAC
 Run Oracle Installer as oracle user
# cd /mnt/db/Disk1
# ./runInstaller
When the following message is displayed, enter y
Has ‘rootpre.sh’ been run by root?[y/n](n) y
Click Next
35
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle 10g RAC – Specify File Locations
Check Destination Name
and Path
Click Next
36
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle 10g RAC – Specify HW Cluster Installation Mode
Select pvl41 and
Click Nest
37
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle 10g RAC – Select Installation Type
Select Installation Type and
Click Next
38
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle 10g RAC
Click Next
39
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle 10g RAC – Select Database Configuration
Select Database Configuration
And Click Next
40
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle 10g RAC – Specify DB Configurations Options
1. Specify Global Database
Name And if need,
check Create Database
with sample schemas
2. Click Next
41
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC – Select DB Management Option
42
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC – Specify DB File Storage Option
1. Click Raw Devices
2. Specify Raw Device
Mapping file
3. Click Next
43
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC – Specify Backup and Recovery Options
44
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC – Specify DB Schema Passwords
1. Specify password for
All the accounts
2. Click Next
45
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC - Summary
1. Check Cluster Nodes
2. Check Remote Nodes
3. Click Install
46
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC
1. Installing on the local node
2. Copying to the remote node
47
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC – run root.sh on each nodes
After installation on the local node finished,
1. Set DISPLAY env. on the each nodes
2. Run /oracle/product/10.1.0/db_1/root.sh
on the local node
3. Click OK
48
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC
49
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC – Network Interfaces
Select en0 for vip
and Click Next
50
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC – Virtual IPs for Cluster nodes
Specify IP Alias Name,
IP address and Click Next
51
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC
1. Click Finish,
2. Configuration Assistants
Process Dialog가 Display
되고 Configuration이 완료
되면 OK Click
52
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC
1. Click Exit
2. Run /oracle/product/10.1.0/
db_1/root.sh on the remote
node
53
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Check Status of Oracle CRS
Check status of Oracle CRS using
crs_stat –t on the each nodes
54
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC
1. Click OK on the Setup
Privileges window
2. 자동으로 Configuration Assistants이 수행
55
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC
1. 자동으로 Database Configuration Assistant(DBCA)가
실행
2. If you change password, click Password Management,
else click OK
56
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC
57
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Install Oracle10g RAC
58
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Check Oracle CRS and Oracle10 RAC
 Run crs_stat –t on the each node as oracle user
59
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Check Network Status
 Run netstat –i on the each nodes
60
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Oracle Startup and Shutdown using srvctl tool
61
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Correct CSS script
 Oracle CRS와 Oracle10g RAC Installation종료 후 /etc/init.cssd
script를 다음과 같이 수정
– # vi /etc/init.cssd
VG_ACTIVATE=“/etc/varyonvg –c”
VG_DEACTIVATE=“/etc/varyoffvg”
VG_ACTIVATE=“/usr/sbin/varyonvg –c”
VG_DEACTIVATE=“/usr/sbin/varyoffvg”
62
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Oracle Enterprise Manager(EM)
 Check EM Status
– # emctl status dbconsole
< Cluster Database >
– # emctl status agent
 Start EM
– # set DISPLAY Environment
– # emctl start dbconsole
 Access from Client PC
– http://server:5500/em/
 Login to Database
– User Name : sys
– Connect as SYSDBA
63
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Oracle Enterprise Manager(EM)
< Memory Parameter >
< Cluster Database Instance >
64
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Oracle10g RAC clean-up procedure including CRS(1)
 Stop Oracle CRS daemons(by root user)
– # /etc/init.crs stop
 Remove the files which are invoked by init(by root user)
– # rm /etc/init.cssd
– # rm /etc/init.crs
– # rm /etc/init.crsd
– # rm /etc/init.evmd
– # rm /etc/rc/rc2.d/K96init.crs
– # rm /etc/rc/rc2.d/S96init.crs
– # rm –rf /etc/ora*
 Stop GSD Daemon(by oracle user)
– $ gsdctl stop
 Remove Oracle10g Software
– $ rm –rf $ORACLE_BASE
65
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation
IBM eServer pSeries
Oracle10g RAC clean-up procedure including CRS(2)
 Clean-up the OCR and Voting disks using dd command(by root
user)
– # dd if=/dev/zero of=/dev/rrac_ocr bs=8192 count=12800
– # dd if=/dev/zero of=/dev/rrac_vote bs=8192 count=2560
 Change permission and owner of OCR, Vote diks
– # chown oracle:dba /dev/rrac_*
– # chmod 660 /dev/rrac_*
 Confirm no VIP addresses on each nodes
– # netstat -in
66
Skyhawk Residency
2015-07-21
© 2004 IBM Corporation