Transcript Administering Oracle E-Business R12 in Highly Available
Administering Oracle E-Business R12 in Highly Available Environments
Mika Banda Applications DBA Team Leader Amway [email protected]
Disclaimer:
Content provided for knowledge sharing purposes only.
Images belong not to me.
Content intentionally intended to educate.
I do not know everything.
Agenda
• Introduction – Amway and my background • Part 1 – Personal High Availability – Cool to the new guy, okay to old timer – High Availability starts with you the support person • Part 2 – Oracle E-Business High Availability at Amway – Cool to the old timer, wow to the new guy – Lots of metalink notes • Part 3 – Bonus [Time Permitting] – One thing to know about EXA World • Open Discussion
Introduction
• Mika Banda – 18 Years in Oracle – Old school – command prompt – Oracle 6 – 11g (Saw ver. 5 & forms 2.5), EBS 10.7 NCA – R12 – Delta*, Government, Enron, GE, and now Amway • Amway: – Oracle E-Business R12.1.2 and 12.1.3, Oracle 11.2.0.x
– Shared appltop, PCP, SRDF FS – Exadata, Exalogic, Blades, AIX, RAC, HACMP, EMC DMX – Passive DR, active standby
FACTS – Not Paid By Keystrokes
Type Less To Do More • Unleash your tricks • Login Time • Typo Time • GUI Time • Format Time • Security Time • Translation Time • Time to connect Ps –ef pipe cat grape peemon
FACTS – Not Paid By Keystrokes
• Fast Access Tools • Crackberry Compatible • Short Coded names • Team Language • Tele-Unix • Manager Friendly • FAST Diagnostics
FACTS Example – EBS Forms
• Forms Process Diagnostics • All or limited to 1+ PIDs • 4 Characters
FACTS Example – EBS FNDLIBR
• psg = ps –ef| grep $@ • fndlibr is 7 characters: – Credentials – SQL*Plus – Process Details
FACTS Example – EBS CP
• 4 characters • Sub-second turnaround • Overloaded – 0+ parameters • A load of information 4 characters away
FACTS Example – Blockers
• Manager Safe • Needs time – format challenge.
• Change management – team effort
More FACTS Examples
• • • • • • • • • • • • • • • • appnodes mycommands invalids mypatches myurl dobackup backups myPO allsga mydblinks mydown hotproc lastddl lastusers lcl mydirectories • • • • • • • • • • • • • • • • myPatches mytop10segs mytopmem mytree myprocess mypmon myoraerr mysess mysidmem myspid mysgatot mycpus mymem myinstances myconcreq mywfm • • • • • • • • • • • • • • • • ASH appnodes oraports COI oh sgatot blocks lsnr mtns CLONE p file2patch SR SHORTS getPatches psg
Over and Above FACTS
• AMP 4.0 Plugin • OEM 11g/12c • Nimbus Tool$ • HP PPM Tool$
EBS HA/DR Options Implemented
Application Tier – Multiple Nodes – Shared Appltop – Load-balancing – Redundant Hardware – Virtual Servers – Capacity On Demand – SRDF (Symmetrix Remote Data Facility) Database Tier – RAC – Data Guard – PowerHA/HACMP – HW Redundancy – Virtualization – Replication – Timefinder BCV
EBS HA: Shared Application-Top
Always in sync, no multiple copy overhead, above all allows Distributed AD Node 1: Start adpatch with Distributed AD options: $ adpatch workers=27 localworkers=9 Node 2: Start adctrl - Distributed is yes. Workers prompted.
$ adctrl distributed=y [Enter the worker range: 10-18] 1 Shared Filesystem 2 3 Node 3: Start adpatch - Distributed is yes. Workers prompted.
$ adctrl distributed=y [Enter the worker range: 19-27] See Notes 384248.1, 1324602.1, 236469.1, 233428.1
EBS HA: Parallel Concurrent Processing Multiple Concurrent Processing Nodes (assume use of all three): Setup - ** Remember FNDSM check before starting ** • Activate Internal Monitors for all nodes (+ worker + shift) • Define Service Managers for all nodes (+ worker + shift) • Point $APPLCSF on all nodes to shared file system • Point $APPLPTMP on all nodes to shared file system • $APPLPTMP must be part of UTL_FILE_DIR • Set PO 'Concurrent: PCP Instance Check' to OFF • Enable Reviver : 's_cp_reviver = enabled’ • Define node1 & node2 for all* managers.
1 Shared Filesystem: tech_st + appl_st 2 3 Interconnect Notes 1359612.1*, 783044.1, 1389261.1, 551895.1, 466752.1
EBS HA: Multi-Node AppTier + RAC
Multiple Concurrent Processing Nodes (assume use of all three): Client Connections
Database Load Balancing:
• Point CP_TWOTASK and jdbc_url to loadbalanced entries • cp_twotask => concurrent managers TWO_TASK • jdbc_url => DBC file APPS_JDBC_URL (Java Managers – e.g. OPP, WF) • Remote Listener unset to disable for PCP 1 Load Balance 2 3 Application Load Balancing: • Web Host entry point = s_webentryhost • Web domain entry point = s_webentrydomain • Active Web Port = s_active_webport • Login Page URL = s_login_page • External URL info for 3 rd party Access = s_external_url • Chronos URL = s_chronosURL • EndUserMonitoringURL = s_endUserMonitoringURL Load Balance Interconnect Shared Filesystem: tech_st + appl_st See Notes 727171.1, 380489.1, 217368.1
, 477835.1,
EBS DR 1 – Match Production Load
Primary Secondary • Production Load DR • Read-Only DR • Reporting offload • Backup Offload • Outage Reduction Options
HA/DR: Reducing Time To Patch EBS
Pre-reqs : Duplicate, updated snapshot, updated rapid clone/autocfg, same platform Unwrapping a 30 hour Patch : a) Copy files b) Bin linking c) JSPs + Jars d) Data + SQL C D = 18hrs = 4 hrs e) Gen Forms f) g) G = 8hrs Gen Reports Patch History … Primary 4 3 1 6 5 Copy 2 Improved 8hr Process : 1 – Clone 2 – Apply Patch to Copy 3 – Sync files 4 - databaseportion 5 – Patch History 6 – Other files See Note 734025.1, 387859.1,406982.1
HA/DR: Reducing Time To Patch EBS
4
- $ adpatch options=nocopyportion,nogenerateportion driver=u_merged.drv
6
Do
NOT
copy the following files: $APPL_TOP/admin/
$COMMON_TOP/html/US/ICXINDEX.htm
$COMMON_TOP/html/_pages $COMMON_TOP/html/iby_debug.log
$COMMON_TOP/html/iby_error.log
$FND_TOP/log $FND_TOP/out
Export Patch History:
$ perl $AD_TOP/bin/adphmigr.pl userid=apps/apps startdate='2012/04/10 00:00:00' enddate='2012/04/10 02:00:00' appsystemname=stage appltopname=FastPatch nodatabaseportion=Y
Import Patch History:
Copy 2 files (javaupdates
The next adpatch [now or later] – will load these files to update history.
See Note 734025.1, 387859.1,406982.1
EBS EXA Implementation [Bonus]
Understand Your Network Connections For each Compute Node: Client TNS entries differ from Application FB based entries
Client Network (10GB) Infiniband Network (40GB)
App Compute Node 1 [A1,A2,A3] A3 A2 CN One Administration Network (1GB) A1 D1 D2 CN One D3 DB Compute Node 1 [D1,D2,D3]
EBS EXA Implementation [Bonus]
Understand Your Network Connections For each Compute Node: Client TNS entries differ from Application FB based entries
Client Network (10GB) Infiniband Network (40GB)
App Compute Node 1 [A1,A2,A3] A3 A2 CN One Administration Network (1GB) A1 BAD TNS D1 D2 CN One D3 DB Compute Node 1 [D1,D2,D3]
EBS EXA Implementation [Bonus]
Understand Your Network Connections For each Compute Node: Client TNS entries differ from Application FB based entries
Client Network (10GB) Infiniband Network (40GB)
App Compute Node 1 [A1,A2,A3] A3 A2 CN One Administration Network (1GB) A1 Inefficient TNS D1 D2 CN One D3 DB Compute Node 1 [D1,D2,D3]
EBS EXA Implementation [Bonus]
Understand Your Network Connections For each Compute Node: Client TNS entries differ from Application FB based entries
Client Network (10GB)
Best TNS
Infiniband Network (40GB)
App Compute Node 1 [A1,A2,A3] A3 A2 CN One Administration Network (1GB) A1 Best TNS D1 D2 CN One D3 DB Compute Node 1 [D1,D2,D3]