O RACLE 11 G R EAL A PPLICATION C LUSTERS A DVANCED A DMINISTRATION Natik Ameen Session #352

Introduction

• Natik Ameen, Senior Oracle DBA • DBA for 8 years, with 4 years on RAC 9i, 10g • Financial, Gaming and Aviation industries • Intended for everyone who has basic knowledge of RAC

Objective

• RAC overview • ASM • Nodeapps • Metrics collection • Wait events • Best Practice • 11g New features

RAC Overview

• Clustered nodes • Cluster Interconnect • Virtual IP • Shared storage • NodeApps

RAC Architecture

• Clustered nodes • Interconnect Network • Shared storage

SGA Structure & Processes

Node 1 Node 2

DIAG LMON LMD0 LMON LMD0 DIAG Global Resource Directory

Dictonary Cache Buffer Cache Log Buffer

LCK0 LMS0

LGWR SMON DBW0 PMON Interconnect

Global Resource Directory

Dictonary Cache Buffer Cache Log Buffer

LCK0 LMS0

LGWR SMON DBW0 PMON Shared Storage • Global Resource Directory : Maintains mapping of data in SGA • Global Cache Server: Synchronizes Buffer cache across instances • Global Enqueue Services: Maintains coherency in data dictionary & library caches • Lock Manager Server: Transports blocks for Cache Fusion between nodes.

Cache Fusion

• What is it • Concurrency & Consistency • Scalability

Interconnect Network

• Main component for “Cache Fusion” • IC uses private network • User Datagram Protocol(UDP) • Reliable Datagram Socket Protocol(RDS) > 10.2.0.3

• Typical bandwidth utilization 20-30%, saturated > 70%

ASM Overview

• ASM Instance • Disk Group • Disk Files • ASMLib

ASM Instance

• Scaled down version of DB instance • DB instance access DB files

Disk Groups

• Striping • Mirroring • Failure groups

ASM Files

• Name pre-pended with ‘+’

ASMLib

• Kernel driver with efficient access to DB files

NodeApps

• Listener • Oracle Notification Services • Fast Application Notification • Fast Connection Failover

Metrics Collection

• AWR Reports • OS Utilities • Bench Marking

AWR Report

The RAC specific statistics include:

• Instance Global Cache Load Profile • Global Cache Efficiency percentages • Messaging Statistics • Top % CR and Current Block Segments

Cache Fusion data block & messaging traffic Calculate network traffic

RAC Wait Events

Most of these wait events occur during Cache Fusion • GC cr/current block 2-way, 3-way • GC cr/current block congested • GC cr/current block busy • GC current grant busy • GC cr/current block request • GC cr block lost

GC wait events

• Classified under “cluster wait lass”. • Characterized as “current” or as “CR”. • “current” classified as all blocks read into memory for the very first time • CR denotes block for read access • Master node maintains complete information on the resource, others maintain information on locally held locks

gc current block 2-way

1 A requests block from Master B "gc curent request" event on A

Instance A

3 gc current block 2-way

Instance B

Instance B finds block & sends to A 2 Instance A requests block from master instance B If the block is available on B then it is sent to A

gc current block 3-way

1 A requests block from Master B "gc curent request" event on A 2 B Forwards request to C

Instance A

4 gc current block 3-way

Instance B Instance C

C sents block to A 3 • Instance A requests block from master instance B • B does not have block but directs instance holding block or directs request to disk • Three hops maximum, not dependant on number of nodes

• • •

GC cr/current block congested

Repeated requests by foreground processes, not serviced by LMS Indicates LMS not able to keep up Queue lengths & scheduling delays in OS, can cause LMS delays • •

GC cr/current block busy

Delay for some reason, before block sent to requestor Could be due to I/O saturation •

GC current grant busy

Permission to access the block granted, but blocked by other requests ahead of it •

GC cr/current block request

Wait time, when cr or current block is being retrieved •

GC block lost

Lost blocks possibly due to faulty interconnect network

Where to get more information

• Other sessions here at IOUG-A Live!

• RAC Special Insterest Group(RAC SIG) • Consulting companies (depending on specialization)

Questions?

• Lots of things to cover • If we didn’t cover something you wanted to hear, please contact me

O RACLE 11 G R EAL A PPLICATION A DVANCED C LUSTERS A DMINISTRATION THANK YOU!

Please fill out evaluations!