Transcript Oracle Data Integration An Overview with Emphasis in DW Appliances Where Does Data Integration Fit? Essential Ingredient for Information Agility • Data Services •

<Insert Picture Here>
Oracle Data Integration
An Overview with Emphasis in DW Appliances
Where Does Data Integration Fit?
Essential Ingredient for Information Agility
• Data Services
• Information
services
• Process
Integration
• Event driven
data integration
• Governance
and Impact
Analysis
Business
Intelligence
SOA
Data
Integration
Enterprise Data
Warehousing
Master Data
Management
• Extract, Transform, Load
• Data Migration, Bulk Data
• Data Quality, Profiling
• Data Integration for
BI applications
• Event driven BI
• Heterogeneous Data
Access
• Information-based
analytics
• Report to Source
Data Lineage
Oracle Data Warehouse
Breaks Data Bandwidth Bottlenecks
• More Pipes
• Modular storage “cell” building blocks
organized into Massively Parallel Grid
• Bandwidth scales with capacity
• Bigger Pipes
• InfiniBand interconnect transfers data 5x
faster than Fibre Channel
• Ships Less Data Through Pipes
• Query processing is moved into storage
to dramatically reduce data sent to
servers while offloading server CPUs
• Runs the Oracle Database
• Database and options run unchanged
Exadata Moves a Lot
Less Data a Lot Faster
Oracle Master Data Management
Comprehensive MDM Application Solutions
Data Governance
Operational
Systems
Analytical
MDM Apps
Operational MDM Apps
Analytical
Systems
Analytical
Financial
Install Base
PeopleSoft
Product
SAP
Site
DW
Employee
EBS
Supplier
Dashboards
Customer
Siebel
BI &Datamarts
Planning
JDE
Custom Apps
External
Apps
Enterprise schema & Common
services
Oracle / Hyperion
Data Relationship
Management
Application Integration Architecture
Oracle Fusion Middleware
Financial
Consolidation
Budgeting
Use Cases for Oracle Data Integration
Unified Data Integration Platform for Enterprise Projects
Data Integration
BI & Data Warehouse
• Bulk-load historical data to new application
• Real-time Data Warehouse for BI
• Synchronize new and old applications
• Populate Warehouse with High Performance ODI
• Complex transformations, CDC
• Aggregates and aligns data for operational
analytics, performance management, etc
• Ensure database-level consistency across
applications bidirectional
Modernization
Data Migrations & Consolidation
• Mainframe / iSeries sources to Oracle RAC
• Upgrade Applications or Migrate to New Schema
• Re-architect legacy batch processing to SOA and
ODI-EE
• Single-time Bulk load and/or keep in Sync with
ongoing translation, delivery of data.
• Unification of structured and unstructured data
• Applications merge support
SOA Initiatives
Master Data Management
• Establish Messaging Architecture for Integration
• Create Single View of the Truth in real time
• Incorporate Efficient Bulk Data Processing with ODI
• Aggregating multiple operational sources
• Providing data access, transformation etc. as
services within SOA
• Synchronize Data with ODI-EE
• Supports master data stores, publishes master
data changes to all consumers
Introducing ODI-EE
Key Data Integration Products
• Comprehensive Integration
• ELT/ETL for Bulk Data
• Data Delivery Services
• Process Orchestration
• Service Bus
• Human Workflow
• Data Federation
• Replication & Migration
• Data Governance
• Data Grid
New
• ELT/ETL for any DB
• Changed Data Capture
• Native SQL Code Gen
• Declarative Design
• ELT for Oracle DB
• Database Modeling
• PL/SQL Code Gen
• Flow-based Design
• Business Data / Metadata
• Statistical Analysis
• Time Series Reporting
• Integrated Data Quality
• Cleansing & Parsing
• De-duplication
• High Performance
• Integrated w/ODI
Oracle Core Features
Comprehensive Data Integration Platform
• E-LT Data Processing
• Set-based Transformation
• Configurable Deployment (E-LT, E-T-L, or others)
• Native Code Generation
• Declarative Integration Designer
• Reverse Engineering of Sources
• Intuitive Data Mapping
• Built-in Reporting
• Metadata Lineage
• Graphical Impact Analysis
• Native SOA Interoperability
• SOA Sources, Targets, or Invoke-able E-LT Jobs, SOA Workflow, etc.
• Automatic Error Handling with Data Recycling
• Changed Data Capture
• Built-in Support for Realtime Data Capture and Streaming
• Log-based or Trigger-based
• Inline Data Quality
• Extensible Knowledge Module Framework
Data Integration Core Use Case
Comprehensive, Lightweight Data Integration
• Key Architecture Benefits: 100% Java, Open APIs, very fast E-LT
• Embeddable Java Agent consumes very little CPU or RAM
• E-LT Architecture uses DBMS CPUs, not the Application CPUs
• Open APIs, Open Metadata, Open XML Knowledge Modules
ERP
Application
Business Intelligence
& Data Warehouse
ODI
Agent
JKM
A
ODI Agent may be
deployed in any part
of the architecture
LKM
IKM
D
B
C$_0
LKM
C$_1
I$
File
C
E$ (Errors)
IKM
CKM
RKM
Extract-Load
Transform
Check-Load
ODI-EE Value Proposition
Agile Data Integration
Move and transform data. Mixed
sources and targets.
BENEFITS
1.
2.
3.
4.
5.
Performance
Flexibility
Productivity
Open
Hot-Pluggable
KEY DIFFERENTIATED FEATURES
>>>
>>>
>>>
>>>
>>>
Heterogeneous “E-LT”
Event-Driven Platform
Declarative Design
100% Java and SOA Native
Knowledge Modules
11
Differentiator: E-LT Architecture
High Performance
Conventional: Separate ETL Server
•
•
•
•
Proprietary ETL Engine
Poor Performance
High Costs for Separate Standalone Server
IBM & Informatica’s approach
Conventional ETL Architecture
Extract
Transform
Load
Oracle: No New Servers
• Lower Cost: Leverage Compute Resources &
Partition Workload efficiently
• Efficient: Exploits Database Optimizer
• Fast: Exploits Native Bulk Load & Other
Database Interfaces
• Scalable: Scales as you add Processors to
Source or Target
Benefits
• Optimal Performance & Scalability
• Better Hardware Leverage
• Easier to Manage & Lower Cost
Next Generation Architecture
“E-LT”
Transform
Transform
Extract
Load
1212
Differentiator: Changed Data Capture
Event-Driven Framework for Realtime Data Warehouse
Real Time Data Warehouse
Source Systems
Source DB’s
Redo Logs /
Triggers
Change
Tables
Logs
CDC
CDC
CDC
Records
Records
Records
Transactional
RDBMS
Logs
Transactional
RDBMS
Transactional
RDBMS
Logs
CDC
CDC
CDC
Records
Records
Records
CDC
CDC
CDC
Records
Records
Records
Data Warehouse
Subscription
Views
Staging
Area
Data
Warehouse
Focus on Oracle BIEE Suite Plus
Unify Data Integration with Business Intelligence
•
Oracle BIEE Suite Plus
Interactive
Dashboards
Ad hoc
Analysis
Proactive
Alerts
Reporting
& Publishing
Microsoft
Office
• Support for relational, R-OLAP,
OLAP sources & targets
Common Enterprise Information Model
Oracle Business Intelligence Server
Design
& Drill
Enterprise Data
Warehouse
•
• Oracle RAC
• Teradata
• Netezza…etc
•
Data Distribution & Delivery APIs
Metadata
Lineage
Bulk/Trickle
Loading
Changed Data
Capture
Master
Data
Data Quality
& Profiling
ODI Knowledge Module Framework
Bulk and Real-Time
Data Processing
Information
Assets
Other
Sources
Oracle
EBS
CDC
SAP/R3
PeopleSoft
Report-to-source lineage
• Drill-in from a Report
• Get query, DW schema, source
tables, transforms
• Gain confidence in analytics
Data
Flow
Oracle Data Integration Suite
Integrated with Oracle BIEE
Plus
Unified infrastructure
•
•
•
•
Unified metadata & lineage
Integrated Data Quality
Unified data access,
Common administration &
monitoring
• Integrated scheduling & security
• Common auditing & tracing
• Common error handling
Data
MessageWarehouse
Queues
14
Report to Source Data Lineage
Explore Lineage from Target Columns to Source Columns
Focus on OBI Analytic Applications
Overview
•
Oracle BI Analytic Applications
Sales
Contact
Center
Order
Mgt
Analytics Data
Warehouse
Supply
Chain
MKting
•
HR
Finance
•
•
•
Oracle BI Server and Semantic Layer
•
Data
Flow
•
•
•
Metadata
Repository
Pre-Built ODI Interfaces and
Transformation Packages for
Analytic Applications
Bulk
Transform.
Changed
Data
Capture
•
Pre-built Metrics, 5000+ Dashboards, Data
Warehouse 350+ Star Schema
Implementation Time: 3-4 Weeks
Works with EBS, PSFT, SAP, SEBL, JDEdwards Sources
Biggest Savings: Pre-built ETL
Faster & Lower Cost to Build & Maintain
ETLs
•
Oracle Data Integrator
ODI Knowledge
Modules for BI
Analytic
Applications
Pre-Built BI Analytic Applications
Knowledge Module Architecture Enables Efficient
Development of Packaged ETL
Reduce Complexity and Size of Code
Reuse DW Creation, Update, Load Times
Value-add Modules (e.g. SOX)
Extensible by Customers & System
Integrators
Build from Scratch
Data
Integrity
Data
Quality
Pre-built BI Apps
Training
Metrics &
Dashbrds
ODI Connectivity Framework
DW Design
Bulk and Real-Time
Data Processing
Informatio
n Assets
Other
Sources
Oracle
EBS
CDC
People
Soft
SAP/R
3
M
es
sa
g
e
Q
u
e
u
es
50%
ETL
Mapping
Data
Warehouse
Years or quarters
Weeks or months
Focus on Oracle | Hyperion Applications
Leveraging EPM with Embedded Data Integration
Oracle Hyperion
Planning
Oracle Hyperion
Financial Mgt
Planning API
Oracle Hyperion
Essbase
HFM API
Oracle Hyperion Application Adapters
Essbase API
Metadata Discovery
& Model Creation
Oracle | Hyperion Data Access
Authentication
Data Services
Oracle Data Integration Suite
Data Distribution & Delivery APIs
Metadata
Lineage
Bulk/Trickle
Loading
Changed Data
Capture
Master
Data
ODI Knowledge Module Framework
Bulk and Real-Time
Data Processing
Information
Assets
Other
Sources
Oracle
EBS
CDC
SAP/R3
PeopleSoft
Data
MessageWarehouse
Queues
Hyperion
Essbase



Use
Essbase KM


Extracts Dimension
Members
Use
Essbase KM


Loads Data



Loads Dimension
Members



Other Features
Data Quality
& Profiling
Hyperion
Financial
Management
Extract Data
Logging Services
API Layer
Hyperion
Planning
Cube Refresh
Consolidate
Calculate
Focus on Hyperion MDM
Foundation for Data Relationship Management (DRM)
•
Hyperion Data
Relationship
Management
• Derivation/inheritance
• Classification, categorization
• Attribute management
DRM
Import/Export
Profiles
•
•
ODI Knowledge Modules for DRM
Bulk
Transform.
Changed
Data
Capture
Data
Integrity
Bulk and Real-Time Data Processing
M
Information
Assets
Other
Sources
SAP/R
3
People
Soft
es
sa
g
e
Q
u
e
u
es
Data
Warehouse
Business-user driven
•
•
•
•
Data
Quality
ODI Connectivity Framework
Oracle
EBS
CDC
Change management
• Streamlined, automated
• Valid-from, Valid-to ranges
• Compare any 2 versions
Oracle Data Integrator
Metadata
Repository
Hierarchy management
•
Easy user declarative tool
Specify business rules
Multi-user collaboration
What-if & historical analysis
Meets IT requirements
• Synchronize with operational systems
• Complete auditing, fine-grained
security
• Configurable, no coding
18
Hyperion Data Relationship Management
Closed-loop Master Data Management
Hyperion Data Relationship Management
Operational
Analytic
Hierarchy Governance
Legacy
Analytics
Key Performance
SAP
Siebel CRM
People Soft
Oracle
Data
Integrator
Oracle
Data
Integrator
&
&
Oracle
Data
Quality
Oracle
Data
Quality
Query / Reporting
Enterprise
Reporting
Financial Apps
E-Business Suite
Master Hierarchies and Versions
JD Edwards
HR
Cost
Centers
Entities
Accounts
Org Struct
Geography
Budgeting
Planning &
Forecasting
19
ODI-EE A Look Ahead
ODI for Oracle Fusion
Data Integration as a Key Fusion Differentiator
• Fusion Applications
• Built-in, Automated Data Replication among ERP Applications
• Fusion Business Intelligence
• Pre-built Analytics from Fusion ERP Applications
• Automatic Mapping and Model Updates (Application-driven)
• Fusion Master Data Management
• Data Replication, Conflict Detection and Conflict Resolution
• Fusion w/Essbase
• Fusion Financials (Automatic Financial Consolidation)
• Pre-built Essbase Content with Apps and Business Intelligence
High-Level Data Integration Roadmap
Towards a Unified ETL Platform
Unified
Team
2009
2011
OWB
10gR1
OWB
10gR2
OWB
11gR1
OWB
11gR2
Unified
Platform
ODI
10gR3
ODI
11gR2
ODI
11gR1
2009
2010
• OWB/ODI Investments are
Fully Protected
• No Forced Migrations
• Natural Upgrade Path
• Unified Platform aims to be
a Superset of Existing
Products – no regression
ODI J2EE Deployment Architecture
Logical Architecture Overview
Any Web App
Any Java App
JDeveloper
Any Java App
WLS
ODI in JDev
Any Application
ODI SDK APIs
Designer
Operator
Security
Topology
Servlet Container
MBeans Server Registry
Any Application Container
Web Service Container
ODI Agent
Any MBeans App
Any Application
ODI Transform.
Service
ODI Thin Client
ODI MBeans
for ODI Agent
ODI SDK APIs
ODI Data
Services
Data Sources Connection Pool
RDBMS – ODI Repositories
ODI Master
Repository
Repository Data Sources
Sources Data Sources
Repository Data Sources
Sources Data Sources
Repository Data Sources
Sources Data Sources
ODI Work
ODI Work
Repository
ODI Work
Repository
Repository
Target Data Sources
Target Data Sources
Target Data Sources
Sources and Targets
---------------
---
Legacy
ERP
PLM
--------------CRM
--------------Best-of-Breed Applications
ODI 11gR1 New Features
Availability Planned for 1HCY10 Timeframe
•
•
•
•
•
•
•
•
•
•
•
•
•
JDeveloper-based Tooling
J2EE deployment option (using Weblogic Server)
Standalone Option is Preserved – using Jetty
JPS-based Security (external password management)
Enterprise Scheduler (ESS) Integration
Enterprise Manager Integration
New Thin Client (consolidated + more mgmt features)
Set-based Operators and Temporary Index Management
Java SDK APIs for Improved Embedability
JDE Enterprise One Knowledge Modules (v8.x and higher)
SAP ERP Knowledge Modules (v4.6 and ECC 6.0)
SAP iDocs Knowledge Modules (v4.6 and ECC 6.0)
SAP BW Knowledge Modules (v3.5 and v7.0)
24
Customer Examples
Enterprise Data Warehouse
Raiffeisen Bank
Business Challenges
•
•
Costs for Enterprise Data Warehouse
infrastructure were growing too fast
EDW processes not keeping up with 24x7
operations, ETL batches too slow
IT Challenges
•
•
Performance Issues with existing ETL Tool
(IBM DataStage)
New functions required for the next DWH
releases (realtime, replication, etc)
Oracle Data Integration Solution Benefits
•
•
•
•
•
•
Meets legal requirements for data consolidation in the head office
(Basel II, Investor reporting)
Seamless and realtime Capital Market reporting to investors and stakeholders
Single-source of truth for business information – supports Business Intelligence and EDW
One standardized enterprise data model over the whole RZB Group
Reduction of core integration/development costs through central development practice
Reduced time to market for new Group applications through central interface development
Order Replenishment, Online Catalog
J. Crew
Business Challenges
•
•
•
Order replenishment for online catalog system
Define new corporate standard for data integration
Better integrate data across 175 retail stores and 52
outlet chains
IT Challenges
• Replace antiquated home-grown solution of scripts
• Teradata bulk data support, and help integrate data
across SAP, Red Prairie Inventory systems
Oracle Data Integration Solution Benefits
•
•
•
Simplified the end-to-end data integration with all their core IT systems and database
platforms to reduce the total cost of ownership of their order replenishment and cataloging
Benefited from the Oracle advantage in batch processing, bulk data transformation for
increased performance
Successfully leveraged SAP and Teradata integration to phase out legacy systems and
ultimately reduce costs
Financial Services, Reinsurance
New Reinsurance, Division of the Munich Re Group
Business Challenges
•
•
Needed quality of data but uncompromising on the
cost to meet that objective.
Data interactions between multiple complex
systems, including both SAP and homegrown
insurance applications
IT Challenges
• Hard to modify and maintain manually-developed
data transfer programs.
• Data transfers as part of reinsurance, risk
transactions
Oracle Data Integration Solution Benefits
•
•
•
Seamlessly integrated across (Oracle applications (E-business Suite, Siebel CRM) SAP,
IBM, etc) to manage the regeneration of all code, scripts, and metadata across the
different environments.
Lowered the cost of development by taking advantage of ODI’s declarative model, ease of
use, and team collaboration capabilities.
NRC benefited from the flexibility and standards based approach of ODI, helped leverage
the existing environment.
Improve Inventory Control
Ross
Business Challenges
•
Hard to consolidate consistent, accurate inventory
information across over 800 multiple store chains
Lack of visibility to errors to the business
Higher cost and time to delivery of new value
•
•
IT Challenges
• Current state was complex, didn’t scale, and difficult
to manage
• Non-standard approach required coding paradigm,
inconsistent error handling
Oracle Data Integration Solution Benefits
•
•
•
•
Dain Hansen
Seamless data integration from heterogeneous sources leveraging SOA helped to create
a more flexible, standards based integration approach
73% Bulk data transfer performance improvement
Together with Oracle SOA Suite (Oracle BPEL PM) provided business optimization,
process visibility, exception handling
Closed loop processing using BAM and Data Integration combined eliminated
ordering/replenishing inventory errors by reducing inaccurate data
Shared Data Services
Verizon
Business Challenges
•
Streamline and automate business processes
around new hire on boarding
Provide better business insight and manage FiOS
supply chain
•
IT Challenges
• Lower the cost of rewriting SAP, PSFT & Legacy
interfaces (about 209) through custom code
• Lower the cost of IT infrastructure from multiple
systems
Oracle Data Integration Solution Benefits
•
•
•
Dain Hansen
ODI + SOA helped to build out a shared services layer which helped provide greater
flexibility
These shared services consolidate database (DB2, Sybase, Informix to Oracle),
consolidate ERP (PeopleSoft HR/Payroll, Decommissioning SAP), consolidate middleware
(Oracle FMW) and consolidate BI/ETL
In addition the program helped to decommission Cystal and Cognos and adopt Oracle
BIEE
Sales and Marketing Data Warehouse
Nestle
Business Challenges
• • Enable
Hard tothe
consolidate
inventory
marketingconsistent,
and CRM accurate
team to better
target
information
across
over
800
multiple
store
chains
its campaigns and activities across 50 countries
• • Anticipate
Lack of visibility
errors tochallenges,
the business
future to
reporting
in light of the
• strong
Higherincrease
cost andintime
delivery
new value
theto
volume
of of
production
data
IT Challenges
• Manage a data warehouse for operational
transactions and marketing information
• Scale available for high-value-added actionable data
on clients, orders, stock, etc.
Oracle Data Integration Solution Benefits
•
•
•
•
Dain Hansen
Replaced weekly insertion with daily recording of change in status with time imprint for all
clients—allowing for more effective client follow-up, for example
Improved response times by transferring reporting processes to the data warehouse
Improved reactivity for restocking requirements thanks to daily updates on stock quantity
and value for each store
Provided overall view of the status of business processes, resulting in improved user
communication
Questions
The preceding is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.