Slide 1

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 2

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 3

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 4

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 5

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 6

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 7

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 8

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 9

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 10

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 11

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 12

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 13

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 14

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 15

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 16

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 17

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 18

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 19

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 20

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 21

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 22

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 23

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 24

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 25

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 26

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 27

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 28

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 29

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 30

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 31

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 32

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 33

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 34

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 35

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 36

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 37

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 38

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 39

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 40

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 41

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 42

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 43

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 44

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 45

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 46

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 47

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 48

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 49

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 50

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 51

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 52

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 53

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 54

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55


Slide 55

ADABAS Extraction
&
Change Data Capture

Chris S. Bradley
NatQuery & NatCDC

NatWorks, Inc.
1

The Question…

Where Do You Need
YOUR
ADABAS
Data Today?

NatQuery & NatCDC

2

The Problems…
l

l

l

End-User Extraction
Data Warehouse
Extraction
Web Publishing / Data
Exchange
Adabas

DATA

End-User
Extraction
NatQuery & NatCDC

DATA

Data Warehouse
Extraction

Web Publishing /
Data Exchange
3

The Message…

A Software AG customer who has
ADABAS & NATURAL
already has the best tools to handle
all
requirements for
Data Extraction & Change Data Capture

NatQuery & NatCDC

4

ADABAS - Two Major Issues

l
l

NatQuery & NatCDC

How to Access ADABAS
ADABAS data structures

5

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls
The Only Way to communicate directly to ADABAS is through

Direct Calls

NatQuery & NatCDC

6

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls
Option #1

Option #2

Option #3

Option #4

Option #5

Imbedded

SQL / ODBC

Natural

ADABAS

Vendor

Utilities

Utilities

Direct Calls
NatQuery & NatCDC

7

l

ADABAS Data Structures
All standard data formats are supported:
ä

l

Supports two basic field types
ä
ä

l

alphanumeric, binary, fixed point, floating point,
packed decimal, unpacked decimal, ...
Elementary fields
“recurring fields” (MUs)

Consecutive fields may be grouped
ä
ä

A group may be repeated - Periodic Groups (PEs)
Periodic Groups may contain one or more MultipleValue Fields

NatQuery & NatCDC

8

ADABAS Data Structures continued
l

ADABAS has unique data structures:
ä
ä
ä

l

ADABAS has some “difficult” data types
ä

l

Multi-Valued Fields - “MUs” (array structure)
Periodic-Groups - “PEs” (table structure)
MUs in PEs (multi-dimensional structure)
IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCII
ä
ä

Packed Fields
Sign Byte Handling

NatQuery & NatCDC

9

ADABAS Extraction Facts
l

l

l

l

FACT #1
NATURAL was developed by Software AG
specifically to access ADABAS
FACT #2
NATURAL represents the most widely used AND
best understood solution for accessing ADABAS
FACT #3
What NATURAL should not be used to do, ADABAS
Utilities handle (with support from NATURAL)
FACT #4
Extraction / CDC should be done in BATCH

NatQuery & NatCDC

10

ADABAS Extraction - Conclusions
l

l

l
l

Natural allows flexible ADABAS Access
Natural easily handles all ADABAS data
structures
Natural easily handles all ADABAS data types
Natural will always work with ADABAS

Natural for maximum flexibility
ADABAS Utilities for maximum performance

NatQuery & NatCDC

11

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls
4 Embrace What Exists (ROI)
4 Best Understood Solution
4 High Performing Solution

NatQuery & NatCDC

Option #3

Option #4

Natural

ADABAS

Utilities
12

The Real Problem

NO GENERATION
Generation is needed for:
• Required Natural Objects
• Required ADABAS Parameters
• Required JCL / Script Processes

NatQuery & NatCDC

13

a

Data Extraction Solution
for

ADABAS

Leveraging Proven Technology
ADABAS Extraction
ADABAS™

ADASAV
Backup

DB2™
Predict™

Natural™

ADABAS™
Utilities

Target
of Choice

XML
Tamino™
Load Ready
Data
RDBMS
of Choice

EXCEL

NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice

15

The NatQuery Extraction Solution
NatQuery works by acting as an on-demand Natural
Programmer.
From a graphical user interface a User is enabled to:
l
l

l

l
l

l

Create Query Specifications
Generate Natural Data Extraction programs from
these Specifications
Submit Generated Extract programs for
execution
Remotely monitor execution status
Download Extracted Data
Load extracted data into MS Access, MS Excel
or convert data into XML (with optional XSL)

NatQuery & NatCDC

16

The NatQuery Extraction Solution
Internally, NatQuery can be thought of as having
three components:
l
l
l

Administrative Component
End-User Component
Generation Component
NatQuery
Administration Component
End-user
Component

NatQuery & NatCDC

Generation
Component
17

The NatQuery Extraction Solution
The Administrative Component is used by an Administrator to
capture information that is specific to the platform, environment,
and ADABAS data files that NatQuery will be used against.
NatQuery
Administration Component
End-user
Component

Generation
Component

The Administration Component provides NatQuery with the
ability to capture application-specific intelligence.
NatQuery & NatCDC

18

The NatQuery Extraction Solution
l

The End-User Component
ä

l

allows for the easy entry of extract specifications.

The Generation component
ä

generates a Natural program from a specification.
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
Files / Fields
NatQuery & NatCDC

User-specified
Select Logic

Optional
Variables

Desired
Target
19

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
Natural

Adabas

Natural
Natural
Program

User
Extract
Specification

NatQuery & NatCDC

Natural
Program

NatQuery

Workstation Environment

DATA

DATA

Access

Excel

XML

...

DATA

Other Environment
20

The NatQuery Extraction Solution
Server Environment

l

Extraction
With
ADABAS Utilities
And
Natural

User
Extract
Specification

Natural

ADACMP
Utility

Natural
Program

ADACMP
Parameters

NatQuery

Workstation Environment
NatQuery & NatCDC

Adabas

DATA

Natural
Program
DATA

DATA

Other Environment
21

The NatQuery Extraction Solution
Requests Execute in “batch”
l

l

l

Better control of requests
ä Request execution can be easily scheduled
ä Impact on online production applications are controlled
More efficient execution environment over “online”
ä Significantly Less overhead
NatQuery handles Job Control Language (JCL)
ä Template approach provides easy set-up /
maintenance
ä Dynamic substitution makes templates executable

NatQuery & NatCDC

22

NatQuery Administration Overview
1) Provide Site-Specific Job Control Language (JCL)

JCL

JCL Templates Provide:
• Integration to Natural / ADABAS
• Dynamic Process Customization
Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

23

NatQuery Administration Overview
2) Capture Natural Data Definition Modules (DDMs)

JCL
DDMs

DDMs are obtained:
• Automatically via a User Request
• Manually via an Import function

Administration Component
End-user
Component

Generation
Component

NatQuery
NatQuery & NatCDC

24

NatQuery Administration Overview
3) Capture Expanded DDM Info

JCL
DDMs

Descriptor
Statistics

File
Occurrence
Relationships Information

Administration Component
End-user
Component

NatQuery
NatQuery & NatCDC

Generation
Component

File I/O
Parameters

Predict
Metadata

Administration Information
provides NatQuery with
application-specific
generation intelligence
25

NatQuery Generation Overview
l

The End-User Component allows for the easy entry of
extract specifications.

l

The Generation Component converts an extract
specification to an optimized Natural program.
Workstation Environment
Administration Component
End-user
Component

Generation
Component

Natural
Program

End-user

Required
User-specified
Files / Fields Select Logic
NatQuery & NatCDC

Optional
Variables

Desired
Target
26

Server Integration Overview
l

File Transfer Protocol (FTP) Integration
ä
ä
ä

l

Direct FTP into batch
Indirect FTP into batch
Just FTP (Manual Execution)

Mainframe Environment

Manual Integration
ä
ä
ä

IND$FILE (IBM)
Manual FTP
Other Methods...

Workstation Environment
NatQuery

NatQuery & NatCDC

Automated
Communication
Is Achieved Using
Standard FTP
27

Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written
JES (MVS), POWER (VSE)

Request Executes
ä

l

Server Environment

Execution updates Remote Log, creates
Output

User Retrieves Output
ä

Batch
User
Request

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP
Remote
Log

Request
Output
28

In-Direct FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Server Submits Request
ä

l

Server Environment

ä

NatQuery
FTP Server

Execution updates remote log, creates output

User Retrieves Output
Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

Batch Natural

User
Request

User
Request

User
Log
Request
Output
29

Just FTP Integration Overview
l
l

User Builds Query Specification
User Submits Request
ä

ä
ä

l

Program is generated and imbedded into JCL /
Script
Program and JCL / Script is FTP’ed to the Server
Local Log File is written

Batch Natural

User Manually Submits Request
ä

l

Server Environment

Execution updates remote log, creates output

User Retrieves Output
ä

Output automatically FTP’ed to workstation

Workstation Environment

NatQuery

Local
Log
NatQuery & NatCDC

User
Request
Natural
Program

Request
Output

FTP

User
Request

User
Request

User
Log
Request
Output
30

NatQuery Integration to ETL Tools
NatQuery Generates Descriptions of Extract Layout
l

DSX Generation
ä
ä

l

DataStage Exchange file (DataStage proprietary format)
Allows for Full Integration of Predict Metadata

CFD Generation
ä

DATA

COBOL File Definition (in copybook format)

Workstation Environment
NatQuery

DSX
Files

CFD
Files
Generation
Component
NatQuery & NatCDC

SGT
Files

ETL tool

Import

31

NatQuery Features
l

Optimized Access to Source File(s), Based on UserEntered Selection Criteria
ä

Automatic determination and generation of best access method
–
–
–
–

ä
l

Descriptors, Super-Descriptors, Sub-Descriptors…
Read Logical, Read Physical, Find, Get
Zero coding effort required
Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch)

Automated Integration to Server (FTP)
ä
ä
ä
ä

Download DDMs (direct support for SYSTRANS utility)
Automatic Generation of required Descriptor Statistics
Automated Upload, Execute and Download of Results
Automated Extraction of PREDICT Meta Data

NatQuery & NatCDC

32

NatQuery Features l

Full handling of All ADABAS Field Types
ä

l

Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract Level
ä
ä

l

continued...

Conversion of ADABAS formats to ASCII equivalents
Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” data
ä
ä
ä

ä

MUs, PEs, MUs in PEs
Administratively defined defaults and maximums
User over-ride of defaults (within allowed maximums)
Ability to “Flatten” or “Concatenate” at field level

NatQuery & NatCDC

33

NatQuery Features -

continued...

“Flattening” a recurring data structure

One
Source
Record

First-Name Last-Name Address-Line City ...
Suite 100
454 South Main Street

Two occurrences

NatQuery built Index

Two
Extract
Records
NatQuery & NatCDC

Chris Bradley 1 454 South Main Street Northfield ...
Chris Bradley 2 Suite 100

Northfield ...
34

NatQuery Features -

continued...

“Concatenating” a recurring data structure
First-Name Last-Name Address-Line City ...

One
Source
Record

One
Extract
Record

Suite 100
454 South Main Street

Chris Bradley 454 South Main Street Suite 100

NatQuery & NatCDC

Two occurrences

Northfield ...

35

NatQuery Features l

continued...

Direct Integration to ETL Tools
ä

Automatic generation of interface files
– DSX and / or CFD files

l

Automatic Linking of up to 5 “Primary” files
No User Knowledge Required
Support for Automatic “Look-up” files
ä Code-to-Text conversions, Administratively defined
ä Look-ups are “transparent” to the user
ä

l

NatQuery & NatCDC

36

NatQuery Features l

Support “Full Extract” or “Intersection Set”
ä

l

Redefines, Constants, Expressions, Compress and
Dynamic (date-based and user input)

Direct Support for ADABAS utilities
ä

l

Multi-File Flexible Extraction

Abilitity to Define Variables
ä

l

continued...

ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable
Extraction

NatQuery & NatCDC

37

NatQuery Features l

Full Manipulation of Query Specifications
ä
ä

l

“Test” Extracts

Administratively Contolled User Data Limits
ä
ä

l

Save, Save As, Delete
Query specifications stored with a long and short query
description

User Specifiable Data Extract Limits
ä

l

continued...

Ability to disallow Read Physical
Ability to set Record Limits

One Tool - Dual Use
ä
ä

End-User Extraction
Data Warehouse Extraction

NatQuery & NatCDC

38

NatQuery Features l

Minimal Mainframe Footprint
ä
ä

l

Just Natural, JCL and mainframe disk space
Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source
Natural Can Talk to
ä

l

continued...

ADABAS, VSAM, DB2...

Integration to Desktop Tools
ä
ä
ä
ä

MS Access
MS Excel
XML (with optional XSL)
Tab Delimited or User-Specified Delimiters

NatQuery & NatCDC

39

NatQuery Features l

Integration to PREDICT
ä
ä

l

ä

Natural Security
ADABAS Security

Automatic Update Ability
ä

l

review I/O generation process

Full Support of Native Security
ä

l

All Field-Level PREDICT Metadata is made available within
NatQuery
Administrator can create their own

Ability to Trace I/O generation
ä

l

continued...

Allows for centralized roll-out of new versions

...

NatQuery & NatCDC

40

the

Change Data Capture Solution
for

ADABAS
NatWorks, Inc.

Leveraging Proven Technology
ADABAS Change Data Capture & Transaction Auditing
ADABAS™

ADABAS
PLOG

Natural™

ADABAS™
Utilities

Load Ready
Data
RDBMS
of Choice
EXCEL
NatQuery & NatCDC

ACCESS

Desktop
Tools

XML/XSL

Load Ready
Data

ETL Tool
of Choice
42

The Source of ADABAS Changes
ADABAS Protection Log (PLOG)

l

ADABAS’ transaction recovery mechanism
ä 100% data integrity, all transactions recorded
True “Point-in-Time” snapshot of ADABAS

l

Changed Data Available w/o ADABAS Access

l

Mainframe Environment
PLOG 2

Adabas
NatQuery & NatCDC

PLOG 1

43

ADABAS CDC (Change Data Capture)
Issues in accessing PLOG:
l
l

l

l

PLOG contains all transactions against all Files
PLOG is in compressed format
ä The same compression used by ADABAS
PLOG data is stored in Variable-Length records
ä Different from file to file and within same file
PLOG data requires “conversion”
ä EBCDIC to ASCII, date / time formats

NatQuery & NatCDC

44

ADABAS CDC
ADABAS utilities solve most PLOG issues:
l

ADASEL utility (ADABAS 6):
ä “Splits” PLOG transactions into separate files
– One File for each requested ADABAS File
– Handles Expanded Files

Decompresses PLOG records
ADACDC utility (ADABAS 7):
ä everything ADASEL does
ä direct delivery of Delta changes
ä

l

NatQuery & NatCDC

45

ADABAS CDC
NatCDC / NatQuery solves remaining issues:
l

NatCDC converts variable-length to fixed-length
ä
ä

User-Specified number of MU and PE occurrences
PLOG Header converted
– IBM STCK time, Expanded File ISNs are normalized

l

NatQuery generates all required objects
ä

All Parameters and Programs

NatQuery & NatCDC

46

NatCDC Base Components
Raw
PLOG
Data

ADASEL / ADACDC utility supplied with ADABAS
ADASEL

NatCDC

SORT

Disk

System Sort Program
Tape

NatCDC utility (Single Optimized Natural Program)

NatQuery
Workstation Environment
NatQuery & NatCDC

Server Environment
47

NatCDC Processing Overview
One JCL Stream
for each 20 files
Raw
PLOG
Data

ADASEL

Disk
Tape

One JCL Stream for each file
(or expanded file chain)

Raw
FILE
Data

NatCDC

Disk
Generated
Parameters

Tape

Fixed
Length
Data

SORT

Disk
Generated
Parameters

Tape

Generated
Parameters

Generated
Processing
Program

DWH
CDC
Data
Disk
Tape

NatQuery
DDM

Workstation Environment
NatQuery & NatCDC

Server Environment
48

NatCDC Features
l

l

l

100% Data Integrity
ä All transactions handled, even Backouts
Simple Mainframe Installation
ä One Single Natural object program (NatCDC)
ä One Natural Program for each file
ä One JCL Stream per file
The Fastest and Most Trusted Decompression
ä SAG knows their own compression the best
ä Performance is Critical
– CDC is a frequently occurring activity

NatQuery & NatCDC

49

NatCDC Features
l

Variable-Length to Fixed Length conversion
ä Final Layout is User-Determined
ä Recurring Fields Padded or Truncated
– Exception Reports Produced Automatically

l

Full Handling of all ADABAS data structures
ä MUs, PEs, and MUs in PEs
ä Automatic format translations:
– EBCDIC to ASCII
– Date and Timestamp
– Sign handling of all numeric-based fields

NatQuery & NatCDC

50

NatCDC Features
l

Full Support for Expanded Files
ä

l

Full Generation of all Required Objects
ä
ä
ä

l

Physical to Logical ISN conversion
All Parameters
All Programs
All JCL

Data is supplied with Standard Header
ä

NatQuery & NatCDC

Transaction Date, Time, ISN, Seq#, ...

51

NatCDC Features
l

l

l

Field Selection Options
ä C* values available as data
ä Fields may be selectively omitted
Integration to ETL Tools
ä CFD generation
ä “DSX” generation (Ascential DataStage)
Time Differential Handling Options
ä Automatic
ä Manual

NatQuery & NatCDC

52

NatCDC Features
l

Data Output Options
ä Logical Last
– Single Record flagged as Store, Update or Delete
ä

Logical First and Last
– One or two records flagged as Before or After

All
Extensive Reporting Options
ä Occurrence Exception Processing
ä Store, Update, Delete Counts
ä Total Before and After images
ä ...
ä

l

NatQuery & NatCDC

53

NatCDC Benefits
l

Cost Effective
ä
ä

l

One Tool - Dual Use
ä

ä
l

Uses vendor supplied utility
Uses Natural
Data Warehouse Change Data Capture
(CDC)
End-User Extraction

...

NatQuery & NatCDC

54

NatQuery & NatCDC

55