Transcript EIE 701 How To Integrate to CMDB

Atrium Integration Engine:
What Is It?
How Do You Use It?
7/17/2015
Agenda
› Atrium Integration Engine (AIE) Overview
– What is it?
– Overview of latest features
› Tell me how to do it
– Under the hood details of creating CMDB Integration using AIE
› Q&A
2
7/17/2015
CMDB-Enabled Atrium Integration Engine
API
BMC Foundation &
Topology Discovery
BMC Configuration
Discovery
Oracle
Database
SQL
Database
DB2
Database
CMDB Enabled AIE
with Adapters
•
•
•
•
•
Mapping Technology
Synchronization
Import/Export
Data Transformation
Multi threaded
Delimited
File
XML File
3
© Copyright 7/17/2015 BMC Software, Inc.
BMC Atrium Integration Engine
› What is it?
– A mechanism to manage data transfers between external data sources and
CMDB (also AR Forms)
– Bi-directional and multi-threaded data transfer engine
– An Enterprise Application Integration tool with data transformation capability
– It supports event-based or scheduled information flow
– Ideal for bulk, incremental, data synchronization transfers
– Provides interfaces to external data source and the CMDB
– Provides an intuitive graphical interface for field mapping configuration
4
© Copyright 7/17/2015 BMC Software, Inc.
BMC Atrium Integration Engine
› How is it used?
– Transform data to the CMDB data model
– Efficiently load data in bulk
– Can create relationships between instances as they are inserted in the CMDB.
› Advantages
–
–
–
–
Designed for complex integrations
Easily configurable
Can be used against a variety of applications
Extensible
› When Is AIE a Fit?
– Complex transfers requiring data transformation
– Handling large amount of data transfers
– Multiple integration points
5
© Copyright 7/17/2015 BMC Software, Inc.
BMC Atrium Integration Engine 7.1
› New Name and Repackaging for EIE
› Released in late August 2007
› Broad Enhancement Areas
– Performance and Scalability
• Improved performance of Relationship exchanges
• Better memory management for all update operations
– Usability Improvements
• Lot of usability improvements (please see the release notes for details on new features)
› Packaging
– Ships with Atrium CMDB 2.1 to integrate with Atrium CMDB
– One product, multiple adapters
•
•
•
•
Oracle
SQL
DB2
XML
– XML adapter only supports class based relationships (not vendor)
• Flat file
6
New!
New!
Improved!
© Copyright 7/17/2015 BMC Software, Inc.
How to Create AIE Based Integrations to CMDB
7
© Copyright 7/17/2015 BMC Software, Inc.
The Ingredients To Success
› Data Structure Knowledge
› External Data from Discovery/Inventory
› Atrium Integration Engine
› Atrium CMDB
› Reconciliation Rules
› Workflow Filters – if Needed
8
© Copyright 7/17/2015 BMC Software, Inc.
AIE Terminology
› Data Exchange Application (DEA)
– AR System based application at the heart of AIE
› AIE Service
– DEA passes the service instructions on how to interact with the adaptor
› Data Mapping
– CMDB Class/AR Form mapped to External Table
– CMDB Attributes (or fields of AR Form) mapped to Columns of External Table
– Values mapped to External Table Columns
Data
› Exchange
– Collection of “similar” Mappings
– Definition of when to run
– “Pipe that manages the flow”
› Adapter
– The Plug at the end of the wire
9
© Copyright 7/17/2015 BMC Software, Inc.
AIE
Service
Adapter
Exchange
Application
Data
Exchanges
Data
Mappings
Configuration SQL Example
Discovery Environment
Remedy Environment
AR System
Discovery
SQL Database
AIE
Service
Data
Exchange
Application
Adapter
Data
Exchanges
(can be on either server)
Data
Mappings
CMDB
Discovery
Dataset
CMDB
Reconciliation
Engine
CMDB
BMC.ASSET
Dataset
Data
Cleansing
Workflow
CMDB
Reconciliation
Rules
Remedy
Asset
Management
Database for AR System
10
© Copyright 7/17/2015 BMC Software, Inc.
External Data Source
› Database data
› XML data
› Flat file data
› Source Relationships can be stored as
– foreign key
– relationship tables
11
© Copyright 7/17/2015 BMC Software, Inc.
Atrium CMDB
› Configuration Management Database
–
–
–
–
Store multiple datasets
Reconcile datasets
Store Relationships
Base data for OOTB Applications
› Data Sets
– Data value in a field... but so much more
– Data Sets you should use
• BMC.ASSET – The “Gold Standard”
• BMC.SANDBOX – Temp location for GUI changes to Assets
• VENDOR.PURPOSE.PRODUCT – custom datasets
– Examples: BMC.IMPORT.MARIMBA, MS.IMPORT.SMS, ACME.IMPORT.LEGACY
12
© Copyright 7/17/2015 BMC Software, Inc.
AIE Structure Simple
Data Exchange
On Complete, Trigger Next
1-N
Data Mapping
Relationship Exchange
1-N
Relationship Mapping
13
© Copyright 7/17/2015 BMC Software, Inc.
AIE Structure Complex
Data Exchange
Trigger Next
Data Mapping
Data
Mapping
Computer
Data
Mapping
Computer
Computer
Data Exchange
Trigger Next
Data Mapping
Operating System
14
Data Exchange
Data Exchange
Trigger Next
Data Mapping
Drives
Trigger Next
Data Mapping
Monitor
Data Exchange
Trigger Next
Data Mapping
Software
Relationship Exchange
Relationship Exchange
Relationship Exchange
Relationship Exchange
Relationship Mapping
Computer – O/S
Relationship Mapping
Computer - Drives
Relationship Mapping
Computer - Monitor
Relationship Mapping
Computer - Software
© Copyright 7/17/2015 BMC Software, Inc.
AIE Main Console
15
© Copyright 7/17/2015 BMC Software, Inc.
Configuration > Database Field Menus
› Populates the “vendor” table and column menus
› Run before you start any mapping work
1. Load
3. Create Menus
2. Add Desired Tables
16
© Copyright 7/17/2015 BMC Software, Inc.
How To – Data Mapping
›
›
›
›
›
›
›
›
17
Mapping Name = name for new mapping
Direction = External Data into CMDB
Namespace = BMC.CORE in most cases
Class = Destination for data, pick from the list
External Data Store = SqlServer, Flatfile or Oracle
Table Name = Source of data (pick from list)
– if not in the list, run rulehelper.exe to create the menu items.
Row Element = (XML only) the element that represents a row
Click “Create” to save
© Copyright 7/17/2015 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
If the Table Name does not show in the drop down list,
it is because you have not configured the Database
Field Menus from the main console
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
18
© Copyright 7/17/2015 BMC Software, Inc.
How To – Data Mapping
› Used to map source and destination side primary/foreign keys
› Keys must be unique on both sides
– If multiple keys, stored as AND to define Unique
19
© Copyright 7/17/2015 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
› Map appropriate CDMB Class.Attribute to a Data Store Attribute
› Make sure to specify CMDB mandatory fields such as
–
–
–
–
–
–
20
Fields use in Key mappings
Data Set ID (automatic if “Use Dataset” is checked)
Short Description
Submitter
Name = anything that identifies the entry
AssetLifeCycleStatus = 3 (if Asset Management is installed)
© Copyright 7/17/2015 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
› Rules
– Works like “Function” key in Remedy User and Administrator Tools
– Select by highlighting the row and click “Add”
– Rule is added
– Refer to the Rules box for hints on how to complete the rule
› Available Rules
21
© Copyright 7/17/2015 BMC Software, Inc.
How To – Data Mapping
› Only works if you can write to the Vendor / Source Database
› Can be any value
› Usually is completion status
Data Exchange
– 0 = transfer with no error
– N = error # - corresponds to ARERROR
22
© Copyright 7/17/2015 BMC Software, Inc.
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
› If you “Update Row to Indicate Delete” i.e. Soft Delete
– Set MarkAsDeleted to 1 (Yes)
– Other attributes may be set as desired
› If you choose to “Permanently Delete” – Hard Delete
– No field to set. AIE will do trigger the deletion
23
© Copyright 7/17/2015 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
- Use to limit amount of data in a given
transfer. Important if you are experiencing
performance or memory issues
- If you specified “Restrict to Dataset” on
the Main Tab, you don’t need to include it
here.
24
© Copyright 7/17/2015 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
1
2
3
4
5
6
1.
2.
3.
4.
5.
6.
If set to all data that can be transferred will be. This could result in partial data records.
Transfer as NULL cleans up any data that might be mistaken for null in a field
If trailing blanks are possible in your key values, choose to consider
Stores change history in a diary field you designate. Attribute must exist in CMDB.
If Yes, updates are only made when a change is detected in the source data
› You must select a field that can store a negative integer to store the checksum
Data Exchange
Allows you to retrieve new or change data since last transfer
› You must select a date/time field to store last transfer time
Relationship Exchange
Data Mapping
Relationship Mapping
25
© Copyright 7/17/2015 BMC Software, Inc.
How To - Data Exchange
›
›
›
›
›
Exchange Name – Name you create to identify the exchange
Instance Name – Specify the AIE thread to use
Data Exchange Type – Specify the type of mapping contained in the exchange
External Data Store – Must match the mapping(s)
Data Exchange
Execute Exchange As
Data Mapping
– Schedule Only
• Use Schedule tab
Relationship Mapping
– Event Driven Only
• run eiexfer.exe from command line, filter or active link
26
Relationship Exchange
© Copyright 7/17/2015 BMC Software, Inc.
How To - Data Exchange
› Select the mapping(s) to include in the Exchange container.
› Multiple Mappings
›
27
– Cuts down on queries for source or target
Main Mapping
– Need to designate if multiple mappings included
– Key values only come from Main Mapping
© Copyright 7/17/2015 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Data Exchange
› Interval
– Works like the interval function in escalation workflow
› On Completion Section
Data Exchange
– When this exchange is done, you can configure
another exchange to run
28
© Copyright 7/17/2015 BMC Software, Inc.
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Data Exchange
› Time
Data Exchange
– Works like the time function in escalation workflow
– Set day and time to run
29
© Copyright 7/17/2015 BMC Software, Inc.
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Data Exchange
› To Be Triggered
– Use this option when you want to exchange to run on
completion of some other exchange
– Must list the exchange in the On Completion section
on the other exchange
30
© Copyright 7/17/2015 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Data Exchange
Use if the AR Server involved in
the exchange is NOT the server
that DEA is installed on set to
“No” and fill in the info.
Advantage – 1 license/DEA can
service Dev, Test & Production
You can either set the
external connection
information on each
exchange or on the
Connections link from
the Data Exchange
Console
31
Alternate Method
© Copyright 7/17/2015 BMC Software, Inc.
How To - Data Exchange
› While Testing
– Limit records so you have less to undo
› In Production
Data Mapping
– Set Enable Debug to No
32
Data Exchange
Relationship Exchange
Relationship Mapping
© Copyright 7/17/2015 BMC Software, Inc.
How To – Relationship Mapping
V_Computer
BMC_ComputerSystem
Data Mapping
BMC_DiskDrive
V_DiskDrive
Data Mapping
Vendor Relationship Table
BMC_HostedSystemComponent
Vendor Relationship Mapping
•Map using External Data Store Mapping.
•Trigger on Complete of Data Exchange.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
33
© Copyright 7/17/2015 BMC Software, Inc.
How To – Relationship Mapping
BMC_ComputerSystem
V_Computer
Data Mapping
BMC_DiskDrive
V_DiskDrive
Data Mapping
Class
Relationship Mapping
•Map using CI Class Relationship Mapping
•Put in an Event Driven Only Exchange*
BMC_HostedSystemComponent
*In this case, the relationship data
exchange only runs after the data
transfer for the two configuration
item classes has completed.
34
© Copyright 7/17/2015 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Relationship Mapping
› This is where you define which 2 classes to relate and which relationship
› Must Select Relationship Source on Create
› External Data into CMDB is the only valid Direction
› If you choose Relationship Source of “External Data Store”
Data Exchange
– must specify DB2, FlatFile, Oracle, SqlServer
– Table Name
› It is best to Restrict to DatasetID and specify
35
© Copyright 7/17/2015 BMC Software, Inc.
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Relationship Mapping
› Define mapping criteria that defines the relationship
› Works just like creating Relationship in CMDB
– Source on Left, Target on Right
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
36
© Copyright 7/17/2015 BMC Software, Inc.
How To - Relationship Mapping
Data Exchange
› Used to define values for a relationship instance
› Mandatory attributes shown above must be mapped
› To get a list of Attributes of each CI Class choose class from drop down.
Data Mapping
Attributes list will then appear
37
© Copyright 7/17/2015 BMC Software, Inc.
Relationship Exchange
Relationship Mapping
How To - Relationship Mapping
Data Exchange
› Used to define values for a relationship instance
› Mandatory attributes shown above must be mapped
› To get a list of Attributes of each CI Class choose class from drop down.
Data Mapping
Attributes list will then appear
38
© Copyright 7/17/2015 BMC Software, Inc.
Relationship Exchange
Relationship Mapping
How To - Relationship Mapping
› If you “Update Row to Indicate Delete” i.e. Soft Delete
– Set MarkAsDeleted to 1 (Yes)
– Other attributes may be set as desired
› If you choose to “Permanently Delete” – Hard Delete
– No field to set. AIE will do trigger the deletion
39
© Copyright 7/17/2015 BMC Software, Inc.
How To - Relationship Mapping
› Use if you are doing class based mapping
› Should match query used on class mapping
› If you specified “Restrict to Dataset” on the Main Tab, you
don’t need to include it here.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
40
© Copyright 7/17/2015 BMC Software, Inc.
How To - Relationship Mapping
› Use if you are doing class based mapping
› Should match query used on class mapping
› If you specified “Restrict to Dataset” on the Main Tab, you
don’t need to include it here.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
41
© Copyright 7/17/2015 BMC Software, Inc.
How To - Relationship Mapping
› Use if you are doing class based mapping
› Should match query used on class mapping
› If you specified “Restrict to Dataset” on the Main Tab, you
don’t need to include it here.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
42
© Copyright 7/17/2015 BMC Software, Inc.
How To - Relationship Mapping
› Used only if a External Data Store mapping
› This is used to limit the external records
returned
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
43
© Copyright 7/17/2015 BMC Software, Inc.
How To - Relationship Exchange
› Works the same as a normal Data Exchange
› Data Exchange Type = Relationship Class
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
44
© Copyright 7/17/2015 BMC Software, Inc.
General AIE Hints
› When testing an exchange –
–
–
–
–
Test one exchange at a time (when done chain them to be triggered)
Use Run Now button to trigger an exchange immediately
Set Limit = 1 (so that you only run once)
Be sure you know your debug path
› When writing a Vendor Query –
– Test it with Direct SQL – If it doesn’t work there, it won’t work here
– Use Indexed fields if possible
This won’t work – MacAddress != NULL
This will – MacAddress IS NOT NULL
45
© Copyright 7/17/2015 BMC Software, Inc.
General AIE Hints
› When setting an AR System field in a mapping –
– Doesn’t work like a setfield or pushfield operation
– Must Use Rules to convert functions, keywords, etc.
Example of an Integer Field
This won’t work – 0
This will – Constant|0
Example of Concatenation
This won’t work – $Manufacturer0$+”-”+$MediaType0$
This will – function|concat($Manufacturer0$,”-”,$MediaType0$)
46
© Copyright 7/17/2015 BMC Software, Inc.
Atrium CMDB
› Reconciliation Rules
CMDB
3rd Party
Dataset
– Identify
CMDB
Master
Dataset
• Look for potential duplicates
– Compare
– Merge
– Cleanup / Purge
CMDB
Reconciliation
Engine
• Can delete entire data set from all tables
• Can delete selected items
– Configuration Item Set field MarkedForDelete = Yes
– Create Reconciliation Rule to Purge if field is set
» Define how to handle related items
– Run as a Reconciliation Job
47
© Copyright 7/17/2015 BMC Software, Inc.
CMDB
Reconciliation
Rules
How Atrium Integration Engine is Acquired
48
© Copyright 7/17/2015 BMC Software, Inc.
BMC AIE is bundled with
BMC Service Management Products
– AIE comes bundled at no additional cost with BMC applications which ship
BMC Atrium CMDB 2.0.01 onwards, such as:
•
•
•
•
•
•
•
Service Desk
Change Management
Asset Management
Service Request Management (SRM)
Configuration Management *
Service Impact Management *
Topology Discovery, Foundation Discovery, and Identity Discovery *
* AR with 3 user licenses are also bundled (paper license) limited to support the CMDB (and DSL)
– Existing ITSM customers can upgrade to v7.0.02 and later versions to get AIE
at no cost to integrate with BMC Atrium CMDB only
– Customers need to purchase AIE if it is used to load data into AR Forms
directly instead of BMC Atrium CMDB
– AR User licenses would be needed to use AIE Application
49
© Copyright 7/17/2015 BMC Software, Inc.
BMC AIE Licensing
– If AIE did not ship with a BMC product they can download it through EPD
• For customers using AIE to load data into BMC Atrium CMBD (2.0.01 onwards)
there is no charge
• For customers using AIE to load data into any AR System form directly, they would
need to purchase AIE
– AIE is our recommended approach for mapping 3rd party data sources or
discovery tools to the BMC Atrium CMDB common data model.
• BMC Atrium CMDB customers can use our open API interfaces and/or AIE at no
charge to integrate applications
• Other APIs don’t include data transformation and syncing technology that is
available through AIE.
50
© Copyright 7/17/2015 BMC Software, Inc.
How to Deal with AIE and adapter legacy customers
› If customer bought Enterprise Integration Engine (EIE)
– They now own AIE. They can upgrade to AIE 7.1 seamlessly.
› If customer bought Remedy Link for SQL or Remedy Link for Oracle
– Remedy Link for SQL and Oracle has been withdrawn with the release of AIE 7.1
– For use with CMDB integration they are entitled to AIE at no extra charge (upgrading
to latest ITSM/SIM/TD might be needed)
– No product migration is being offered for these link products
– If a purchased copy of AIE is needed (to load data into AR System Forms directly)
• Customers would need to purchase AIE and pay the difference
• Maintenance cost would be based on AIE’s base price
› If customer bought Remedy Discovery Services Adapter for SMS
–
–
–
–
Remedy Discovery Service for SMS has been withdrawn with the release of AIE 7.1
For use with CMDB integration they are entitled to AIE at no extra charge
They can use SMS2CMDB Open Source Integration for the mappings.
If a purchased copy of AIE is needed (to load data into AR System Forms directly)
• Customers would need to purchase AIE and pay the difference
• Maintenance cost would be based on AIE’s base price
51
© Copyright 7/17/2015 BMC Software, Inc.