Transcript About the Speaker

Programming Integrations Between Vault and
Other Systems
Doug Redmond
Software Engineer, Autodesk Inc.
© 2012 Autodesk
About the Speaker
Doug Redmond has been a software engineer
for Autodesk for over 12 years. And has been on the
Vault team for over 8 year.
In addition to being the Vault API expert, he has worked as development
lead on several integration projects including Vault/SharePoint and
Vault/PLM 360.
Doug is the author of “It’s All Just Ones and Zeros” which is a blog on
Vault API development. Check it out at
http://justonesandzeros.typepad.com/
© 2012 Autodesk
Class Overview
This class a problem:
Every integration is different. I can’t go over everything there is to know
about integrating Vault with every other possible system. Even if I knew
how, I couldn’t go over it all in 90 minutes.
Solution:
Writers are supposed to write what they know. So I will stick to
integrations that I know. This class will cover 3 Vault integrations that I
either wrote myself or served as development lead.
© 2012 Autodesk
Learning Objectives
At the end of this class, you will be able to:

Know at least 3 architecture patterns for building an integration.

Balance the pros and cons of an integration architecture.

Know the scalable and reliable Vault features you can use as a basis for
an integration.

Understand how the SharePoint 2010 and PLM 360 integrations work.
© 2012 Autodesk
Overview

Level of expertise:

How many are professional programmers?
 How many are non-professional programmers (scripters)?
 How many have no programming experience?
This class will go into technical details, but not into source code.
Handout contains extra technical content.
© 2012 Autodesk
Overview

Introduction
 Case Studies

SharePoint 2010 (40%)
 Vault ♥ Bluestreak (10%)
 PLM 360 Sync (50%)

Q&A
© 2012 Autodesk
Case Study: SharePoint 2010 Integration
© 2012 Autodesk
SharePoint 2010 Integration

Goals


Challenges



Integrate Vault and SharePoint 2010
“Shrink-wrap” style application
No customer feedback
Bonus

Partnership with Microsoft
© 2012 Autodesk
SharePoint 2010 Integration

Feature Set




Display Vault files, items and change orders as SharePoint lists.
Ability to use SharePoint search to display Vault files, items and change
orders.
Ability to use Vault user-defined properties.
Limitations



Read-only in SharePoint
Common Vault login used
On-premise SharePoint only
© 2012 Autodesk
SharePoint 2010 Integration - Demo
© 2012 Autodesk
SharePoint 2010 Integration
The lists are created using the BCS (business connectivity services)
framework in SharePoint.

BCS allows you to take an object from another system and represent it
as a SharePoint object.
 BCS objects are created at runtime. There is nothing stored or
indexed in SharePoint.
 BCS can be referenced by other SharePoint objects. For example, you
can use SharePoint workflow.
 BCS was formerly known as BDC.
© 2012 Autodesk
SharePoint 2010 Integration
Architecture pattern: System X displays a view of system Y data
Pros:
 Data exists in only one system
 Data is always up to date
Cons:
 Performance
 Requires working network
© 2012 Autodesk
SharePoint 2010 Integration - Architecture
BCS
Vault
Connector
SharePoint
Vault API
Vault Web
Services
Vault
© 2012 Autodesk
SharePoint 2010 Integration
Order of operations:
1. SharePoint user views a list in their web browser.
2. BCS invokes Vault Connector asking for object set.
3. Vault Connector makes Vault API calls.
4. Vault Connector formats Vault data as SharePoint objects and
passes data back to BCS.
5. SharePoint formats the data as a list view.
© 2012 Autodesk
SharePoint 2010 Integration
My advice for working with BCS:

Write a custom connector. The BCS feature built-in to Visual Studio
2010, was very good in my opinion.

Data limit is required. BCS has no idea what the page layout is. So it
always asks for the full data set each time. What is displayed to the
user is usually a subset of that data.

Caching is required. Many user operations result in BCS asking for the
full data set again.
© 2012 Autodesk
SharePoint 2010 Integration
Principle: Worst of Both Worlds
If a limitation exists in system X but not system Y, the integration
inherits the limitation.
If a useful feature exists in system X but not system Y, the integration
cannot make use of the feature.
In other words, the integration represents the worst of both worlds.
© 2012 Autodesk
SharePoint 2010 Integration
Working with Vault:

Search is the most optimal way to get data from Vault. It’s flexible,
scalable and performant.



API calls: FindFilesBySearchConditions, FindItemsBySearchConditions, etc.
Not used: GetLatestFilesByFolderId, GetLinksByFolderId
BCS object is a property set from Vault.

API call: GetProperties
 Not used: File object, Item object, ChangeOrder object
© 2012 Autodesk
SharePoint 2010 Integration
Additional components:
 BDCModelEditor – BCS uses a fixed data model. So we created a
utility that allows you to select which Vault properties make up the data
model.
 Admin pages – ASP.NET page was created to configure admin
settings. Also a page to create new lists with proper formatting.
 Thumbnail image library – Since thumbnails take a long time to transfer
from Vault, we use a SharePoint library to store thumbnail data.
 Deployment scripts – Components need to be installed to SharePoint
in a specific way. BAT files were provided to help the process.
© 2012 Autodesk
SharePoint 2010 Integration - Search
Displaying Vault data in SharePoint search does not use BCS. A new
mechanism was created based on the Federated Search feature in
SharePoint.
In SharePoint, federated search means that the results are coming from
outside the SharePoint farm. Again no data is being stored or indexed
in SharePoint. All data is pulled live from Vault.
© 2012 Autodesk
SharePoint 2010 Integration - Architecture
Search Results page
Federated results
Web part
OpenSearch
XML
Federation location
ASP.NET page
SharePoint
Vault API
Vault Web
Services
Vault
© 2012 Autodesk
SharePoint 2010 Integration - Architecture
If I had to do it again….
Search Results page
Vault API
Web Part
SharePoint
Vault Web
Services
Vault
© 2012 Autodesk
SharePoint 2010 Integration
What’s next?
It’s up to you!
Let us know what you want.
Vault Idea Exchange:
http://forums.autodesk.com/t5/Autodesk-Vault-Idea-Exchange/idb-p/2
© 2012 Autodesk
Vault ♥ Bluestreak
© 2012 Autodesk
Vault ♥ Bluestreak
What is Bluestreak?

A project up on Autodesk Labs.

Applies social media principles to AEC project.

Features:



Users and Groups
Activity streams
File storage
© 2012 Autodesk
Vault ♥ Bluestreak

Goals


Challenges



Integrate Vault and Project Bluestreak
Almost no customer feedback
No internal feedback
Bonus

Experimental app. Not a real product.
© 2012 Autodesk
Vault ♥ Bluestreak

Feature Set

File releases, Item releases and Change Order state changes are sent to
Bluestreak activity stream.
 Vault files can be uploaded to Bluestreak

Limitations

User must explicitly log in to Bluestreak
© 2012 Autodesk
Vault ♥ Bluestreak - Demo
© 2012 Autodesk
Vault ♥ Bluestreak
Architecture pattern: System X pushes data to system Y in real-time in
response to user actions.
Pros:
 Real time updates
 User interaction
Cons:
 Delivery not guaranteed
 Client app needs to be installed
© 2012 Autodesk
Vault ♥ Bluestreak
Working with Project Bluestreak:

It’s a closed API. So there is no point in going over these details.
© 2012 Autodesk
Vault ♥ Bluestreak
Working with Vault:

The app makes use of the Vault Explorer extension and event handling
features of the API.
 Web service events are used to gather up changes and Vault Explorer
command events are used to determine when to update the activity
stream.
 Communication with Bluestreak happens in a worker thread. This way
the sync is invisible to the user; they can still continue working in Vault.
© 2012 Autodesk
Vault ♥ Bluestreak – Future Plans
I have no plans to maintain this app.
It generated a lot of attention internal to Autodesk, but not from outside
the company.
Lessons learned from Project Bluestreak are being applied to the
Autodesk cloud offerings. Likewise the lessons learned from this tool
may be applied to future Vault/cloud integrations.
© 2012 Autodesk
PLM 360 Sync
© 2012 Autodesk
PLM 360 Sync

Goals



Integrate Vault and PLM 360
Release at the same time as Vault 2013 and PLM 360 launch
Challenges
“Shrink-wrap” style application
 No customer feedback
 Integral to product strategy


Bonus

Can add new features into both systems
© 2012 Autodesk
PLM 360
Design pattern: R1 is the Starting Point
If you give a customer a blank page and ask them to tell you want they
want, you will be left with a blank page.
If you release R1 and ask customers what they think, you will get a lot
of good feedback.
Decisions requiring feedback are kept to a minimum in R1.
© 2012 Autodesk
PLM 360 Sync

Features



Push BOM data from Vault to PLM 360
“Seamless” integration
Limitations

BOM data only
 Data flow is one-way
 PLM cannot open a connection to Vault
© 2012 Autodesk
PLM 360 Sync - Demo
© 2012 Autodesk
PLM 360 Sync
Architecture pattern: System X integrates with system Y
asynchronously using a queue
Pros:
 Guaranteed delivery
 Failure recovery mechanism
Cons:
 Conflicts may occur
 Operations that span both systems can’t be done in real time
 Duplicate data
© 2012 Autodesk
PLM 360 Sync - Architecture
Business
Logic
Vault
Job Queue
Vault API
PLM API
PLM 360
Sync
Service
PLM 360
Vault Server Process
Firewall
© 2012 Autodesk
PLM 360 Sync
New Vault feature: Change notification Jobs

When the sync service starts up it tells Vault to log all File and Item changes to the
job queue. When the service shuts down, the service tells Vault to stop logging
changes.

All new and edit operations are logged, even if it’s not relevant to the sync tool.

Deletes are not logged.

You only need to enable notifications on a single site/workspace. Replicated data is
handled automatically.
© 2012 Autodesk
PLM 360 Sync
Undocumented API function: JobService.EnableChangeObservation

Summary: Enables or disables the feature where Vault logs item and file
changes to the job queue.

Parameters:
 string type – When jobs are added to the queue, they will have this type.
 string desc – When jobs are added to the queue, they will have this
description.
 bool enabled – If true, it enables the feature. If false, it disables the feature.
Note: Because the function is undocumented, there is no official API support.
© 2012 Autodesk
PLM 360 Sync
New Vault feature: Persistent IDs





The Vault API makes no guarantee about the long-term lifetime of IDs like
File.Id, File.MasterId, Item.Id and so on. They may change when Vault
upgrades to a new release.
Persistent ID feature was added to provide a unique ID that is good forever.
When objects are created in PLM, the persistent ID is used to tie it to the Vault
object.
Was designed for files and items only, but will probably work with other entity
types.
Persistent ID is unique for a given Vault. Sync tool concatenates this with a
Vault GUID to create an ID unique across multiple Vaults.
© 2012 Autodesk
PLM 360 Sync
Undocumented API function: KnowledgeVaultService.GetPersistentIds

Summary: Gets the persistent IDs for a set of entities.

Parameters:
 string entityClass – The type of entity.
 long [ ] entityIds – An array of entity ID values.
 EntPersistOpt persistOpt – If History, you get an ID for a specific version of
an entity. If Latest, you get an ID that always points to the latest version.

Returns string array representing the persistent IDs of the entities.
Note: Because the function is undocumented, there is no official API support.
© 2012 Autodesk
PLM 360 Sync
Undocumented API function: KnowledgeVaultService.ResolvePersistentIds

Summary: Gets a set of entities based on their persistent IDs.

Parameters:
 String [ ] persistantIds – An array of persistent IDs.

Returns an Ent array showing the entity ID and entity class ID values.
Note: Because the function is undocumented, there is no official API support.
© 2012 Autodesk
PLM 360 Sync - Architecture
Vault
Server
Vault
Job Queue
Vault API
PLM API
PLM 360
Sync
Service
PLM 360
Vault Server Process
Firewall
© 2012 Autodesk
PLM 360 Sync

Sync Service

All the sync operations are handled by the Sync Service
 The Sync Service works like the Job Processor in that it polls the job queue.
 The Sync Service is different in that it is a true Windows service.
Vault trick: Job handlers do not have to be written as a Job Processor extension.
API functions used: ReserveNextJob, UpdateJobFailure, UpdateJobSuccess
© 2012 Autodesk
PLM 360 Sync
PLM 360 Sync Service – order of operations:
1. Wake up
2. Check the job queue for jobs of our type
3. For each job
1.
4.
For each PLM connection
1. Determine if this is a change that needs to be synced
2. If so, update PLM 360 to match the Vault data
Go back to sleep
© 2012 Autodesk
PLM 360 Sync
What happens during a sync:

We sync by BOM, not by Item. PLM requires children to be added
before parents.

Revision history is part of the sync. PLM requires revisions to be
added in order.

I broke the sync into smaller operations and used a stack to insure that
operations occur in the correct order.
© 2012 Autodesk
PLM 360 Sync
If the goal is to sync X rev C,
the operation stack may look
like this.
Item X
(rev C)
Item X
(rev B)
Item Y
(rev B)
Item X
(rev A)
Item Y
(rev A)
Item Structure in Vault
Item Y:A
Item X:A
Item Y:B
Item X:B
Item X:C
Operation stack
© 2012 Autodesk
PLM 360 Sync
Order dependency and redundancy:
 Jobs may show up in the queue in any order.
 PLM may be up-to-date for all or part of the operation.
Solution: Each operation check with PLM to see if updates are needed.
© 2012 Autodesk
PLM 360 Sync
Stack approach still works,
but some operations do
nothing.
Item X
(rev C)
Item X
(rev B)
Item Y
(rev B)
Item X
(rev A)
Item Y
(rev A)
Glowing items already
exist in PLM.
Item Y:A - noop
Item X:A - noop
Item Y:B - noop
Item X:B - sync
Item X:C - sync
Operation stack
© 2012 Autodesk
PLM 360 Sync
PLM aspects:
 Each operation is autonomous
 No way to transact the entire sync
 As a sync is happening you can see the BOM being built in PLM
 Failure may happen in the middle of a sync
 No rollback support
Solution – Mark job as error. Job can be re-run at a later time.
© 2012 Autodesk
PLM 360 Sync
Item X
(rev C)
Network goes down mid way
through.
Item X
(rev B)
Item Y
(rev B)
Error
Item X
(rev A)
Item Y
(rev A)
Item Structure in Vault
Item Y:A - done
Item X:A - done
Item Y:B - done
Item X:B
Item X:C
Operation stack
© 2012 Autodesk
PLM 360 Sync
Network is back online.
Re-run the job and the sync
tool can continue where it left
off.
Item X
(rev C)
Item X
(rev B)
Item Y
(rev B)
Item X
(rev A)
Item Y
(rev A)
Glowing items already
exist in PLM.
Item Y:A - noop
Item X:A - noop
Item Y:B - noop
Item X:B - sync
Item X:C - sync
Operation stack
© 2012 Autodesk
PLM 360 Sync - Configuration
Configuration is done through a custom command UI in Vault Explorer.
The configuration contains the following information:
 The PLM 360 login details.
 Details on where the Item data should go.
 Details needed to revision the PLM data.
 Property mapping between Vault and PLM fields.
 The set of Vault items to sync.
© 2012 Autodesk
PLM 360 Sync - Configuration
Determining the Item set.
Rejected approaches:
 Set an Item property. Can’t edit a released item.
 Create folder links and sync everything in the folder. Too time
consuming.
 Explicit command. Can’t keep up with new items.
© 2012 Autodesk
PLM 360 Sync - Configuration
Determining the Item set.
Solution:
 Determine the set through an Item search

Flexible
 Automatic
 Bonus: By using saved searches, we didn’t have to re-write search dialog.
© 2012 Autodesk
PLM 360 Sync
Feature: When duplicating data between systems, an integration
should provide a ‘Resync’ command.
Pros:
 Satisfies a bulk-load requirement
 Provides a way to fill in gaps that might arise in failure cases
 Great tool for testing and debugging
Cons:
 May take a while to run for large data sets
© 2012 Autodesk
PLM 360 Sync - Configuration
Things added by the PLM team:
 An API – See Rest API page on PLM 360 wiki.
 Override revision locks permission – Allows us to edit history.
 Lifecycle changes without a revisioning item.
© 2012 Autodesk
Wrap up
© 2012 Autodesk
Wrapup

Check your handout for technical information:

Bulk-load techniques

Item Service vs. Package Service

Item-File association types

Useful links
© 2012 Autodesk
Related Classes

PL3738 - Above and Beyond: Autodesk Vault and Microsoft SharePoint
Wednesday, 5:00-6:30

PL2734 - Where Does PDM Stop and PLM Start?
Thursday, 8:00-9:30

CP1962 - Programming Custom Objects with the Autodesk Vault API
Thursday, 1:00-2:30
© 2012 Autodesk
Autodesk, AutoCAD* [*if/when mentioned in the pertinent material, followed by an alphabetical list of all other trademarks mentioned in the material] are registered trademarks or trademarks of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and
services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document. © 2012 Autodesk, Inc. All rights reserved.
© 2012 Autodesk