Transcript PDMWorks Enterprise Data Loading Strategies

ENOVIA SmarTeam
Data Loading Strategies
Marc Young, CEO
xLM Solutions, LLC
Strategies
ease the
migration
pain…
With
diligent
planning
and
thorough
testing,
Loading,
or to
migrating,
data
into ENOVIA
the process
cana go
smoothly
SmarTeam
can be
daunting
task…
1. Pre-Migration Data Analysis
•
•
•
•
Data Clean-up and Process Review
Data Formats
Data Loading Rules
Migration Cost Estimation
2. Data Loading Options
• Bulk Loader
• Import Tools
• Custom Scripts
3. Execution
•
•
•
•
Testing the Migration Process
Data Verification
Pre-Data Load Planning
Production Data Load
Analyze the Data to be
Loaded
Data Analysis
Determine
legacy data
nature and
location
• File-/Folder-based
• Legacy systems
(databases, vaults,
etc.)
• Hard copy (e.g.,
aperture cards)
• Type of data: CAD
(with internal links),
office, etc.
Anticipate data
access issues
• All sources
available?
• Third-party systems
accesses?
• Additional experts?
• Additional
resources?
Is CAD data
conversion
needed?
Data volume and
estimated time
to load
• e.g., ProE → CATIA • Will/Can downtime
be scheduled for
• V4 → V5
data loading?
Analysis: Data Clean-Up
Analysis: Data Clean-Up
What types of data clean-up are
required?
•
•
•
•
•
•
•
Duplicated part numbers per file
Duplicated file names
Missing or incorrect file attributes
Missing or incorrect file associations or links
File revision not current
Non-current parts in current assemblies
Non-compliance or old revision/versioning
scheme
Validate data accuracy
Analysis: Process Reviews and Updates
• An opportunity to review business
processes
• Opportunity to evaluate and update user
experiences
• Important: new changes to the
underlying data model must be verified
with the migration processes
Analysis: Data Formats
CAD Data
• File-based
• Typically has relationships among files
• Attributes and properties contained within the file
Meta Data to be Loaded or Added
• Text description of data external to the associated (CAD) file
• Users
• Import from existing Active Directory
• Lookup Lists
• Projects
• File location pointers
Combined Data
• CAD data with embedded relationships; but…
• Attributes and properties reside outside of the CAD file
• e.g., in an Excel spreadsheet
• Construct a link between the CAD file and attribute data
Analysis: Determine the Data Loading Rules
What are the rules to determine
where the data goes?
• Should the data be located in a folder?
• Should a folder be automatically
created for the data?
How will data fields be mapped
• Text, Lookup, Reference To Class, URL
from legacy system to SmarTeam?
What state(s) will the data be
loaded to?
Will the data be linked to other
SmarTeam data?
• If so, what is the ruleset?
How to handle the cases with
missing data?
a full data history is desired,
Which data revision to load: latest • Ifdetermine
how the revision history is
or full history?
maintained in the legacy system
Analysis: Effort and Cost Estimations
Time (data
loading tools
development
and testing)
Impact of
downtime
for users
Data Load
Project Cost
Quantification
Resources
involved
Price
1. Pre-Migration Data Analysis
•
•
•
•
Data Clean-up and Process Review
Data Formats
Data Loading Rules
Migration Cost Estimation
2. Data Loading Options
•
•
•
•
•
Manual Processing
SmarTeam Import and Export Tools
Custom Scripts (executables, add-ins, etc)
Combined Manual Processing and Custom Scripting
Pre-Developed Data Migration Programs
3. Execution
•
•
•
•
Testing the Migration Process
Data Verification
Pre-Data Load Planning
Production Data Load
Overview SmarTeam Bulk Loading Tool
• Load CAD File data that
SmarTeam has integration for
• CATIA, SolidWorks, SolidEdge,
Inventor, AutoCAD
• Highlight a set of files that should
be loaded into SmarTeam
• The bulk loader will open each file
in the CAD application one at time
and perform a SmarTeam Save
• Recommended way to bulk load
standard parts
Setup for SmarTeam Bulk Loading Tool
• CAD Integration should be configured
• Associated CAD File Type to SmarTeam Class
• Map CAD Attributes to SmarTeam field
• Sequence needs to be assigned to the primary Id
• Batch Mode Save Option should be turned on
• Otherwise, you will have to hit okay for each file to
be loaded.
• Default Class to be specified
Issues with SmarTeam Bulk Loading Tool
• Speed – an open and save performed on all files being loaded
• CAD data integrity – If links are bad/missing in the CAD data:
• the files will not be saved
• may require user intervention per occurrence
• Attribute data needs to exist in CAD properties*
• No log created for data load*
• Difficult if CAD data is to be associated with more than one class*
• An Engineering Part, Manufacturing Part, Standard Part
• All data is linked to only one project*
• May require copying CAD to data from various directories to one
directory
• Does not handle loading of multiple revisions
*Avoidable with scripting
Overview of SmarTeam Import Tool
•
•
•
•
•
Out of the box tool provided by
SmarTeam to load meta data
The ability to pick a class to load data
• Leaf Classes
• Link Classes i.e. Document Project
Links
The ability to map data from:
• Database – SmarTeam DB or non
SmarTeam DB
• Text File – Delimited or Fixed Width
• Default Value – Defined in
SmarTeam
• Default Value – Specific for the data
load
• Populate field by a Sequence
Log created for data that is imported
Import Tool can run in an Add or Update
mode
Issues with SmarTeam Import Tool
• Limited to 32,000 records per load
• Greater than 32,000, data needs to be split
and multiple loads conducted
• Tool can be “buggy”
• Crashing from time to time
• Data Mappings may get lost
• Tool is manual
• For each class to be imported mappings
should be configured
• Can be tedious to do for many classes
• Tool does not handle certain condition (with
out Scripting)
• Loading multiple revisions and specifying
the latest revision
• Loading data to internal SmarTeam fields or
Reference to Class fields
Overview Custom Scripts / Programs
• Best suited for larger data
migrations
• Almost any type of data loading
rule can be created
• Determine what class to load
the data
• Modify data to be loaded
based on data in the system
• Re-formatted before going
into SmarTeam
• Need to create or build link
relationships in the system
• Various complex Data load
scenarios will be conducted
• Need custom logging an error
handling
Custom Script Data Loading - Simple Example
•
•
•
•
•
•
•
Data exists in a legacy Folder
File Name contains meta data attributes such as
ID_Descritpion_Rev
.doc files will go into an Engineering folder
.xls files will go into an Manufacturing folder
Project will need to be specified
Maintain latest revison
SmarTeam Custom Script Tool Example
Combination Bulk Load and Custom script / Import
Utility
• Bulk Load CAD data into
SmarTeam
• Hook scripts to the SmarTeam
“Before Add” Event to retrieve
information
• i.e. from the file name to
update fields on the Profile
Cards
• Write custom code that will
query the database based on
the file name
• Update information from
another source where the file
name is used to link the file
data /object already in
SmarTeam and the other
external source for data
1. Pre-Migration Data Analysis
•
•
•
•
Data Clean-up and Process Review
Data Formats
Data Loading Rules
Migration Cost Estimation
2. Data Loading Options
•
•
•
•
•
Bulk Loader
SmarTeam Import and Export Tools
Custom Scripts (executables, add-ins, etc)
Combined Manual Processing and Custom Scripting
Pre-Developed Data Migration Programs
3. Execution
•
•
•
•
Testing the Migration Process
Data Verification
Pre-Data Load Planning
Production Data Load
Execution:
Testing the Migration Process
• Legacy
data issues
• Test
the migration
process!!!
» Missing data files
• Mitigate
all discovered issues
» Orphan links and/or blank fields
» Inconsistent
legacy data such as revision schema
» Bad
formats
– E.g., Attempts to load characters into integer fields
– Class determination rules fail
» Repairing the legacy data
– Manual clean-up
– Change data load rules (i.e., change the code)
– Modify the SmarTeam data model to better accommodate the
legacy data
•
Bug(s) in the migration tool code
• Document
the procedural
migrationissues
procedure or update
• Data loading
the tool
to failure
automate
migration
process
» e.g.,
to deletethe
all files
with a creation
date before 12/31/1999
• Test again!!!
 Crucial to implement a data verification
strategy
 Analyze migration process log files
 Spot-check data
 Randomly verify the data objects loaded
 Check at least 10% of data
 User validation (done through testing with the
pilot system) – an important part of the
verification process
 User validations
IT/CAD administrators
sign-offs
 Develop a tool to compare and analyze the
data in SmarTeam and legacy system(s)
 Verify the number of records between systems
 Ensure pertinent fields and data are preserved
in both systems
Execution: Pre-Data Load Planning
• When will the legacy system be turned off?
• How long will it take to load the data?
• Will the data loading occur during nonbusiness hours?
• Specify the data load process:
o What are the required pre-data loading
activities/events?
• e.g., legacy data clean-up
o Document the data load sequence
• Populate the lookup tables
• Import users before documents
• Import projects and documents, followed by the
associated links
• Create references to class data
•
Identify personnel responsible for the various
parts of the data load process
Execution: Final Data Load
• With proper planning and testing, the
final data load process “should” go
smoothly
• Advices:
• Follow the documented
procedures
• Allocate buffer time for
unexpected issues
• Prepare to support users in the new
system
Thank
You!
Questions?
Script example shown is available at:
http://www.xlmsolutions.com/COE2009/example.zip
Marc Young
xLM Solutions, LLC.
248-926-5932
[email protected]