Transcript Metadata requirements for archiving structured data Alice Born Statistics Canada Joint UNECE/Eurostat/OECD Work Session on Statistical Metadata (9-11 April 2008)

Metadata requirements for archiving
structured data
Alice Born
Statistics Canada
Joint UNECE/Eurostat/OECD Work Session
on Statistical Metadata (9-11 April 2008)
IMDB business model – archive focus
Applications/
Software
Survey
Frame and
sample
Survey
instance
Datafiles in
archive
repository
Questionnaire
Datasets
Products
(COR)
Survey design
Data elements
Value domains
Outline
• Overview of the integrated metadatabase
(IMDB) in the survey life cycle
• Archive and disposal processes
• Metadata requirements for archiving
• Registration
IMDB in the survey life cycle
Data Warehouses
Operations
Management
Quality
Assurance
Metadata
IMDB
Design
Analysis
Collect
Operational
Data
Edit
Estimate
Registers
Dissemination
IMDB
Tabulate
Survey
Data
Operational Data Stores
Publish
Administrative
Data
Archive +
disposal
Processes for archiving and disposal
8. Archiving and disposal
8.1 Manage archive repository
• define archival format, format data, load data,
record event dates, links to metadata registry
(IMDB)
8.2 Preserve data and associated metadata
• Identify data/metadata, record archive request,
transfer data/metadata, etc.
8.3 Dispose data and associated metadata
• Identify data to dispose, remove from archive, etc.
Types of metadata for archiving
structured data (1)
1. Administrative metadata
•
•
Maintaining and keeping track of the archived
datafile
Link to an IMDB record – SDDS or unique
datafile identifier
2. Structural metadata
•
•
•
•
Name of the datafile
File format – record layout
Software
Storage media, location
Types of metadata for archiving
structured data (2)
3. Survey and definitional metadata
•
•
•
•
•
•
•
Already in the IMDB
Survey description
Questionnaires, questions, response choice
Methodology
Measures of data quality
Variables (ISO 11179)
Additional documentation – interviewer guide,
coding tool
Administrative layer
Statistical Activity
Organization
Survey
Stewardship
Contact
Universe
Documentation
Frame
Identification
Survey instance
Time Frame
Instrument
Keyword
Question
Identification
Classification
Theme
Data file
Methodology
Administered
items
Instrument design
Sampling
Data source
Error detection
Imputation
Estimation
Quality evaluation
Disclosure control
Revisions and seasonal
adjustment
Data accuracy
Data Element
Data Element Concept
Object Class
Property
Formula
Conceptual Domain
Value Domain
Registration status
Registration
Authority
(Completeness, accuracy,
adherence to quality and
terminological description
standards)
Disposed
Preferred standard
Archived
Standard
Retired
Standards Division
Registrar
Qualified
Regular
Registrar
Recorded
Responsible
Owner
Candidate
(Content)
Steward
Submitter
Incomplete
Historical
Corporate Memory: Data and metadata for archive and
disposal phases
Operational
data
Registers
Survey
Data
Administrative
Data
Metadata for archive
and disposal
IMDB
Public Use
Microdata Files
Clean Master
Files
Archive
repository
+
Disposal