Transcript Slide 1

ESCWA SDMX Workshop
Session: SDMX Introduction
© Metadata Technology
What is SDMX?
• The problem space (2001):
– Statistical collection, processing, and
exchange is time-consuming and resourceintensive
– Various international and national
organisations have individual approaches for
their constituencies
– Uncertainties about how to proceed with new
technologies (XML, web services …)
© Metadata Technology
What is SDMX?
The Statistical Data and Metadata
Exchange (SDMX) initiative addresses
these challenges and opportunities that
have just been mentioned:
– By focusing on business practices in the field
of statistical information
– By identifying more efficient processes for
exchange and sharing of data and metadata
using modern technology
© Metadata Technology
Who is SDMX?
• SDMX is an initiative made up of seven
international organizations:
–
–
–
–
–
Bank for International Settlements
European Central Bank
Eurostat
International Monetary Fund
Organisation for Economic Cooperation and
Development
– United Nations
– World Bank
• The initiative was launched in 2001
© Metadata Technology
Official Recommendations
• SDMX has been officially recommended:
– February 2007: SDMX endorsed by the
European Union’s Statistical Programme
Committee (now ESS Committee)
– March 2008: UN Statistical Commission
declares SDMX to be the preferred standard
for data and metadata
© Metadata Technology
SDMX: Benefits
• For Data Consumers
– Search for, locate, and query data and related metadata
– Query directly the database and metadata repositories of
the data and metadata producers
– View data and related metadata in table, charts, graphs,
maps using SDMX visual components
• For Data Collectors
– Share the data collection burden
– Eliminate many and varied formats for data and reference metadata
– Automate the database and metadata repository loading process to
assist validation
SDMX: Benefits
• For Data Reporters
– Common reporting format (including coding schemes) for
all reporting needs
– Ability to open databases and metadata repositories to
direct query for data collectors
• For Data Publishers
– Automated web dissemination system for data and
reference metadata
– As soon as data is published it is available for query, and
linked to the reference metadata
Exchange Patterns
• Bilateral: Institutions exchange data
according to bilateral agreements
regarding format, timing, protocols, etc.
• Gateway: Institutions share the data they
collect with their peers, in agreed formats
among counterparty communities
• Data-sharing: standard exchange of data
using standard formats and protocols
© Metadata Technology
Bilateral Exchange
© Metadata Technology
Gateway Exchange
© Metadata Technology
Data/Metadata-Sharing Exchange
query/retrieval
Registration of
Data Source
Data/Metadata
Reporters
© Metadata Technology
RSS or SDMX
Notification
Data/Metadata
Consumers
Adopters/Interest
• The following are known adopters (or planning to adopt):
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
US Federal Reserve Board and Bank of New York
European Central Bank
Joint External Debt Hub (WB, IMF, OECD, BIS)
UN/TRADECOM at UN Statistical Division
NAAWE (National Accounts from OECD/Eurostat)
European Statistical System (Eurostat and National Statistical Institutes)
Mexican Federal System
IMF (BOP, SNA, SDDS/GDDS)
Food and Agriculture Organization
Millennium Development Goals (UN System, others)
International Labor Organization
Bank for International Settlements
OECD
World Bank
Marchioness Islands (Spanish/Portuguese Statistical Region)
UNESCO (Education)
Australian Bureau of Statistics
International Energy Agency
World Health Organization
There are many others
© Metadata Technology
SDMX Status and Future
• Revised version 2.1 standards have been released (end April)
– Increasing availability of SDMX tools
• In May 2011 the IMF and World Bank hosted a 3-day SDMX Global
Conference
–
–
–
–
250 people attended
Many implementation scenarios
Focus on web services
There was a vendor demonstration area on day 3
• Commitment of the Sponsors
– 2011 – Web “Sandbox” for users to play with SDMX
– 2012 - Global SDMX Registry
• Retrieval of common structural metadata by local registries
• Sharing of structural metadata
• Portal to find data and SDMX web services
– 2015 - 20 International Data Structure Definitions
– World Bank and IMF asked to help with training and tools for
developing nations
© Metadata Technology
SDMX and Domain Standards
• SDMX are keen that organisations
prominent in a specific domain take the
lead to define the domain standards
– Structural definitions
•
•
•
•
Code lists
Concepts
Data structure definitions
Metadata structure definitions
– Quality metadata and quality frameworks
• Based on the SDMX content oriented guidelines
© Metadata Technology
SDMX Version 2.0/2.1: High Level Overview
Process
Data Set or
Metadata
Set
Data
Provider
© Metadata Technology
Data
or Metadata
Structure
Definition
Structure and
Item Scheme
Maps
Category
Scheme
Categorisation
Data or
Metadata
Flow
Provision
Agreement
Category
Content
Attachment
Constraint
Constraint
Registered Data
Source or
Metadata Source
Data
or Metadata
Structure
Definition
High Level
Overview:
Flow
Category
Scheme
Category
Categorisation
Data or
Metadata
Flow
Data Provider
Scheme

Provision
Agreement


Data or
Metadata
Source
© Metadata Technology
Registered Data
references
Source or
Metadata Source
Data
Provider
R
e
g
i
s
t
e
r
Existing Websites Using SDMX
• BIS
– http://stats.bis.org
• ECB
– http://www.ecb.int/stats/exchange/eurofxref/html/index.en.html
– http://stats.ecb.europa.eu/stats/sdmx/visualisation/icp/dashboard/r
c1/
Code available at (http://flex-cb.googlecode.com/ )
• OECD
– http://stats.oecd.org/WBOS/index.aspx
– http://stats.oecd.org/OECDregionalstatistics/
© Metadata Technology
SDMX: Course Introduction
• At the end of the Sessions on SDMX you will
have
– Understood the scope of SDMX
– Gained a high level understanding of the SDMX
Information Model
– Know the key SDMX structures that can be used to
create, report, store, maintain, visualise, MDG data
and structural metadata
© Metadata Technology
SDMX: Course Introduction
• In order to do this you will learn about
– SDMX Structural Metadata
•
•
•
•
•
•
•
•
•
•
DSD and related Concepts and Code lists
MSD and related Concepts and Code lists
Category Scheme and Classification
Data flow and Metadata flow
Provision Agreement
Registration
Subscription and Notification
Registry
Database
Query
– SDMX Data and Metadata
• Data set formats
• Metadata set
– SDMX Web Services
© Metadata Technology
Tools Used on the Course
SDMX Registry/Repository
© Metadata Technology
Tools Demonstrated
SDMX Data Viewer
SDMX Relational Database
© Metadata Technology
SDMX Structure and Data Converter
Now – The Hard Work
© Metadata Technology