Transcript Measuring Data Quality and Compilation of Metadata Second Regional Workshop on the Compilation of Basic Economic Statistics in African Countries: Addis Ababa Thierno Aliou Balde United.

Measuring Data Quality and
Compilation of Metadata
Second Regional Workshop on the
Compilation of Basic Economic Statistics in
African Countries: Addis Ababa
Thierno Aliou Balde
United Nations Statistics Division
October 16-19 2007
1
Outline
1.
2.
3.
4.
Quality Assessment Frameworks
(DQAF) for Basic Economic
Statistics (BES)
Quality Indicators versus Direct
Quality Measures
Metadata on BES
Recommendations - Conclusion
2
1. QAF for BES


Quality measurement of BES is concerned with
providing the user with sufficient information to
judge whether or not the data are of adequate
quality for their intended use, i.e. to judge their
“fitness for use” .
For example, data users must be able to verify
that the conceptual framework and definitions
that would satisfy their particular data needs are
the same as, or sufficiently close to those
employed in collecting and processing the data.
3
1. QAF for BES


Several statistical organizations and countries
have developed definitions of quality and
integrated them into QAF.
Although the existing quality assessment
frameworks slightly differ in their approaches to
quality and number/name of quality dimensions,
they complement each other and provide
comprehensive and flexible structures for the
qualitative assessment of a broad range of
statistics
4
1. QAF for BES
1.
The IMF quality concept: covers the
prerequisites and five dimensions of quality
– assurance of integrity, methodological
soundness, accuracy and reliability,
serviceability and accessibility.
2. The European Statistical System (Eurostat):
considers six criteria – relevance, accuracy,
timeliness and punctuality, accessibility and
clarity, comparability and coherence.
3.
The OECD quality concept: quality is viewed in
terms of seven dimensions – relevance,
accuracy, credibility, timeliness, accessibility,
5
interpretability and coherence.
1. QAF for BES



The overall aim of QAFs is to standardize and
systematize statistical quality measurement
and reporting across countries.
They allow an assessment of national practices
to be made against internationally accepted
statistical approaches for quality measurement.
QAFs could be used in a number of aspects,
including for:
- Guiding countries’ efforts for strengthening
their statistical systems by providing a selfassessment tool and for identifying areas of
improvement;
- Technical assistance purposes;
- Reviews of particular statistical domains
performed by international organization;
6
- Assessment by other groups of data users.
1. QAF for BES
Dimensions of Quality:








Prerequisites
Relevance
Credibility/Assurance and Integrity
Accuracy/ Accuracy and Reliability
Timeliness/ Serviceability
Methodological Soundness/Interpretability
Coherence/ Consistency
Accessibility
7
1. QAF for BES
Dimensions of Quality
(cont.)


These dimensions of quality form a complex
relationship:
- overlapping
- interrelated
Measurement of BES data quality is not a simple
task.
8
1. QAF for BES
Dimensions of Quality (cont.)




Prerequisites of quality refer to all institutional
and organizational conditions that have an impact
on the quality of BES data.
Relevance: reflects the degree to which BES data
meets the real needs of the users.
Credibility: refers to the confidence that users
place in the data based on the image of the
statistical agency that produce the data.
Accuracy: it is the degree to which the data
correctly estimate or describe the characteristics
9
they are designed to measure.
1. QAF for BES
Dimensions of Quality (cont.)


Timeliness: refers to the delay between the end
of the reference period to which the data pertain,
and the date on which the data are released.
The methodological soundness/interpretability:
refers to the application of international
standards, guidelines and good practices in
production of BES.
- The metadata provided along with BES play
a crucial role for assessing the methodological
soundness of data.
- Closely related to the interpretability which
reflects the ease with which the user may
understand and properly use/analyze the data.
10
1. QAF for BES
Dimensions of Quality (cont.)


Coherence/consistency: reflects the degree to
which the data are logically connected and
mutually consistent:
- Coherence within dataset
- Coherence across datasets
- Coherence over time
- Coherence across countries.
Accessibility: refers to the ease with which data
can be obtained from the statistical office.
11
2. Quality Indicators versus
Direct Quality Measures
Quality Measures:



Items that directly measure a particular
aspect of quality.
Example: the time lag from the
reference date to the release of the
data.
In practice quality measures are
difficult or costly to calculate.
12
2. Quality Indicators versus
Direct Quality Measures
Quality Indicators:



Summarized quantitative data to provide
evidence about the quality of the data.
They do not measure quality directly but provide
enough information for the assessment of quality.
Types of quality indicators:
- Key indicators
- Supportive indicators
- Indicators for further analysis
13
3. Metadata on BES
Statistical data consists generally of:



Micro data: data on the characteristics of units of
the population.
Macro data: derived from the micro data by
grouping or aggregation.
Metadata: “data about data”, describes the micro
data, macro data or other metadata.
14
3. Metadata on BES


Statistical metadata describes or document
statistical data.
The most fundamental purpose of metadata is to:
- help users understand, interpret and
analyze the data.
- help the producers of statistics to enhance
the production and the dissemination of the data.
15
3. Metadata on BES



A bi-directional relationship between metadata
and quality:
- metadata describe the quality of statistics
- metadata are a quality component.
Metadata provide a mechanism for comparing
national practices in the compilation of BES.
A broad spectrum of metadata requirements has
to be addressed for the wide range of possible
users and uses of metadata.
16
3. Metadata on BES

Six dimensions of metadata:
- Data coverage, periodicity and timeliness
- Access by public
- Integrity of the disseminated data
- Quality of the disseminated data
- Summary methodology
- Dissemination format
Each of these dimensions is characterized by a
few elements that can be observed or monitored
by the users.
17
4. Recommendations Conclusion
It is recommended that:



Countries take into account the above dimensions
of quality in the process of producing BES.
Countries use the dimensions of quality when
measuring and reporting the quality of BES.
Countries undertake a quality review of BES
every four or five years or more frequently if
significant methodological changes or changes in
the data sources occur.
18
4. Recommendations Conclusion
It is recommended that:



Countries pay careful attention to maintain a
correct balance between different dimensions of
quality and use a minimum of indicators.
Countries satisfy the following criteria when
defining the quality indicators for BES:
- Coverage of part or all of the 8 dimensions
of quality
- Well established methodology for their
compilation
- Indicators are easy to interpret.
Countries adopt the segmentation of users into
groups and a layered approach to metadata
presentation.
19
4. Recommendations Conclusion
It is recommended that:

Countries adopt a development of a
coherent system and a structured
approach to metadata across all areas of
economic statistics, focusing on improving
their quantity and coverage.
20
4. Recommendations Conclusion
Countries are encouraged to:



Use a system of quality indicators and develop
their own BES quality framework based on the
quality dimensions and the specific circumstances
in their economy.
Issue regularly, quality reports as part of their
metadata.
Give a high priority to the development of
metadata and to consider their dissemination an
integral part of dissemination of BES.
21
Thank You
22