Transcript Quality Assurance Frameworks

The use and convergence of quality assurance
frameworks for international and supranational
organisations compiling statistics
The European Conference on Quality in Official Statistics
July 8-11 2008, Rome, Italy
Antonio Baigorri and Håkan Linden
Statistical Governance, Quality and Evaluation
European Commission, Eurostat
The context of Total Quality Management
 To have an encompassing approach with respect to quality work.
 To implement the principles of institutional quality frameworks and in
particular the principles related to statistical processes and outputs.
 To improve the measurement, monitoring and management of data
quality.
 To coordinate ongoing quality initiatives (process descriptions, quality
reports, evaluation activities etc.).
 To build on existing quality work (standards, best practices etc.).
 To promote a culture of systematic quality improvement work.
2
Institutional Quality Frameworks
TQM
TQM
TQM
User
User
needs
needs
Code
DQAF
of Practice
User needs
Management
Management
systems
systems
andand
leadership
leadership
Management
systems and
leadership
Support
Support
processes
processes
Support
processes
Statistical
Statistical
products
products
Statistical
products
Production
Production
processes
processes
Production
processes
Institutional
Institutional
environment
environment
Institutional
environment
11 4.
Relevance
Serviceability
12 Accuracy and reliability 13 Timeliness and
CCSA
Principles
Punctuality
14 Coherence and comparability 15 Accessibility
5. Accessibility
And clarity
Principles: 1.1, 1.2, 4.3, 4.4, 4.5, 4.6,
5.2, 6.2, 7.1, 7.2, 8.1, 10.2, 10.5
7 Sound
2. Methodological
methodologySoundness
8 Appropriate statistical procedures
9 Non-excessive
3.Accuracy and
burden
Reliability
on respondents 10 Cost effectiveness
Principles: 4.2, 5.1, 5.3, 5.4, 5.5,
8.3, 8.4, 9.2, 9.3, 9.4, 9.5, 10.1
1 Professional
0. Prerequistes
independence
of Quality 2(Legal
Mandate
and for
institutional
data collection
3 Adequacy
environment,
of resources
Resources,
4 Quality
Relevance,
commitment
Other quality
5 Statistical
management6 Impartiality and objectivity
confidentiality
1. Assurance
of 1.4,
Integrity
(Professionalism,
Principles:
1.1, 1.3,
1,5, 2.1,
2.2, 2.3, 3.1, 3.2,
4.1,Transparency
5.3, 5.6, 6.1, and
6.2, Ethical
7.1, 8.2,Standards
9.1, 10.3, 10.4
Source: Eurostat (2007)
Institutional frameworks, like the Principles Governing Statistical Activities, the
European Statistics Code of Practice and the IMF Data Quality Assessment Framework,
can be seen as general superstructures forming the necessary basis for all other measures
an International organisation needs for improving quality at statistical output and product level.
3
Quality Assurance Frameworks
 Quality assurance frameworks (or frameworks for statistics production)
have the objective to establish, in a specific statistical organisation, a
system of coordinated methods and tools guaranteeing the adherence
to minimum requirements concerning the statistical processes and
products. Similarly to institutional frameworks, this includes some kind
of assessment.
 Product/ output quality requirements are being explicitely documented.
 Processes are defined and made known to all staff.
 The correct implementation of the processes is monitored on a regular
basis.
 Users are being informed on the quality of the products and possible
deficits.
 A procedure is implemented that guarantees that the necessary
improvement measures are being planned, implemented and
evaluated.
4
Data quality aspects
1) The perception of the statistical product by the user.
2) The characteristics of the statistical product (or key statistical outputs)
3) The characteristics of the statistical production process.
5
Relationship between process and output quality
RELEVANCE
ACCURACY
ACCESSIBILITY/
CLARITY
TIMELINESS/
PUNCTUALITY
COMPARABILITY
COHERENCE
Conceptual framework (2)
Follow-up (10)
User needs
(3)
Data
collection
(4)
Validation
Country level (5)
International level (6)
Confidentiality
(7)
Documentation Dissemination
(8)
(9)
IT conditions (11) – Management, planning and legislation (12) – Staff, work conditions and competence (13)
Source: Eurostat Process Quality Assessment Checklist [Eurostat, 2007]
6
Product/ Output Quality Components
OECD:
relevance, accuracy, credibility, timeliness (and punctuality), accessibility,
interpretability, coherence (within dataset, across datasets, over time, across
countries)
Eurostat: relevance, accuracy, timeliness and punctuality, accessibility and clarity,
coherence (within dataset, across dataset), comparability (over time, across
countries)
ECB:
accuracy/reliability, methodological soundness, timeliness, consistency
IMF:
prerequisites of quality, accuracy and reliability, assurances of integrity,
methodological soundness, serviceability (timeliness and periodicity),
accessibility, serviceability (within dataset, across dataset, over time, across
countries)
FAO:
relevance (completeness), accuracy, timeliness, punctuality, accessibility,
clarity (sound metadata), coherence, comparability
UNESCO: relevance, accuracy, interpretability, coherence
UNECE:
relevance, accuracy (credibility), timeliness, punctuality, accessibility, clarity,
comparability (across datasets, over time, across countries)
7
Product/ Output Quality Components

Relevance

Accuracy (and reliability)

Timeliness

Punctuality

Accessibility

Clarity/ interpretability

Coherence/ consistency

Comparability
8
Quality and metadata

Collection and sharing of metadata
- SDMX technical standards
- SDMX Content oriented guidelines (incl. Cross domains concepts)

Dissemination of metadata on quality
- Special Data Dissemination Standards (SDDS)
- Euro-SDMX Metadata Structure (ESMS)

Assessment and monitoring of metadata

Integrated information on quality assessment
9
Methods and tools for the assessment of statistics production
User requirements
Standards
III. Conformity
Labelling
II. Evaluation
I. Documentation
Measurement
Self assessments
Process
variables
Quality
indicators
Production
processes
Quality reviews
Quality
reports
User satisfaction survey
Statistical products
Improvement
actions
User
perception
Institutional/ legal environment
N.B. Figure derived from draft Handbook on Data Quality Assessment Methods and Tools (DatQAM), version 31.01.2007.
10
How to apply process oriented quality assessment tools
 The office-wide management approach
 Institutional preconditions (procedures and legislations)
 Assessment methods already in use
 Relevance – size and periodicity
 Relevance – importance and specific legal frameworks
11
Quality Assessment Packages
Labelling
Advanced
package
Key process variables
User satisfaction surveys
Quality reviews
Intermediate
package
Quality indicators
Self assessments
Quality reports
Fundamental
package
Process descriptions, product documentation, quality guidelines
N.B. Figure derived from draft Handbook on Data Quality Assessment Methods and Tools (DatQAM), version 31.01.2007.
12
The assessment methods and tools
 Documentation and measurement
- process descriptions
- quality reports (“Full Quality Report”, “Summary Quality Report”, and “Basic Quality
Information”).
- user satisfaction surveys
 Evaluation
- self assessments of all production processes (Quality Assessment Checklist)
- quality reviews for key statistical outputs
 Conformity
- a process for labelling of key international statistics
13
Principles for implementation
 Minimise burden for production domains
- test the approach in advance
- provide support
- flexibility
 Build on existing information
- process analysis
- metadata on quality (quality reports etc.)
 Profit from synergies with other horizontal activities
- evaluation function requirements
- cost/ benefit analysis
- input for management programming
14
Data quality assessment recommendations

Top management commitment

The role of middle managers

Data quality assessment is a long term project

Most methods should be implemented and fine-tuned in pilot projects

Standardise the use of the methods

Establish clear responsibilities and authorities

Sufficient resources allocated for supporting the assessments
15
Conclusions

Each international organisation should have a quality assurance framework in place.

The framework and the applied quality principles should be made explicit.

A quality assurance framework needs to be compatible with the general quality
management model and office-wide procedures and rules.

It should be built into the organisational structure.

It contributes to increased awareness of quality concepts and promotes best
practices.

It provides a mechanism for reengineering and quality improvements

It should always acknowledge performance/ cost.

Convergence of quality assurance frameworks by applying common concepts,
standards, methods and tools (both content oriented and technical).

Development and sharing of “best practices” for statistics production between all
stakeholders is maybe the most important for continuous quality improvement of a
global statistical system.
16