Transcript QUALITY ASSURANCE RESULTS

Recent Quality Related
Initiatives at Statistics
Canada
Claude Julien
Statistics Canada
Quality Secretariat
Q2008, Rome
Introduction
Statistics Canada’s Quality Assurance
Framework
 Research on adding another component to the
framework presented at Q2006
 Soon after, a few unfortunate errors led to
three quality related initiatives

Outline
Introduction
 Review of quality assurance practices
 Quality assurance learning and awareness
exercise
 Additional quality checks and analysis in the
data release step

Review of quality assurance
practices in statistical
programs
Design

Objectives:



Identify quality risks to address
Identify quality best practices to share
Scope:





Operational checks throughout the production process
Analytical checks to validate statistical information
Data release operations
Focus on production, not design
Focus on processes, not outputs
Design (cont.)
Coverage - Nine mission critical programs
 Assessment approach

Concurrent internal independent review
 Three reviewers per program
 Three programs per reviewer

Findings
Good overall approach
 Sound QA practices in most production steps
of most programs
 More common risk factors affecting programs:

Loss of expertise
 Lack of staff / staff turnover
 Timeliness pressures on monthly programs
 Complexity of data release operations

Outcome
Report disseminated to the public
 Allocation of additional resources to address
more urgent needs
 Development of ongoing review program
integrated into existing planning, priority
setting and review mechanisms
 Input into other initiatives

Quality assurance learning
and awareness exercise
Design
Main objectives: Bring survey managers
together to discuss QA practices in their
programs (learning)
 Other objectives:

Surface main areas in their programs at risk of
causing significant errors (risk awareness)
 Provide managers with basis to deal with risks
(take action – risk management)
 Identify areas that are more at risk collectively

Design (cont.)
E-learning exercise (computer based training)
 Group exercise
 Discuss and answer over 100 questions on QA
practices gathered from:

existing production and risk management practices
 input from quality review (means of sharing best
practices)

Outcome
 Broad
coverage in very short time; over
800 participants in less than 4 months
 Useful discussion and exchange of
opinions among participants
 Presence/absence of over 100 QA
practices in 80 statistical programs
Findings



Most of the risk lies among certain types of
QA practices (e.g. documentation) rather than
in certain types of programs
Support for the scope of the reviews on
quality assurance practices
Need to review learning and development
strategy on QA, project management and risk
management
Quality checks and analysis in
the data release step
Complexity of data release
Increased relevance of information
 Increased timeliness and accessibility

5 releases per day on average
 Release article, data tables, publications and
metadata all at once

Numerous systems with many
interdependencies
 Many people involved from many parts of the
organization

Recent initiatives
Additional checks
 Moving checks upstream
 Systematic recording and analysis of
corrections made to articles just prior to or
after release (i.e. get the facts)
 Feedback to managers
 Regular reporting to senior management

Recording and analysis of corrections

Characteristics





When (month and frequency of release)
Where in article (text, tables, charts or metadata)
What in article (words, numbers, reference period, link,
etc.)
How important (potential impact on quality)
Underlying causes and factors of more important
corrections


Where in survey process did the error occur?
Why did the error occur?
Findings
Many corrections do not affect accuracy
 Nearly all corrections after release are made on
the day of release (within a few hours)
 Errors earlier in the survey process are less
likely, but have higher potential impact
 Errors in the data release steps are more likely,
but have lower potential impact
 Some suggestion of seasonality

Summary
Quality Assurance Framework - Before

Design





User consultation
Subject matter
consultation
Subject matter training
Quality Guidelines
Standards

Production



Knowledge,
experience, skill
Project management
Quality Control
Quality Assurance Framework - Now

Design





User consultation
Subject matter
consultation
Subject matter training
Quality Guidelines
Standards

Production






Knowledge,
experience, skill
Project management
(More) Quality Control
(in data release step)
Quality reviews
Quality learning and
awareness
Risk management
Outcome
Percentage of articles corrected after release
Summer 2007 Spring 2008
All articles
7%
3%
Corrected for accuracy
4%
2%
Corrected for accuracy
with potential impact
2%
0%
Grazie! / Merci!
For more information, or
to obtain a French copy
of this presentation,
please contact:
Pour de plus amples
informations ou pour obtenir
une copie en français du
document, veuillez
contacter :
Claude Julien
Email / Courriel: [email protected]
Phone number / Téléphone: 613-951-6937