Transcript Slide 1

European Conference
on Quality in
Survey Statistics
Quality in Official
Statistics:
Some Recent and Not so
Recent Developments
Lars Lyberg
Statistics Sweden
Q2006
Why We Have a Q Conference
• One of the LEG recommendations
• The ESS mission, where it is stated that ESS shall
provide the EU and indeed the world, with high quality
information, available to everyone, on various areas and
levels for decision-making, research and debate.
• The ESS vision with keywords such as world leader,
scientific principles, continuous improvement,
harmonization, and basis for democracy and progress.
• Westat, Inc.
Contents of Q so Far (# ms)
• Evaluation of data quality (92)
• Sampling and estimation (65)
• Nonresponse (44)
• Questionnaire development
and testing (22)
• Confidentiality (17)
• Burden (5)
• Knowledge economy (5)
• Quality management of
systems and organizations (63)
• Frameworks (11)
• Reporting (52)
• Process control (36)
• Auditing and
self-assessment (11)
• Customers (36)
• Standards (11)
• Harmonization (12)
Somewhat Neglected Topics
• Respondents
• Costs
• Trade-offs
• Standardization
• Fitness for use
• User perception of quality
• Trust
• Audits and self-assessment
• The nitty-gritty of QM
Issue No.10
The Concept of Quality
• Statistical Process Control (30’s and 40’s)
• Small errors indicate usefulness
(Kendall, Jessen, Palmer, Deming, Stephan, Hansen,
Hurwitz, Tepping, Mahalanobis)
• Decomposition of MSE around 1960
• Data quality (Kish, Zarkovich 1965)
• Quality frameworks 70’s
• CASM movement 80’s
• Quality and users
• The UN Fundamental Principles
Components of Quality
Defining Quality
• Fitness for use or fitness for purpose
• Framework components
• Getting the job done, on time, within budget, so that it
meets the specified requirements
Quality Assurance and Quality Control
• QA makes sure that the processes are capable of
delivering a good product
• QC makes sure that the product is actually good
Controlling Quality
Quality Level
Main stakeholders
Control
instrument
Measures
Indicators
Product
User
Product specs
Framework
dimensions,
error est., MSE
Process
Survey designer Process
variables, SPC,
CBM, SOP ,
checklists
Variation via
control charts,
other paradata
analysis
Organization
NSI, owner,
Scores,
society
Excellence
models, CoP,
Reviews,
Audits, Selfassessments
Strong and
weak points,
Are we
measuring up?
Issue No. 9
Quality Measurement and Quality Reporting
• Objective: To ensure that users have access to measures
or indicators of quality, presented in ways that meet their
particular needs
• The typical framework: relevance, accuracy, timeliness
and punctuality, accessibility and clarity, comparability,
and coherence
Examples of Reports
• Dataset-specific quality assessments for different kinds of
economic statistics (IMF)
• Process data handbook (LEG/UK)
• Quality guidelines (Stats Canada, Stats Finland)
• Questions and Answers (OMB)
• National or organizational frameworks
• Quality profiles
• Guidelines for quality reporting (Stats Can, ONS,, Stats
Sweden, FCSM)
Concerns
• The user has not been
consulted
• Some quality indicators are
dubious
• How should dimensions be
measured?
• Dimensions are in conflict
• How do we handle information
gaps?
• What happened to total survey
error or total quality? Särndal
and Platek (2001)
• Do we need global
harmonization?
Issue No. 8
Deming’s 13 points
• The 13 factors that affect the usefulness of a survey
• To point out the need for directing effort toward all of
them in the planning process with a view to usefulness
and funds available
• To point out the futility of concentrating on only one or
two of them
• To point out the need for theories of bias and variability
that correlate accumulated experience
The 13 Points
1. Variability in Response
2. Differences between Different Kinds and Degrees of
Canvass
3. Bias and Variation Arising from the Interviewer
4. Bias of the Auspices
5. Imperfections in the Design of the Questionnaire and
Tabulation Plans
13 Points Continued
6. Changes that Take Place in the Universe before
Tabulations Are Available
7. Bias Arising from Nonresponse
8. Bias Arising from Late Reports
9. Bias Arising from an Unrepresentative Selection of Data
for the Survey or of the Period Covered
10. Bias Arising from an Unrepresentative Selection
of Respondents
13 Points Continued
11. Sampling Errors and Biases
12. Processing Errors
13. Errors in Interpretation
Issue No. 7
The Race for the No.1 Spot
• Started with The Economist’s ranking
• There is an element of positioning in some of the visions
presented by statistical organizations
But:
• There is no justification for competition
• There is no framework, jury or reward
• Statistical organizations have the same problems and
tasks and need to collaborate
• Statistical organizations should capitalize on their
strengths and develop excellence centre networks and
share knowledge
Global Coordination
• Kotz (2005): The statistical community is witnessing an
astonishing lack of coordination between many hundreds
of statistical offices and agencies scattered throughout
the world….
• ..without an overall planning, some of the efforts of civil
servants and researchers are largely wasted….
• …well-planned international measures are urgent….
• …new basic global definitions of basic concepts need to
be developed…
Issue No. 6
Quality Management
• TQM, Business reengineering, Balanced scorecard,
business excellence models, Six Sigma
• Tools and core values
• Aversion to QM acronyms
• The management principles cannot be used uniformly
across countries and companies
• Operations vs research culture
• Culture eats strategy for breakfast
• We are left with a set of very useful tools and work principles
Examples
• The process view
– Key process variables, paradata, control charts
• Spirit of continuous improvement
• Extensive user involvement
• Adoption of the PDCA cycle
• The importance of leadership
– Organizing work, inspiration, focussing on important issues,
going for root cause, benchmarking, developing staff
competence, evaluating approaches used, promoting good
examples, empowerment, communication
From Good to Great
Jim Collins
• What’s so special with businesses that have
• been very successful for at least 15 years?
• Level 5 leadership
• First who, then what
• Confront the brutal facts, yet never lose faith
• The hedgehog concept
• Culture of discipline
Issue No. 5
Competence
• Staff competence
– Excellent programs within the U.S. Federal System (JPSM,
USDA)
– Stats Canada, INSEE, ONS, ABS
– Excellent university programs
• User competence
Competence Issues
• Existing programs heavy on methodology
– Sampling and estimation
– Software
• Specialization
• Not much on broader aspects of quality
• Many NSIs talk about the need to skill up
• Any examples of vigorous attempts vis-a-vis the user?
Issue No. 4
Comparative Studies
Comparative studies are increasingly
• important:
• Short term economic indicators
• Literacy surveys
• Social surveys (EU-SILC, ESS)
• Education surveys
Examples of challenges
• Existing systems for input and output harmonization are
not sufficient
• Developing a questionnaire that works in all countries
and languages
– Concepts, questions, translation, interpretation
• Extensive quality control and supervision
• Varying methodological and financial resources
• Increased distance between user and producer
Issue No. 3
The Process View
• Traditional large-scale evaluations are expensive and
results come too late
• Small-scale evaluations must be conducted to get
estimates of error components (gold standard, latent
class analysis, responsive designs, multi-level modelling)
• Long-term improvements are achieved via improved
processes controlled by paradata
Generic Control Chart
Upper control limit (UCL)
The central
limit (CL)
Lower control limit (LCL)
Time
Understanding Variation (I)
Common cause variation
• Common causes are the process inputs and conditions
that contribute to the regular, everyday variation in a
process
• Every process has common cause variation
• Example: Percentage of correctly scanned data, affected
by people’s handwriting, operation of the scanner…
Understanding Variation (II)
Special cause variation
• Special causes are factors that are not always present in
a process but appear because of particular
circumstances
• The effect can be large
• Special cause variation is not present all the time
• Example: Using paper with a colour unsuitable for
scanning
Action
• Eliminate special cause variation
• Decrease common cause variation if necessary
• Do not treat common cause as special cause
Standards
• Purposes
– To control processes, variability and costs
– To improve comparability
– To define a minimum level of performance
• Examples
–
–
–
–
Classification
CBMs and checklists
Standard Operating Procedures
ISO
Problems with Standards
• They must be adhered to
• They must be maintained and updated
• In stovepipe systems it’s easy to find excuses to deviate
• Standard, policy, guideline, best practice,
recommended practice……?
Issue No. 2
The User
In place:
Problems:
• The principle of openness
(OMB 1978)
• How should quality information
be communicated?
• Responsibility to inform users
(many agencies in the 70’s)
• How do we distinguish between
different kinds of users?
• Dissemination procedures
• How do users and producers
use quality information and
metadata?
• Customer satisfaction and
image surveys
• Councils and service level
agreements
• How do producers and users
collaborate on fitness for use?
(ABS)
Issue No. 1
Image Is Everything
A.
Eliminate special cause variation
B.
Decrease common cause variation if necessary
European Conference
on Quality in
Survey Statistics