Quality Indicators at the Federal Statistical Office Germany
Download
Report
Transcript Quality Indicators at the Federal Statistical Office Germany
Statistisches Bundesamt
DESAP - A New Self
Assessment Checklist
for Survey Managers
Manfred Ehling
Thomas Körner
Anja Nimmergut
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
DESAP
is the first generic checklist for a simple self
assessment programme for survey managers in the
European Statistical System (ESS)
goes back to recommendation no. 15 of the
Leadership Group (LEG) on Quality: “A generic
checklist should be developed for a simple self
assessment programme for survey managers in the
ESS”
applies to individual statistics collecting micro data
is fully compliant with the ESS quality criteria and
comprises the main aspects relevant to the quality of
statistical data
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Main components of quality („quality vector”)
in the ESS:
Relevance
Accuracy
Timeliness and Punctuality
Accessibility and Clarity
Comparability
Coherence
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Self assessments (DESAP) amongst other
assessment tools:
EFQM model:
tool of selfassessment for
the whole
organisation
(with help from
experienced
quality award
examiners)
Simple quality
checklists: focus on
the processes and
products; working with
these increases the
awareness of quality
issues and reveals
areas in need of
improvement (specific)
Documentation/ Quality reports: are
more detailed then simple checklist and
hence it is more effort to produce them
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Complement
any measure
or first step
before
Internal or
external
audits:
assessment
from outside
Statistisches Bundesamt
Objectives of DESAP:
Raising awareness for the quality components and survey
quality concepts
to provide a tool for a systematic, even though subjective,
assessment of statistical products and processes
to provide helpful guidance in the consideration of
improvement measures
Additional potential applications:
assistance for a basic appraisal of the risk of potential
quality problems
to provide a means for simple comparisons of the level
of quality over time and across domains
to provide support for resource allocation within
© Statistisches
Bundesamt, Institute
for Research
and Development
in Federal Statistics
statistical
offices
or for
the training
of new staff
Statistisches Bundesamt
Structure of DESAP: process-orientation
(1)
Decision to
undertake a
survey
(2)
Survey
Design
(3)
Data
collection
(4)
(5)
(6)
(7)
Data
Capture
and Data
Processing
Data
Analysis
and
Output
quality
Documenta
-tion and
Dissemination
Improving
statistics
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
24 Process elements of the checklist
(I)
(II)
(III)
(IV)
Decision to
undertake a
survey
Survey
Design
Data
collection
Data
Capture and
Data
Processing
(V)
Data
Analysis
and
Output
quality
(VI)
(VII)
Documenta
-tion and
Dissemination
Improving
statistics
(1) (2) (3) (4) (5) (6) (7)
Survey
concepts
Metadata
documentation
Analysis
Data
ofuser
Data
sources
needs
Relevance
capture
Adaptability
/ flexibility
Sample
deployment
Dissemination
strategies
Survey
objectives
Interviewer
design
Editing
Accuracy
procedures
Expertise
andin
training
relevant areas
Development
and
testing
of the
measurement
Data
management
Reducing
Imputation
Timeliness
non-response
procedures
Quality
and
punctuality
management
instruments
Fieldwork
Comparability
Coherence
Data analysis
Disclosure control
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Typology of DESAP questions
In order to fulfil the various function of the checklist different
types of questions have been developed and integrated in
the checklist:
questions with numerous response categories
(“improvement questions”)
assessment questions
open questions
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Questions with numerous response categories
sum up to an inventory for (data) quality assurance
measures in NSIs
make the process of completing the checklist less time
consuming because the survey managers just have to
tick the answer(s) that apply to their survey
regarding the different alternatives might stimulate some
ideas of possible improvements for the survey
production process
separate page where the ideas for possible
improvements can be noted down
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Example question with numerous response
categories :
5
Which of the following procedures are in place for user consultation?
Relevant governmental departments are involved in the planning process
Producer-user groups (e.g. sub-committees of statistical councils), discussing subject specific questions
Scientific board/ council
Customer survey exploring the needs of a larger group of users
Formalised agreements (e.g. service level agreements) with priority key user(s)
Available marketing concepts which mention the use of statistical information
Co-operation with partners in social- and economic sciences as well as market research
Informal contacts with relevant users
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Assessment questions:
questions are worded in a harmonised way, using the
same terms for the same quality levels
questions are highlighted using a special colour and
design (deep blue background and dark blue bar at
the side) in order to facilitate the evaluation
five point scale of the answers
score of the respective answers can be drawn in a
cobweb diagram which finally gives a graphical feedback of the quality profile for the respective survey
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Example for an assessment question:
6
How do you appraise the problem of overcoverage?
1 - Major overcoverage: An unacceptably high number of units not belonging to the target population
were included in the frame; or the overcoverage rate not known
2 - Considerable overcoverage: Many units not belonging to the target population were included in the
frame
3 - Some overcoverage: Some units not belonging to the target population were included in the frame
4 - Slight overcoverage: Few units not belonging to the target population were included in the frame
5 - Nearly no overcoverage: There is only a very small difference between the frame and the target
population
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
The DESAP assessment diagram:
Relevance
Coherence
Section V/28
Coherence within
the same socioeconomic area
Section V/27
coherence for
results of different
frequencies
Section V/2
user satisfaction
5
Section V/3
overall
relevance
Accuracy
4
3
Section II/6
overcoverage
2
Section V/25
comparability
over time
Section II/7
undercoverage
1
Comparabilit
y
Section V/24
comparability across
non-geographical
domains
Section II/8
misclassification
Accuracy
Section V/22
Publication
dates kept?
Timeliness
&
Punctuality
Section V/21
time lag
reference period/
publication
Section IV/4
necessity of
editing
The larger this area
- theSection
better
the
V/6
CVs for key
variablesprofile
quality
Section V/15 unit
Section V/18
item nonnon-response
Accuracy
response
© Statistisches Bundesamt, Institute for Research
and Development in Federal Statistics
Statistisches Bundesamt
Open questions:
sometimes not possible to give response categories
aim to confront the survey manager with pertaining
questions to develop improvement ideas
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Example for an open question:
12
If you would have 20% more or 20% less of your budget, what would you do?
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Items which could be solved in co-operation:
Experiences faced during the implementation of self
assessment programmes (joint workshop?)
Methods of evaluation esp. of the improvement
questions should be further developed (OECD or IMF
experiences?)
Are there similar problems encountered in the
framework of the OECD self assessments?
How could a comprehensive assessment (judgement/
grading) be achieved that gives a better feed-back to
the survey managers?
Could this problem be solved by refining/ tailoring
checklists for statistic areas following recommendations
© Statistisches
Institute for Research and Development in Federal Statistics
for Bundesamt,
best practice?
Statistisches Bundesamt
Recommendations of the DESAP group for
further steps concerning the checklist:
recommendation of the implementation of the
checklist among the members of the Statistical
Programme Committee (SPC)
checklist in all languages of the EU
broad publication of the project results
IT implementation of the checklist (electronic version
could facilitate the checklists use)
workshop concerning improvements of the checklist
after one or two years of experiences with its use
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics
Statistisches Bundesamt
Thanks for your attention!
Further questions?
[email protected]
[email protected]
[email protected]
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics