Implementation of Quality Assurance Activities in Eurostat

Third CCSA Data Quality Conference Rome 7-8 July 2008

A. Baigorri, M. Hahn, H. Linden Eurostat

1

Structure of the presentation

 Quality assessments in Eurostat: methodology and implementation  Redesigned Quality Barometer: short term and long term perspectives  Plans to enhance quality assurance in the European Statistical System

2

Quality assessments methodology Each process is matched with an assessment category

Self Assessment Prod. unit: fills in Checklist, produces outputs DDG02: provides support on demand Supported Self-Assessment Peer review Prod. unit: in control of assessment, Same as Supported Self-Assessment participates in all steps with one or more peers (int/ext) DDG02: pre-fills the Checklist, taking part in all steps produces the outputs Rolling review A comprehensive review based on the checklist implemented by ext.contractor, involving partner & user surveys and cost-benefit analysis Process 1 Process 2 Process 3

Characteristics:

- Periodicity - Legal Basis - Output - ESTAT intervention -availability of quality report Process 4 Process 5

3

Quality Assessments Implementation Plan: Principles



Minimize burden for units

– – – Test of the approach in advance.

Support from unit DDG-02 Flexibility 

Build on existing information

– – Quality reports Process analysis 

Integrate QA with other horizontal activities 4

III. CONFORMITY

Integrating Quality Assessments

User Requirement Standards

Labelling Quality assessments Rolling reviews Evaluation function Cost/benefit 5YP II. EVALUATION

CVD steps

ASSESSMENT CHECKLIST

| | | | | | Quality Barometer

MEASUREMENT description Quality reports SDMX User satisfaction survey

Best practices Quality diagram Improvement actions

AMP Production processess Statistical products User perception Eurostat Institutional/Legal environment 5

Quality Assessments Implementation Plan: Support



Training

- in-house courses on data quality reporting - lunchtime presentations on the ESPAC - peer review results - seminars on quality assessments 

Communication

- intranet site on quality - regular information on Cybernews - Eurostat infos - dissemination of good practices best practices

6

Quality Assessments Implementation Plan: Agenda 2008

Sector

Quarterly non-financial accounts for general government Quarterly Institutional Sector Accounts Prevention of illegal migration and stay statistics Asylum statistics Waste and Recycling Statistics International Trade in Services M-3 and M-1 questionnaires Harmonised unemployment rates Household Budget Survey Quarterly financial accounts for general government ICT usage in enterprises Structural Business Statistics-A. 1 to 4 Annual Questionnaires Quantities Farm structure survey Electricity and gas prices Forestry statistics Meat and livestock statistics - livestock Meat and livestock statistics - slaughterings Meat and livestock statistics - production forecasts Meat and livestock statistics - supply balance sheets PRODCOM - Industry Production LUCAS Catch Statistics E-communication. telecommunication Landing Statistics Aquaculture Statistics Fishing Fleet Register (from DG FISH data base) Demographic data Food safety: Statistics on Control and monitoring Activities Food safety: Organic farming statistics System of health accounts (SHA)

Id

C5-4 C2-2 F1-5 F1-3 E3-1 C4-4 G4-3 F2-3 F3-4 C5-3 F6-2 G1-1 G4-1 E1-3 G4-4 E3-13 E2-4 E2-5 E2-6 E2-7 G1-6 E1-2 E2-16 F6-3 E2-17 E2-18 E2-19 F1-1 F5-2 F5-3 F5-4

QR

Y P Y Y Y P P Y Y-5 years Y Y-annual P L Y Y Y P/F P/F P/F P Y Y P N P N P N N N N

A Cat

SSA SSA SA SA SSA SSA SA SA SSA SSA SSA SSA SSA SSA SA SA SA SSA SA SA SSA PR SSA SSA SA SA SA SA SA SA SA

Estimated start

03/03/2008 23/03/2008 01/04/2008 03/04/2008 08/04/2008 14/04/2008 21/04/2008 01/05/2008 15/05/2008 19/05/2008 01/06/2008 01/06/2008 01/06/2008 01/07/2008 01/07/2008 04/08/2008 01/09/2008 01/09/2008 01/09/2008 01/09/2008 01/10/2008 13/10/2008 01/11/2008 01/11/2008 01/12/2008

Sector

Causes of death statistics (COD) ESAW: Accidents at work statistics Railway transport Air transport

Id

F5-7 F5-8 G5-2 G5-3

QR

N N F Y

A Cat

RR RR RR RR

Timing

February - October 2008 February - October 2008 April 2008 - March 2009 April 2008 - March 2009

7

Quality assessments in Eurostat: next steps



Evaluation of the 2008 program of assessments

Need for adjusting the instruments used Indentify recurrent weaknesses in statistical processes to feed improvement actions into the AMP Promote initiatives to be implemented by horizontal units to facilitate quality improvements by production units.



Elaboration of the 2009 plan

focussed on more objective assessments as peer and rolling reviews.

8

Redefined Quality Barometer: characteristics

Purposes: - to measure the quality level at both EU level and country level; - to monitor change; - for comparisons across statistical domains and across countries.

Users: - production side: top-management, departments/units, survey managers.

user’s side: international organisations, policy makers, researchers, etc.

Scope: - cover all types of statistical processes and their statistical outputs; - the quality of key data (or main time-series); - monitor quality on an annual basis.

9

Redefined Quality Barometer: short term perspective



Restricted QB based on quality indicators that are: - applicable to most statistical domains; - covering the main quality dimensions; - well defined and easy to provide; and - easy to interpret and understand.



The restricted set of Quality Indicators prioritised for the QB: - Rate of available statistics (R2) - Coefficient of variations (A1) - Unit response rates (A2) - Average size of revisions (A7) - Time lag reference period and date of first results (T2) - Number of accesses to databases (AC2) - Rate of completeness of metadata (AC3) - Length of comparable time series (C1) 10

Compilation of a restricted Quality Barometer



Including many statistical processes but only covering the restricted set of quality indicators



The Euro- SDMX Metadata Structure should be the main source of information



Continue the MS’s Expert Group and Eurostat network for methodology for indicators and analysis of results



First RQB planed for the end of 2008 11

Redefined Quality Barometer: long term perspective

I. INTRODUCTION IIa. Social statistics Labour Force Statistics

Relevance Accuracy Timeliness Comparability Coherence Summary for Labour Force Statistics

IIb. Business statistics III. OVERALL SUMMARY 12

Enhancing quality assurance in the ESS



Regulation on European Statistics and Article 10 “Statistical quality”:

Quality of the MS’s and Eurostat data shall be assessed according to the ESS quality dimensions.

- Specific quality requirements, such as target values and minimum standards for the statistical production, in sectoral legislation.



Euro-SDMX Metadata Structure (ESMS)

- Harmonizes quality reporting integrating the ESS quality dimensions - improves metadata sharing and information to users on quality of statistics 

Development and sharing of “best practices” for statistics production 13

Enhancing quality assurance in the ESS



Revision of the Eurostat/ESS quality reporting guidelines:

- Standard Quality Report - Handbook: How to make a Quality Report 

Facilitate the implementation of CoP improvement actions plans by NSIs

- Eurostat 2009 grants on the implementation of ESS standards, handbooks, guidelines and application of tools - ESS Quality Coaching focusing on the implementation of quality management and quality assurance

14

Thank you for your attention!

http://ec.europa.eu/eurostat/quality

15