A tool for a snapshot assessment of a national statistical system

Quality conference – 3-5 June 2014 Vienna Claudia Junker, Head of Unit "Statistical Cooperation", Eurostat 1

Content of the Presentation

1. Background 2. Objectives 3. Presentation of the tool 4. Filling in the tool 5. Using the tool 6. Conclusions

2

Background

 Original context is development cooperation, but the tool can be used also in a wider context  Results-based management and performance assessments are central in development  The monitoring of development policies and of aid delivery (and Budget support in particular) require the setting of benchmarks and the construction of systems of indicators  These benchmarks and systems of indicators are fed with public statistics which are provided by the countries themselves  Countries need baseline and benchmarks for assistance projects and their own development

=> What do we know about the quality of these statistics and how can we assess it in a user-friendly way?

3

Objectives:

To design a user-friendly tool to help different stakeholders to assess:  the quality of the data donors receive from the partner countries to monitor the results achieved by the national policies and reform programmes,  the performance of the support given by the EC to these policies and reform programmes,  The success of assistance projects visible in corresponding official statistical data  The state-of-play in the development of the NSS in a country,  The efforts to improve the performance of the NSI/NSS

=> A natural vocation for supporting quality 4

Eurostat's commitment to quality statistics

   European Statistics Code of Practice Regular self-assessments by the EU member states Peer reviews and Global assessments with partner countries (East and South)  Support to the African Charter on Statistics and similar regional initiatives

=> A natural vocation for supporting quality 5

Presentation of the tool

The tool consists of two modules: 

Module 1

: analyse and document the functioning of the national statistical system (NSS) as a whole, with a special focus on institutional, legal and organisational aspects. 

Module 2

: analyse and document the statistics in specific sectors. A wide range of sectors can be addressed (such as: education, health , agriculture, food security, private sector development, climate change and environment, economic growth, governance, infrastructure, external trade of goods, employment, living conditions, macro economic stability, population).

6

Presentation of the tool

Module 1: Overall National Statistical System (56 questions)

Themes Mainstreaming Organization Resources Quality Users Content Legal, institutional and strategic foundations (i.e. legal and institutional framework for the whole NSS, , integration and consistency of the statistical development strategy in the national development strategic framework) Organisation and coordination of the NSS Adequacy of resources for statistics in the NSS (i.e. personnel in adequate quantity, quality of the personnel involved in statistics, equipment and infrastructure, financing) Determinants of data quality (i.e. quality commitment, professional independence, impartiality, objectivity, methodology and appropriate statistical procedures) Relations with users (i.e. relevance, accessibility)

TOTAL

Number of questions 13 14 12 13 4

56 7

Presentation of the tool

Module 1: National Statistical System (56 questions)

     

Examples of questions:

 Are statistics used in the development, implementation and monitoring of national development policies?

 Are there statements made by high-level officials (president, ministers, parliament, etc.) on statistics development?

Is statistical independence a guiding principle for the functioning of the NSS?

Does the NSS have mechanisms for assigning of responsibilities in the national statistical process?

Is there a human resources policy (recruitment, career development, education and training) for the NSI?

Which proportion of the staff has a (permanent) internet connection?

Are there internal handbooks / guidelines / recommendations for the statistical production processes?

Are there formal processes in place to consult users about their statistical needs?

8

Sector matrix

Presentation of the tool

Development sectors / Issues Health Education Dimensions Corresponding statistical sectors

Health statistics Education statistics

Statistics producers (NSS)

Ministry of Health Ministry of Education (Primary, Secondary, Tertiary)

Sources

DHS, MICS Administrative data from health structures Administrative data from education structures Population data

Agriculture Employment External trade of goods

Agriculture (economic accounts, price, structures, production) Offer and demand of work (balance between opportunities and capacities) Labour market statistics Employment statistics External trade statistics Ministry of Agriculture, Ministry of Rural Development, Ministry of environment NSI Employment agency, Social security agency Ministry of Labor Ministry of Budget NSI Customs Partner countries Agriculture census and surveys Farm register Population census Household survey (Labour Force Survey) Administrative. data Public employment Customs records

Main related indicators

Child mortality Maternal mortality Main endemic diseases Budget share Other WHO and UNICEF indicators Literacy rates Enrolment rates Budget share Other UNESCO indicators Production Costs Prices Other FAO indicators Active population (by gender) Employment rates Unemployment Other ILO indicators Imports and exports (quantities and values) Terms of trade indexes

9

Presentation of the tool

Module 2: Sector statistics (between 59 and 61 questions) Themes Part 1 Framework Resources Quality Content Sector information system assessment

Legal, institutional and strategic foundations (i.e. legal and institutional framework of the concerned sector information system, integration and consistency with the strategic framework supporting sector statistics development) Adequacy of resources for statistics within the sector information system (i.e. personnel in adequate quantity, quality of the personnel involved in statistics, equipment and infrastructure, financing) Determinants of data quality (i.e. quality commitment, professional independence, impartiality, objectivity, methodology and appropriate statistical procedures)

Number of questions One data producer Several data producers

7 12 22 8 9 22 11 11

Users

Relations with users (i.e. relevance, accessibility)

Part 2 Indicators Assessment of key indicators

Set of key criteria to roughly assess the quality of the indicator 9 9

10

Presentation of the tool

Module 2: Sector statistics

  

Examples of questions:

  Is the development of sector statistics included in the Sector Development Strategies? Which proportion of the staff involved in producing sector statistics is equipped with a modern computer?  Is information on the methods and procedures used for the sector statistics published on the NSI or Ministry website? Do all users get access to statistical releases at the same time? Are there partnerships or joint studies/works with the scientific community? Are their quality controls and checks on the reporting made by the concerned administrative units (schools, health centres …)?   How much of the statistical outputs in the sector are available via the internet? Is there a contact point to assist users?

11

Easy colour coding:

Presentation of the tool

Far from complying with quality requirements Not totally complying with quality requirements Complying with quality requirements Objective criteria – Column in the matrix with management comments for filling in the tables 12

Additional coding:

Presentation of the tool

Recent progress More subjective elements No changes Deterioration 13

Presentation of the tool

Module 1 LSB web-site: www.nsc.gov.la

Theme 1: Country strategy for statistics Q1 : Statistical development mainstreamed as part of the national development policy Detailed questions

Are statistics used in the development, implementation and monitoring of national development policies?

Was statistical development addressed as part of these development policies?

Potential sources Situation

Some evidences found in the main policy papers. However, the reference to official statistics is not systematic and doesn't seem to be articulated around a clear diagnostic and a comprehensive data framework.

No evidences found in the main policy papers. No additional information from the field mission

Color coding

Partially No

Remarks

Statistics should be an integral part of the national development policy, including for the design, monitoring and evaluation of Poverty Reduction Strategies, sector strategies, and other national development plans, as well as assessing progress made towards the MDGs. Most of the policies are linked to the achievements of the MDGS.

Perspec tive coding

↑

Identified sources

Socio-economic Development Strategy 2020, 7th National Socio-economic Development Plan NSEDP 2011-2015, SDNSS 2010 2020

Initial assess ment

↑

Management of the table

Look for evidence of whether statistics (i) have been used to design key country policy documents (diagnostics of the situation at macro and sector levels, indicators of achievement) and (ii) are addressed in these documents (attention given to the development and improvement of statistics systems at macro and sector levels). In some cases, the strategic matrix doesn't build on quantitative indicators. It may be necessary to go back to the preparatory studies.

Are there statements made by high-level officials (president, statements, Government decisions, ministers, Official National parliament, etc.) Budget, on statistics NSDS, development?

National policy documents Several statements and articles mentionned during the field mission by the Vice Prime Minister and other high ranking officers from the government Is the Statistical Day/Week celebrated? (SRPC), Reports in the media Yes, statistical week in October 2010. A leaflet on the SDNSS has been prepared to this occasion.

Yes, but occasionaly Not regularly Statistics are included in the Government web site but under "Information".

No plan for further celebrations mentioned during the field mission. Additional clarification to put through the LSB or to find in the WB appraisal report During the latest celebrations, was the government represented? The Vice Prime Minister opened and closed the ceremonies. This level is considered high in the lao context.

President or Prime Minister represented It is not clear if other high level official participated in other events than the openning and closing sessions What is the degree of openess vis-a-vis external partners-donors or peers on statistical issues?

Assessment of the capacity to produce core economic Updated indicators - Study made in July 2011 joiintly by the Lao GDDS or recent Statistics bureau and the statistical Division of ESCAP (draft version) review/asses sment Assessment by ESCAP July 2011. The obstacle to the adhesion to the GDDS must be looked for at the level of the line Ministries and not of the LSB → → Look in official documents, newspapers and other media for evidence of interest of decision makers in statistics.

↓ → ↑ Discussion with the LSB Deputy DG on November 13, 2012 ↓ Regularly' means that the statistical day is celebrated each year or at least every two years.

→ ↑ Yes' doesn't automatically mean green. Consideration must be given to the authority level of the Government representative. President or Prime Minister means green.

Find reviews, studies, assessments of the statistical system that have been made with outside partners or relations established with international systems for assessing statistical data quality (GDDS/SDDS and peer reviews or global/sector assessments mainly).

14

Presentation of the tool

Module 2 Part 1 - Assessment at Sector/Ministry level EDUCATION Theme 1: Legal, institutional and strategic framework at the level of the sector (both the NSI and the sector Ministry) Q1: Legal and institutional framework supporting the production of the sector statistics Detailed questions Potential sources Situation Color coding

Is there a law or an official agreement that organizes efficiently the production of sector statistics?

Does the National Statistical Office or other body responsible for statistics in the sector have access to the relevant administrative data?

No specific law. But there is a practice withinthe Minsitry of Education that has been established during the last 20 years and that involves the schools, the districts and the provinces. There are also regular relations with the LSB (training, consultation on statistical operations) Statistical laws, Government decisions, Memoranda of Service Level Yes. No mandate is Understanding, specifically written, but in practice, data are collected Agreements, regularly from schools and Questionnaire villages.

to DG Yes partialy

Remarks

The law addresses the rights and responsibilities of the line ministries for the production and the dissemination of statistics. A decree from the Ministry of Education established the Information Centre (role and responsibility) in 2006. There will be a new decree soon as ICT and statistics will be split in two different bodies.

Perspec tive coding

→

Identified sources

Statistical law The system is managed by the Education Statistics and Information Technology Center of Education and Sport (SITCES). The system is improving year after year.

↑ Interview with Deputy DG SITCES

Initial assess ment

→ ↑

Management of the table

In most case, there will not be a specific law at the sector level. It is generally expected that the relations between the NSI and the other data producers in a country are driven by memorandum of understanding that sets the respective responsibilities, including data sets to be exchanged and calendar. Is the actual work organization (mandate and attribution) in line with the law or the official agreement?

The LSB is responsible for the coherence of official statistics mainly through its responsibilities in the Yes management and direction of the NSS. The Ministry of Education gather annual data from schools and villages.

See Article 32 of the statistical law. → Statistical law → To check if mechanisms (communication, dialogue, exchange of data …) that ensure coherence of methods and outputs between the different structures (NSI, sector Ministry, ) involved in the production of sector statistics

15

Presentation of the tool

Part 2. Quality assessment – at the level of Indicators INDICATOR 1: MinJus - Number of judges Detailed questions Potential sources Situation Colour coding Remarks Perspective coding Identified sources Initial assessm ent Management of the table

Are there more than one national indicator on this issue?

No. Data produced by the MinJus No Is the definition used in line with the international recommendations? Is the methods used in line with the international recommendations from? Questionnaire to the statistical service in charge, Compliance report National methodology No No If the methodology has changed in the last years, are the changes documented and old estimates recalculated according to the new methodology? or No change since 2009 Methodology changes are not a problem as far as the changes are documented, properly explained to the users, the revised methodology is applied to old series/data.

No information available How long is the time series for this indicator Since 2009 LSMP 2009-2020 Check if it is a one shot production or a sustainable one well integrated in the national system.

Are the sources used and their coverage in line with the international recommendations? Vertical informations system No Is the data frequency in line with the international recommendations? Is the indicator timely enough for the use made by the EC?

Survey data (Demographic and Health Survey RHS) – DHS, Multiple Indicator Cluster Survey – MICS, Reproductive and Health survey Is the indicator publicly available and easy to find?

Reports from Provinces on a quarterly basis Data available for the annual Round Tables Certainly available for the Round Tables but not directly to the Public Yes No

16

Filling in the tool

What are the basic sources?

Aspects linked to the legal and policy framework, to the organization and functioning of the system For Module 1 (mainly themes 1 and 2)

Country macro-economic and social plan/strategy (Growth and poverty alleviation strategy or any similar policy document) National Strategy for the development of statistics (NSDS) Generally available on the Government web site Information documented and regularly updated on the Paris 21 web site. Also generally available on the NSI web site The statistical law and related application decrees Other information on the NSS (coordination, exchanges among the data producers/users) Generally available on the NSI web site

For Module 2 (mainly theme 1)

Sector strategy documents/plans Information on the Sector Information system Web sites of the related sector Ministries

17

Filling in the tool

What are the basic sources?

Aspects linked to practices: complements to the basic sources

For all modules (mainly themes 3, 4 and 5 for Module 1 and themes 2, 3 and 4 for Module 2) Reports by the NSS (annual reports/plans, self assessments, studies, guides and methods…) Reports by other donors Contacts/discussions with the actors of the NSS Not always available on the web (paper documents) Not always easy to identify and to gather particularly when there is no coordination mechanism for statistics Direct knowledgeable source but sometimes

18

Filling in the tool

Who will do it?

In the framework of development cooperation

Staff from the EC Delegation or other donors alone Experts statistician mobilized by the EC Delegation or other donors EC Delegation with other donors EC Delegation or other donors and National Statistical System (NSS) EC Delegation, other donors and NSS NSS alone Even if only part of the question are answered, the tool can give useful indications, particularly if the objective is to support statistical improvement in areas most in needs This could help for the first exercise, repetition over the years being done then by the Delegation Filling in the tool could be the result of a coordination and information sharing exercise with the donors most interested and involved in the sector This would be the best way to initiate and feed a regular and articulated dialogue on the strengths and weaknesses of the NSS and to potentially open the way to concrete support Same as the above but involving all the interested donors. In some countries coordination for statistics already exist and could be fed with the results form the tool This would be the result of a full appropriation of the tool by the NSS. Certainly the best solution in the long run.

NSS alone

In the framework of the NSS

Experts statistician mobilized by the NSS This would be the result of a full appropriation of the tool by the NSS. Certainly the best solution in the long run.

This could help for the first exercise, repetition over the years being done then by the NSS

19

Using the tool

Results Module 1: example

8 7 6 1 0 3 2 5 4 Module 1 - Global results

20

Results Module 1: example

Strategy exists but not fully mainstreamed in policies 2 1 0 8 7 4 3 6 5

Using the tool

Gap in resources allocated to statistics Module 1 - Global results Legal framework in place but poor coordination No rigor in quality controls Poor relevance of and accessibility to data

21

Results Module 1: example

Using the tool

Module 1 - Detailed results

22

A snapshot of an NSS – using the tool

Results Module 1: example

The law is very Lack of trained and experienced personnel Share of responsibilities and coordination is established but poor execution in practice poor internet access Limited information available on financing Number of PC ok but Sectors with no handbooks and no systematic quality controls comprehensive but the notion of confidentiality is not yet fully internalised Module 1 - Detailed results No rules for the correction of serious errors No release calendar Participatory process for the design of the strategy but not translated into wok plans No cooperation with academics No systematic consultation with the users GDDS not joined, only a few statements by high level officials on statistics Poor usability of the web site for data dissemination

23

Using the tool

Results Module 2: specific indicators

33% Number of judges 22% 45%

24

Using the tool

Results Module 2: specific indicators

Only one producer and availability on time Number of judges Definitions and sources not in line with international recommendations 22% 33% 45% Only short term series and data not easy to find for users

25

Using the tool

An instrument of statistical quality assessment for the EU Delegations

 The tables could be filled and updated regularly as a monitoring instrument for the EU Delegations to assess the development of the statistical system of partner countries

26

Using the tool

An instrument of dialogue on statistics between donors (EU or others ) and the partner countries

 The tool could be embedded in annual reviews or monitoring exercises that could be done in the framework of a political dialogue between the donors and the country authorities,  The tool could ease identifying statistical areas in need to be supported by the donor community and help defining assistance programmes

27

Using the tool

An instrument of assessment of the progress made in the NSS

 Filling the tables could be the result of a coordination work within the NSS. The results could be presented and discussed each year (regularly) at a meeting of the National Statistical Council or a similar entity. This would help adjusting work plans.

 Examples and good practices could be drawn from exercises carried out in other countries.

28

Conclusions

What the Snapshot is: In photography

: a photograph that is "shot" spontaneously and quickly.

The tool is a complex camera (comprehensive and articulated framework) that allows shooting spontaneously (the tool is available and can be used/understood by non-statisticians) and quickly (the tables can be filled in rapidly with available information)

In computing

: the capture of an object's state at a specific moment in time The tool gives a picture of the statistical system of a country (or of sector statistics in a country) at the moment when the related tables are filled in. This picture can be taken at regular intervals (e.g. yearly) in order to monitor the progress made

29

Conclusions

What the Snapshot is not:

An instrument for rating and/or comparing countries  The tool is “process-oriented” and aims at supporting a dialogue among the various actors involved in the development of statistics in a country A detailed analysis of the sector statistics  There are more sophisticated and targeted frameworks for some statistical sectors (such as Education/UNESCO) that detail the statistical processes leading to data production in more depth. The tool can be fed from these in-depth analyses and can also be an initial step before going deeper into the analysis.

30

Conclusions

 Freely available from Eurostat upon request to send at: ESTAT [email protected]

   Available in three languages: French, English and Spanish Manual and a user-guide E-learning training course on how/why to use it

31