Application of a DQAF for Education Statistics

Brian Buffett ([email protected]) Conference on Data Quality for International Organisations Newport, Wales April 27 - 28, 2006

www.uis.unesco.org

Outline

  Assumptions Introduction   Overview of the DQAF The DQAF in practice  Observations & Summary www.uis.unesco.org

Assumptions

 Familiarity with the concept of Data Quality Assessment Frameworks;  Awareness of the IMF DQAF; www.uis.unesco.org

Introduction

1999 ……….

2001 ……….

2003 ……….

UNESCO Institute for Statistics (UIS) was established UIS moved from Paris to Montreal, Canada Statistical Capacity Building Programme (SCB) introduced UIS Mission: • Internationally comparable statistics in UNESCO’s areas of competence: Education; Science & Technology; Culture; Communication Currently: • 100 staff members in Montreal • Regional Advisors in: Chile, Senegal, Thailand, Ethiopia, Samoa www.uis.unesco.org

Why a Statistical Capacity Building Programme?

 Statistical Capacity Building is one of the four Main Lines of Action in UNESCO Institute for Statistics statutes.

 The SCB programme serves two purposes: • It supports member states to meet their own needs for production and use of statistics in UNESCO domains.

• It supports primary UIS data programmes.

» Since countries are the source of UIS primary data – improving country data is an essential ingredient for improved international data.

www.uis.unesco.org

Why an Education DQAF?

 The SCB programme needed a mechanism in order to efficiently and effectively engage countries and assess the statistical systems within education ministries.

 Within education ministries, what was often lacking was not awareness of the need for better data and statistics – but an awareness of

what

the problems were and a road map for

how

to go about addressing them.

 The UIS desired a broad framework that focussed on the quality related features of the

governance

of statistical systems, their core statistical

processes

, and their statistical

products.

www.uis.unesco.org

The IMF DQAF

 The IMF DQAF is not limited solely to timeliness and accuracy  Six dimensions to the IMF DQAF: • Prerequisites of quality • Integrity • Methodological soundness • Accuracy and reliability • Serviceability • Accessibility www.uis.unesco.org

Why extend the IMF DQAF?

  The DQAF seemed to meet the overall requirements; The work was reduced to a fraction – and domain specific;  Cost and timelines to implement were attractive;  The IMF was willing; www.uis.unesco.org

Extending the DQAF to Education

 Developed in 2003 in collaboration with the World Bank.

 Addresses: • International Standards and Classifications (ISCED) • Best practices and guidelines specific to education • Verifies statistical system measures and reports on: » Structure and normative characteristics of education system » Supply of education » Demand for education » Quality of learning outcomes » School environment » … www.uis.unesco.org

The DQAF in Practice

 Used to diagnose the situation of national information systems on education, paying particular attention to national information needs.

 These diagnoses are a major element to devise action plans to strengthen national capabilities on education statistics.

 International reporting requirements are addressed but not the primary objective.

 Flexibility in developing action plans: • If there are significant problems related to international data reporting, ISCED is a critical element; • if the major problems are related to nationally-specific challenges, other items are addressed.

www.uis.unesco.org

The DQAF in Practice

 How have the diagnostics been carried out?

• Weighted the DQAF components and developed a scoring guide; • Development of common methods and best practices; • Scoring is to a significant extent an expert judgement. Diagnostic missions carried out by a small number of trained staff using common methods.

 Regional activities can be facilitated by ensuring coherence across countries www.uis.unesco.org

The DQAF in Practice

  Will result in systematic diagnoses under a common framework for half of the region.

 In Latin America and the Caribbean region, by the end of 2006: • Completed in Honduras, Ecuador, and El Salvador; are in revision in Costa Rica, and Nicaragua, and being prepared for Guatemala, Uruguay, Paraguay, Peru and Colombia; • Are being prepared for a similar number of Caribbean countries.

The identification of common challenges will: • permit grouping of countries; • provide the basis for country-to-country cooperation;  Will have the necessary information to support national as well as regional purposes.

www.uis.unesco.org

Summary – the Country Perspective

 A useful tool to help strengthen the country’s statistical system by identifying the strengths and weaknesses of the system as well as areas to be improved.

 Some results: • adoption of new questionnaires better responding to country information needs; • improved collection methodology – including training of respondents; • more timely completion and return of completed questionnaires; • more efficient data capture – with edits to ensure data quality; • more efficient processing of the data and production of outputs; • improved access to data; • training of statisticians and policy makers in use and interpretation of the data.

www.uis.unesco.org

Summary – the UIS Perspective

 Reduced resource costs, timeframe, and skills required for framework development; Able to focus resources and efforts on subject-matter specificities;    The framework and diagnostic method are effective; Initial results have been achieved more rapidly due to this approach; Country quality reports on education can be comparable - providing more flexible approaches to capacity building and increasing country-to-country cooperation;  Common best practices need to be followed, such as: • Country ownership; • Broad involvement; • Assessments combined with audits; • Consistent application of scoring guides.

 UIS will benefit from future IMF work on the DQAF; www.uis.unesco.org

Summary – the International Perspective

 An example of collaboration and reuse of existing methods – adapted to International Org. environment/needs;  Factor country needs and situation into any approach;  Statistical activities outside of NSO’s can benefit from the same practices as NSO’s;  A broad definition of quality is important; www.uis.unesco.org

Thank You www.uis.unesco.org