Transcript OECD Review on Evaluation and Assessment Frameworks for

Evaluation, Accountability and
Improvement in Education: Perspectives
from the OECD Review on Evaluation
and Assessment Frameworks
ww.oecd.org/edu/evaluationpolicy
Paulo Santiago
Senior Analyst, Directorate for Education, OECD
Improving Education through Accountability and Evaluation: Lessons from around the World
International Conference
Rome, 3 October 2012
Outline
1. OECD Review on Evaluation and Assessment Frameworks for
Improving School Outcomes
2. Key themes
–
–
–
–
–
The rise of educational measurement and indicators development
The growing prominence of accountability as a purpose of evaluation
Establishing links to classroom practices
Building capacity for evaluation and assessment
Fostering synergies within the evaluation and assessment framework
1. OECD Review on Evaluation and
Assessment Frameworks for
Improving School Outcomes
OBJECTIVES
Purpose: To explore how systems of evaluation and assessment can be used to
improve the quality, equity and efficiency of school education.
A Review of national approaches to evaluation and assessment in school education
[student assessment, teacher appraisal, school evaluation, school leader
appraisal, education system evaluation]
The Review:
 Synthesises research-based evidence on the impact of evaluation and
assessment strategies and disseminate this knowledge among countries.
 Identifies innovative and successful policy initiatives and practices.
 Facilitates exchanges of lessons and dialogue.
 Identifies policy options for governments to consider.
Comprehensive approach: Investigation of each component individually, as well
as the coherence of the framework as a whole (including the links between the
different components).
METHODOLOGY
 Analytical strand
• Identifying the key questions for analysis and the background information needed
from countries
• Reviewing the literature and evidence on the impact of evaluation and assessment
procedures
• Gathering data on countries’ policies and practices (Country Background Reports)
 Country Review strand
• Country Reviews provide specific advice to individual countries.
• OECD-led Review Team with external experts
• The scope and focus is determined by the country in consultation with the
Secretariat
 Synthesis report
• Comparative report to analyse policy options and highlight good practices across
countries.
 Twenty six systems participating
•
Australia, Austria, Belgium (Flemish Community), Belgium (French Community), Canada,
Chile, Czech Republic, Denmark, Finland, France, Hungary, Iceland, Ireland, Italy, Korea,
Luxembourg, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Slovak
Republic, Slovenia, Sweden, United Kingdom (Northern Ireland).
2. Key themes
The rise of educational measurement and indicators development
Trends:
– The growing emphasis on measuring student outcomes; the proliferation of education
indicators; and the establishment of education targets
Potential:
–
–
–
–
Student outcomes as the focal point for analysis
Monitoring key student learning outcomes
Formative use of standardised tests
Hold stakeholders accountable
Challenges:
–
–
–
–
Ensuring breadth of performance measures
Securing fair and meaningful comparisons
Avoiding detrimental effects of uses for accountability
Preventing dominance of quantitative over the qualitative
Options:
– Ensure policy making is informed by high quality measures, but not driven by their
availability [the need to complement with qualitative analysis]
– Ensure a broad approach to national monitoring
– Clarity of purposes for the uses of standardised tests results
The growing prominence of accountability as a purpose of evaluation
Trends:
– Public reporting of school results (greater transparency, reporting requirements); rewards and
sanctions on the basis of evaluation and assessment results (e.g. teachers, schools).
Potential:
– Creation of incentives for improved performance – opportunity to recognise and reward
– Identification of underperformance
– Facilitates school choice
Challenges:
–
–
–
–
Range of potential detrimental effects (e.g. ‘teaching-to-the-test”, “narrowing of curriculum”)
Accountability function of evaluation not to hinder the improvement function
Conveys a ‘control’, ‘compliance’, ‘measurement’ concept of evaluation
Focus on accountability as a result of a strong top-down national vision of evaluation might
constrain the ownership of E&A by school agents
Options:
– Communicate rationale for evaluation and assessment: objective is improvement
– Build on a range of evaluation and assessment components achieving a variety of functions
– Publication of quantitative data to be perceived as fair by schools set in a wider set of
evidence (of a qualitative nature)
– Establish safeguards against overemphasis on student standardised tests (more relevant for
whole-school evaluation)
Establishing links to classroom practices
Trends:
– There is often focus on structures, procedures, programmes and resources in a top-down
approach
Potential:
– National agendas for education to be strengthened by consistency of evaluation procedures
– Ensuring equity of objectives
Challenges:
– Evaluation and assessment have no value if they do not lead to the improvement of classroom
practice and student learning
– Improvement function accomplished more at a local level – need to secure adequate links to
the classroom
– Risks that evaluation procedures do not place adequate focus on what is arguably the most
important area: teaching and learning
Options:
– Critical to ensure that the evaluation of teaching and learning quality is central to
evaluation procedures
– Build on teacher professionalism:
Teachers to have ownership of student assessment, build teachers’ ability to assess against educational
standards, teachers to be involved in school evaluation, emphasis on teacher evaluation for the
continuous improvement of teaching practices.
Building capacity for evaluation and assessment
Trends:
Evaluation and assessment policies often introduced with no due attention to capacity
development (e.g. competencies for evaluation and assessment)
Challenges:
Legitimacy of evaluators and accountability procedures
The effectiveness of evaluation relies to a great extent on ensuring that both those who design and
undertake evaluation activities as well as those who use their results have the proper skills and
competencies.
Examples of areas for capacity development: standardised test development; formative assessment; assessment
against standards; running systems of externally-based student examinations; analytical capacity in education
agencies to use system level information; data handling skills of school agents; formal evaluators of individual
school agents; competencies for classroom observation.
Options:
– Sustaining efforts to improve the capacity for evaluation and assessment
– Strengthen school leaders’ capacity for school development and instructional leadership
– Ensuring support from the centre and identification of best practice
– Need for a strong capability at the national level to steer evaluation
Fostering synergies within the evaluation and assessment framework
Trends:
– Most countries do not have an integrated evaluation and assessment framework but instead a
series of components operating at different levels that developed independently of each other
over time.
Potential:
–
–
–
–
Build synergies
Generate complementarities
Avoid duplication
Prevent inconsistency of objectives
Challenges and options:
Establishing articulations within the evaluation and assessment framework
• Within specific components of the overall E&A framework
e.g. teacher appraisal and teacher professional development; self- and external school evaluation.
• Between specific components of the overall E&A framework
e.g. teacher appraisal, school evaluation and school development; school evaluation and system
evaluation; school evaluation and the appraisal of school principals.
• Processes to ensure the consistency of E&A procedures
Moderation processes for teacher appraisal and teacher-based assessment
Clarity of responsibilities across the framework
Thank you for your attention!
[email protected]
www.oecd.org/edu/evaluationpolicy