Making self review work for you

Julie Foley & Dave Shepherd Education Review Office October 2010 [email protected]

Talk outline

• • Recapping concepts and activities from the 2009 presentation to the CPPA Schools’ self review and ERO’s 2010/11 reviews - how they can complement each other. • Continuing to build evaluation capacity in the sector – ERO’s strategic theme ERO self review presentation 2

Evaluation Capacity Building

ERO’s current thinking about internal and external evaluation

Ko te Tamaiti te Putake o te Kaupapa

ERO’s conceptual framework

•

Complementary evaluation

takes key aspects of internal and external evaluation, and accountability and improvement, and adjusts these to the relevant context. It is informed by other evaluation theories and aims to resolve the conundrum raised by Nevo (2000, p.28) “

Everybody seems to hate external evaluation while nobody trusts internal evaluation”

.

4

ERO’s Reviews 2010

An evaluation approach that balances external ERO review with school self review according to each school’s circumstances.

ERO REVIEW

Schools experiencing difficulties with very limited self review

SCHOOL SELF REVIEW

Schools operating well with established processes of self review Schools sustaining high quality performance and continuous improvement through effective self review 5

Self review is…

• Self review involves investigating evidence about student outcomes and current ways of doing things to find out where improvement is needed. Planning for school improvement requires schools to set goals and targets for better student outcomes and make changes that are necessary to bring about those improvements. (MoE Planning and Reporting, 2003) ERO self review presentation 6

Or…

• …the deliberate and on-going process of finding out how well our practice enhances children’s learning and development. Review allows us to see which aspects of our practice are working well and what we could do better. As a result we can make decisions about what to do to improve. Through review our practice is transformed, and ultimately, our children’s learning benefits. (MoE, ECE, 2006) ERO self review presentation 7

Or…

• Kura are most successful when there is an robust process of on-going internal review that leads to further improvement. Effective internal review focuses on the key goals for students and the evidence of the extent to which they are actually being achieved…[It is] rigorous, informed by excellence in evaluation practice, with findings based on evidence… (ERO, Te Aho Matua, 2008) ERO self review presentation 8

•

Or…

Self evaluation will collect and analyse a combination of qualitative and quantitative data that helps leaders and teachers develop themselves, their practices, promote high performance, help support student progress and develop intervention programmes from a proactive perspective. (Paphitis, n.d.) • Self evaluation is a dialogic process, a meeting of hearts and minds, a forging of new ways of doing, vital and continuing because it is the core of a school’s educational life. It is the essence of the learning community, the intelligent school, schools that learn. (MacBeath, 1994) ERO self review presentation 9

Summary. Self review is:

• • • • • • • Centred around giving all students the most successful educational experiences possible Set in a collaborative culture of continuous self reflection and constructive critique Self initiated by a desire to seek constant improvement – mostly planned but sometimes spontaneous Celebrating and sustaining what is working well while striving to improve what is not A willingness to embrace challenge and change Setting long term goals and mid-term targets and gathering rigorous and wide-ranging evidence to examine the extent to which these have been achieved ERO self review presentation 10

Different types of reviews but all linked to the big picture

ERO self review presentation 11

Balancing accountability and improvement

Eg Assurance statements INTERNAL Eg Self review ACOUNTABILITY Complementary evaluation IMPROVEMENT Eg External audit EXTERNAL Eg ERO education review ERO self review presentation 12

Turning data into evidence…

• • • Schools gather lots of

data

numerical) and ‘soft’; (qualitative/narrative). The raw or aggregated data is only the – ‘hard’ (quantitative/

first

level of analysis. It describes ‘what is’ or ‘what is happening’.

The

second information.

level of analysis takes data and turns it into It explains ‘what this means’. It allows schools to make key statements or comparisons.

The

third

level uses this information as

evidenc

e to support judgments (how well/to what extent), to make decisions (if this so then we need to…) and to prioritise (the most compelling need is…).

ERO self review presentation 13

On going review cycles…

Considering Informing Organising Monitoring Implementing ERO self review presentation 14

What are we learning from our workshops?

Schools do it differently: ERO self review presentation 15

• • • • • • • • • • •

What are we learning from ERO’s good practice in self review case studies?

Importance of leadership to drive the vision Importance of ‘buy-in” from all involved Belief that students can achieve Belief that teachers can make a difference Importance of the big picture Commitment to the long term Importance of prioritising Remembering to celebrate and build on successes Importance of gathering useful data and analysing it effectively Using strengths from in and outside the school There is no one way- each school and community is unique but we can learn from each other ERO self review presentation 16

Discussion

Developing your own definition – purpose statement

•

Mapping your review activities – quandrant diagram

•

Finding an overall structure that integrates your review activities

•

What process works for your school?

Recapping

• • • Thinking about self review Tools and strategies suggested Feedback – •

What did you use?

•

How well did it work for your school?

•

What changes have you made as a result?

•

Share ideas

ERO self review presentation 18

What ERO is finding in schools • Pivotal Questions for internal review •

What is so?

•

Why is it so?

•

So what?

•

Now what?

ERO self review presentation 19

Planning reviews •

What do we need to review?

•

Why?

•

How?

ERO self review presentation 20

Focus of reviews • Balance needed • What is on paper / documented • School practices • Outcomes for students

ERO self review presentation 21

ERO’s reviews

What is the place of school’s self review in 2010 and ahead?

Principles underpinning ERO’s reviews

 Enquiry focused  Responsive  Transparent  Collaborative & constructive  Building evaluation capacity 23

What do we know about effective schools?

• • National and international research, eg, BES ERO’s

Evaluation at a Glance: Effective Schools:

– A focus on the learner – Strong leadership within an inclusive culture – Effective teaching – Quality engagement with parents, whānau and communities – Coherent and consistent policies and practices within a cycle of continuous self review ERO self review presentation 24

ERO’s differentiated review cycle

• Rather than being based on a “one-size-fits-all” three-yearly cycle, the differentiated review cycle is based on ERO’s confidence that the school is providing a quality education for all students and maintaining an on-going process of continuous self improvement • Self review is the tool through which schools monitor and sustain this continuous improvement and provide evidence of what they are doing and what is working for their students ERO self review presentation 25

Deciding on Future Action

• • •

ERO intends to carry out another review over a period of one-to-two years:

Building capacity

ERO is likely to carry out the next review in three years:

Developing capacity

ERO is likely to carry out the next review in four to-five years:

Sustaining capacity

– – –

Criteria Process Moderation Trends

ERO self review presentation 26

ERO’s Reviews 2010

An evaluation approach that balances external ERO review with school self review according to each school’s circumstances.

ERO REVIEW

Schools experiencing difficulties with very limited self review

SCHOOL SELF REVIEW

Schools operating well with established processes of self review Schools sustaining high quality performance and continuous improvement through effective self review 27

Meta evaluation and design

28

Schools operating well with established self review processes School self reviewing & reporting to school community Scoping & review design phase

• using self review to set the scene • plan the evaluation, balancing the need for first hand evidence with evidence from self review

Building & using school capacity to:

use self review as the basis for external review

On site external review phase

• Gathering evidence • triangulate data • ongoing & collaborative synthesis use external review to test, affirm, strengthen & broaden self review

Reporting phase

• internal reporting • external reporting account for progress & achievement of goals, & generate further priorities for development & review

School self reviewing & reporting to school community

29

Schools performing very well with high quality self review processes School self reviewing & continuously improving & reporting to school community Using & building school capacity to: School builds & presents “a case” comprised of valid & significant examples of using self review to:

• maintain high levels of student progress & achievement • make sound decisions & build continuous school improvement

External review phase

• scope & design review using the school’s “case” • meta-evaluation of school’s self review • external reporting

School self reviewing & continuously improving & reporting to school community

challenge their own assumptions about the quality & efficacy of self review processes & findings engage with & use external meta evaluation to enhance self review 30

Schools experiencing difficulties

Development & review planning phase

• planning the action, evaluation & support to improve student achievement

Building school capacity to:

identify purposes & goals, & plan what & how to evaluate

Implementing phase

• implementing development • adapting & modifying (praxis)

Concluding phase

• internal reporting • external reviewing • external reporting systematically implement planning & use data to self review, adapt & improve account for progress & achievement of goals & generate further priorities for development & review 31

32

33

ERO’s updated evaluation indicators

• Six dimensions: – Student learning: engagement, achievement and progress – Effective teaching – Leading and managing the school – Governing the school – Safe and inclusive school culture – Engaging parents, whānau and communities ERO self review presentation 34

Focus on success for Māori

• • • Govt – MoE and ERO priority Based on clear evidence of disparity Expectations outlined in NAGs • •

How well is your school promoting / achieving successful outcomes for Māori?

What does your self review tell you about this evaluative question?

ERO self review presentation 35

• •

Support available

From ERO • Framework for reviews • Evaluation Indicators • • • • • • Good practice case studies (not yet available)

Evaluation at a Glance: Effective Schools

(not yet available) Board/Whānau Assurance Statement Checklist National Evaluations

Getting the most out of your ERO review

booklets Workshops and cluster meetings From other sources: • MoE self review tools for National Standards • • • • • MoE Kiwi Leadership material MoE governance resource Commercial tools Local advisors/consultants Each other!

ERO self review presentation 36

To conclude

• • • • • Be assured that self review is not new and much of what you already do fits under that umbrella Make on going self review, reflection and action a “state of mind” in your school Ensure smaller review cycles feed data into strategic self review in a coherent way ERO are not looking for particular ways of doing things but at the quality of the evidence and the robustness of the processes that you use for your decision making Ask yourself : “How do you know that you make a difference for every student in your school?” ERO self review presentation 37