Transcript THE USE OF EVIDENCE TO IMPROVE EDUCATION AND SERVE THE PUBLIC GOOD? UNIVERSITY OF CAMBRIDGE 1 NOVEMBER 2012 Adrienne Alton-Lee, PhD, Dip Tchg (Distinction) Iterative.

THE USE OF EVIDENCE TO IMPROVE EDUCATION AND
SERVE THE PUBLIC GOOD?
UNIVERSITY OF CAMBRIDGE
1 NOVEMBER 2012
Adrienne Alton-Lee, PhD, Dip Tchg (Distinction)
Iterative Best Evidence Synthesis Programme
Hei Kete Raukura
Ministry of Education
New Zealand
www.educationcounts.govt.nz/goto/BES
BEYOND PLATO’S MYTH OF THE METALS’
• the role of teachers as one of
maintaining inequalities in society by
educating children differently
• gold, brass or iron education for gold,
iron and brass children
• Sorting via education as a peaceful
alternative to a military regime in
maintaining differential status and
access to material wealth in society
What works for valued outcomes for diverse
(all) learners?
Why?
How?
What doesn’t work?
First do no harm in education
What makes a bigger difference in areas of
need?
FIT-FOR-PURPOSE METHODOLOGY
• Influences on valued student outcomes
• Trustworthy bodies of evidence
• Rigorous eclecticism – iterative process
• Comparative magnitude of impact
• Effect sizes – impact & equity index
• Critical role of theory
• Case & vignette
• Context matters
INQUIRY & KNOWLEDGE BUILDING CYCLES
EARLY BEST EVIDENCE SYNTHESIS ITERATIONS
The complexity of community & family influences e.g.
•
Poverty
•
Pre-school access to experiences
•
Undiagnosed hearing loss
•
Family interactions
•
Television
•
Effective school-home partnerships
COLLABORATIVE KNOWLEDGE BUILDING & USE FOR
ONGOING IMPROVEMENT
Research
Policy?
Practice
PRIMARY TEACHERS’ UNION
NZEI Te Riu Roa welcomes the opportunity to comment
on this Best Evidence Synthesis.
Drawing widely and systematically from national and
international research on social sciences education, its
authors have sought evidence of what works, for which
students, and in what circumstances. The synthesis of
findings contributes to our understanding of the
relationship between pedagogy and outcomes and the
importance of context. Teachers will welcome the
many practical ways in which to strengthen practice – a
particular feature of the BES Programme.
National President, New Zealand Educational Institute, 2008.
SECONDARY PRINCIPALS’ ASSOCIATION
12,472
The Leadership BES already has significant traction in
New Zealand secondary schools and is well regarded
by the profession as being both aspirational and
practical in content. We are proud to have been
involved with it from the beginning and commend it to
you as a well-researched, clear and detailed way
forward for leaders at any level of the schooling system.
We hope it gives principals in particular a focus for their
work as well as being a useful tool to help us all find
ways to improve student outcomes.
President of the New Zealand Secondary Principals’ Association, 2011.
SCHOOL TRUSTEES’ ASSOCIATION
(SCHOOL GOVERNANCE)
...the final document reflects only a small proportion of the influence
that this work has had for those of us who have been involved in its
development. The process of developing the BES has triggered
new learnings. It has built stronger links, within and across the
sector, between academics and practitioners, and it has provided
springboards for new initiatives in leadership...best of all, we found
that ‘unpicking each section , as we reviewed it, raised questions in
our own minds and those of our colleagues and helped us to think
in a more disciplined way about what matters; our students, and
how leadership contributes to enhancing their achievements...
President, Manager Training and Development & Manager, Service
Delivery, New Zealand School Trustees Association
SCHOOL LEADERSHIP & STUDENT OUTCOMES: IDENTIFYING
WHAT WORKS AND WHY: BEST EVIDENCE SYNTHESIS
Ensuring an orderly & supportive
Ensuring an orderly & supportive
environment
environment
0.27
0.27
0.31
0.31
Resourcing strategically
Resourcing strategically
Planning, coordinating and evaluating
Planning, coordinating and evaluating
teaching & the curriculum
teaching & the curriculum
Establishing goals and expectations
Establishing goals and expectations
Creating educationally powerful
Creating educationally powerful
-0.04
connections
-0.04
connections
Promoting and participating in
Promoting and participating in
teacher learning
teacher learning
-0.2
0
-0.2
0
0.42
0.42
0.42
0.42
0.28
0.28
0.6
0.6
0.84
0.84
0.2
0.2
0.4
0.4
0.6
0.6
0.8
0.8
1
1
BACKWARD MAPPING
•
•
•
•
•
•
•
Ensure administrative decisions are informed by
knowledge about effective pedagogy
Engage in constructive problem talk
Engage in open-to-learning conversations
Analyse and solve complex problems
Select, develop and use smart tools
Build relational trust
Create a community that learns how to improve
student success
FORMATIVE QUALITY ASSURERS’
WARNING
We believe that any school leader, or system leader, or anyone with an interest
in education improvement or leadership, will find this report stimulating and
valuable. Certainly both of us did.
Having such a high quality review is an important accomplishment, but it is not
enough…The challenge for all partners in New Zealand (and beyond) will be to
make sure that the lessons and implications of this review leap off the pages
and become part of the fabric of education in the country.
The BES report on leadership will be for nought unless there is a concerted
plan to develop the core capacities of effective leadership in all New Zealand
schools. This for us would not mean merely deriving a plan from the findings,
but rather taking a concrete problem, such as raising the bar and closing the
gap in literacy in New Zealand schools and incorporating the key leadership
capacities into the implementation of the very problem to be addressed. It is
always better to start with the concrete and incorporate what is needed to
make specific improvements.
TEACHER PROFESSIONAL LEARNING & DEVELOPMENT BES
14,145
Timperley, H., & Alton-Lee, A. (2008). Reframing teacher professional learning: An
alternative policy approach to strengthening valued outcomes for diverse learners.
Review of Research in Education 32, 328 – 369.
1. Much teacher professional development has no effect on
valued student outcomes
2. Some professional development has negative effects
3. A comparative magnitude of impact analysis shows some
outstanding professional development can lift achievement for
all and accelerate progress of lowest 20% of achievers 3-6
times the rate of business-as-usual practice
http://www.ibe.unesco.org/en/services/online-materials/publications/educational-practices.html
Teacher Professional Learning and Development
What works
Focus on valued student outcomes
What doesn’t work
Assume rather than check influences on
valued outcomes for diverse (all) learner
Ensure worthwhile content
Integrate knowledge and skills
Use a skills-only focus without theory
Use assessment for professional
inquiry
Use assessment for sorting or labelling
Fail to develop adaptive expertise
Ensure multiple opportunities to learn
and apply in ways that ensure trust &
challenge
‘Sit and get’ US Teacher: “I hope I die
during an in-service session because the
transition between life and death would be
so subtle.”
1-2 years for deep change
Ensure responsiveness to teachers
By-pass teachers’ theories
Teacher Professional Learning and Development
What works
Provide opportunities to process new
learning with others
What doesn’t work
Little opportunity to learn with others
Professional communities that are not
engaging with impacts on learners
Knowledgeable expertise – external to
group of participating teachers
Ignore complexity of teaching
Low or narrow expertise risks –ve effects
Ensure active leadership of designated Fail to involve active support of leadership
leaders to create conditions for change for ongoing improvement
Maintain momentum through selfregulation Where am I going? How
am I doing? Where to next?
Move on to the next fad without integrating
the learning into practice
A PERSPECTIVE FROM THE AUDITOR GENERAL
3.21 Various stakeholders told us that the
Ministry did not consistently base its decisions
about funding and providing professional
development initiatives on the evidence it has.
3.22 In our view, it would be helpful for the
Ministry to review the professional
development initiatives it funds against its
BES evidence and any other relevant
evidence on effective professional
development.
Office of the Auditor General 2008
QUALITY TEACHING FOR DIVERSE (ALL) LEARNERS
• www.educationcounts.govt.nz/goto/BES
•See ‘Overview of Findings’ Cross-curricula
•‘The use of evidence to improve education and serve the public
good’
19,276 copies
•
requested in
NZ
2500 schools
To 140
countries
from
UNESCO
10,221
copies
requested
in NZ
EFFECTIVE PEDAGOGY IN SOCIAL SCIENCES
HIGH IMPACT PEDAGOGIES FOR MULTIPLE VALUED
OUTCOMES
•
Highly effective pedagogies in areas of need
•
Approaches that simultaneously advance a range of valued
outcomes – cognitive, well-being, social…
•
Developed through several iterations of collaborative
research & development
•
Disciplined innovation by the profession as a resource for
improvement
BES EXEMPLARS
• Response to feedback from the profession
• Explain the complexity of teaching and the how
• Use of student and teacher ‘voice’
• Not just teaching - professional learning and leadership supports
• An inquiry and knowledge building approach to improvement
• Implementation alerts
• Access to trusted resources, smart tools, DVDs, websites
• Access to Research Behind BES
Think
alouds
HIGH IMPACT PEDAGOGIES
ACCELERATED IMPROVEMENT IN AREAS OF NEED FOR
UNDERSERVED OR DISADVANTAGED LEARNERS
BES Exemplar 1: Developing Communities of
Mathematical Inquiry Dr Roberta Hunter
• Students are scaffolded to engage with the
teacher and peers in mathematical inquiry,
reasoning & argumentation
• 4-5 years of achievement acceleration in one year
• ‘Don’t ‘dis her, man, when she’s taking a risk.”
BES EXEMPLAR 1: DEVELOPING COMMUNITIES OF
MATHEMATICAL INQUIRY
Ava’s Class
60
50
40
30
20
10
0
55
46
42
24
21
12
0
Significantly
lower
0
One level
below
0
At average
age-group
level
% Before
% After
Above
average
0
Well above
average
BES EXEMPLAR 5 LEARNING LOGS – JENNIFER GLENN
THAMES HIGH SCHOOL
Teacher feedback
Then – students read and respond to these questions….
What do you think I’ve said about your writing?
How do you feel about the outcomes/comment?
Set three goals for the next similar piece of writing?
Wendy: ‘I like using this blue book [learning log] because it makes
you really think about what you need to improve
John: ‘This is the best thing that ever happened to my writing’
Brian: ‘We cover things in class that are much more useful to me
and everyone else’
BES EXEMPLAR 5 LEARNING LOGS
Aggregate Achievement NCEA Level 1 English
60
56
50
47
46
40
Learning Logs Class
Comparison Class
School
National
46
38 37
39
30
27
21
20
13
10
5
0
Not achieved
Achieved
Merit
12
5
2 3
3
Excellence
LEARNING LOGS IN PHYSICAL EDUCATION
XANTHE SULZBERGER, AQUINAS COLLEGE
Over time, the emails became electronic “learning logs” that provided
evidence of learning and resources for ongoing improvement, for me as
well as the students. They provided me with rich information as to how
much the students understood, and what I may have to re-address in
class. I could see whether questions needed to be re-framed or concepts
needed to be re-taught. This sometimes meant that I had to adapt my
questioning or how I presented the question. The interaction became a
two-way process, with students asking me for clarification that they may
not have asked during class time’
We also feel more involved as parents from reading the logs, and it has led
to some great discussions around the dinner table with the family all being
interested in the topic. This all contributed, I believe, to our son getting his
first Excellence in a written assignment.”
Parent of student in Xanthe’s class
Xanthe Sulzberger
Physical Education Achievement Standard 2.2: The Badminton
Smash Shot
XANTHE’S USE OF LEARNING LOGS
STUDENT FEEDBACK
EDUCATIONAL POLICY, PRACTICE & EVIDENCE USE
Meta-analyses
ES 0.53 0.54 0.59 for cooperation
ES 0.36 0.23 for competitive or individualistic efforts
Stanne et al (1999); Hattie (2009)
In comparison with schooling practices that are often supported by governments –
such as tutoring, technology use and school restructuring – co-operative learning is
relatively inexpensive and easily adopted. Yet, thirty years after much of the
foundational research was completed, it remains at the edge of school policy. This
does not have to remain the case: as governments come to support the larger
concept of evidence-based reform, the strong evidence base for co-operative
learning may lead to a greater focus on this set of approaches at the core of
instructional practice. In the learning environments of the 21st century, co-operative
learning should play a central role. (Slavin, 2010, p. 174).
Galton, M., & Hargreaves, L. (2009). (Eds). Group work: still a neglected art?
Cambridge Journal of Education, 39(1) 1-6.
THE NEW FRONTIER: THE USE OF EVIDENCE IN POLICY
“Research evidence is perceived as “softly spoken”
because, “empirical inquiry simply cannot make its
voice heard amidst the clatter of other, political
imperatives on policy making” (Pawson, 2002).
RESEARCH INTO HOW EVIDENCE IS USED IN POLICY
COBURN, TOURĒ & YAMASHITA (2009)
• A strong tendency to discount evidence when it does not support the
beliefs decision-makers hold
• Orchestrated consensus to over-ride action on the available evidence
• Extent of use of evidence depends on individuals with positional
authority
• Evidence brought to the resolution of disputes ‘rarely addressed
assumptions about high quality instruction or about how children learn’
•In times of decision-making under pressure and/or contracting
resources ‘decision-making trajectories became more interrupted, last
minute and symbolic’
• ‘symbolic’ = the use of evidence to give legitimacy to a policy position
rather than to specifically guide improvement efforts
POLICY FOR THE USE OF EVIDENCE IN EDUCATIONAL
POLICY
•
•
Little research on linking research to policy in education
Passive dissemination of research is generally ineffective whilst
multi-faceted interventions targeting multiple barriers more effective
Recommendations e.g.
• Knowledge, awareness and skills capacity building in all parts of the
research evidence production-to-use system
• Policy decisions to develop evidence informed policy in education
• Increasing capacity in research on research generation and use
AN EFFECTIVE, SYSTEM-WIDE CHANGE STRATEGY REQUIRES
THE FOLLOWING ELEMENTS:
1.
2.
3.
4.
5.
6.
7.
8.
A small number of ambitious yet achievable and well-grounded
goals, publicly stated
A positive stance on improving all schools and success for all
students
An emphasis on capacity building and a focus on results
Multi-level engagement with strong leadership and a ‘guiding
coalition
Continuous learning through innovation and effective use of
research and data
A focus on key strategies while also managing other interests and
issues
Effective use of resources
A strong implementation effort to support the change process.
USE EVIDENCE TO IMPROVE EDUCATION
AND
SERVE PUBLIC GOOD
Use evidence to strategically resource improvement in education
Focus on improvement in valued outcomes for diverse (all) learners &
accelerated improvement for those underserved or disadvantaged
Establish goals and expectations for improvement
Use trustworthy evidence about the what and the how of improvement
In times of fiscal crisis, give priority to leveraging the evidence of what
makes a bigger difference for multiple valued outcomes
Ensure knowledge of effective pedagogy drives improvement
Do no harm
USE EVIDENCE TO IMPROVE EDUCATION
AND
SERVE PUBLIC GOOD
Foster constructive problem talk, build relational trust & ensure
effective supports for improvement
Build commitment across stakeholders for aligned action to support
ongoing improvement in teaching and learning
Leverage all four areas of influence for accelerated improvement:
pedagogy, educationally powerful connections, professional learning and
leadership
Invest in collaborative R & D expertise as a driver for accelerated systemic
improvement in areas of need.
Develop adaptive expertise & smart tools
Scale up effective implementation of high-impact pedagogies across the
system.