Transcript Exploring Assessment for Learning

Consider the Evidence
Evidence-driven decision making
for secondary schools
A resource to assist schools
to review their use of data and other evidence
3
Evidence-driven decision making
Page 1
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven decision making
Today we aim to
• think about a process for using data and other evidence to
improve teaching, learning and student achievement
• improve our understanding, confidence and capability in using
data to improve practice
• think about our needs and our own evidence-based projects
Page 2
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven eating
You need to buy lunch. Before you decide what to buy you
consider a number of factors:
• how much money do you have?
• what do you feel like eating?
• what will you be having for dinner?
• how far do you need to go to buy food?
• how much time do you have?
• where are you going to eat it?
Page 3
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven teaching
I had a hunch that Ana wasn’t doing as well as she could in her
research assignments, a major part of the history course. What
made me think this?
Ana’s general work (especially her writing) was fine. She made
perceptive comments in class, contributed well in groups and had
good results overall last year, especially in English.
How did I decide what to do about it?
Page 4
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven teaching (cont…)
I looked more closely at her other work. I watched her working in
the library one day to see if it was her reading, her use of
resources, her note taking, her planning, or what. At morning tea I
asked one of Ana’s other teachers about Ana’s approach to similar
tasks. I asked Ana if she knew why her research results weren’t as
good as her other results, and what her plans were for the next
assignment.
I thought about all of this and planned a course of action. I gave
her help with using indexes, searching, note taking and planning
and linking the various stages of her research.
Page 5
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Consider the Evidence
A resource to assist schools
to review their use of data and other evidence
What is meant by ‘data and other evidence’?
Page 6
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence
Any facts, circumstances or perceptions that can be used as an
input for an analysis or decision
• how classes are compiled, how classes are allocated to
teachers, test results, teachers’ observations, attendance
data, portfolios of work, student opinions …
Data are one form of evidence
Page 7
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Data
Known facts or measurements, probably expressed in some
systematic or symbolic way (e.g. as numbers)
• assessment results, gender, attendance, ethnicity …
Data are one form of evidence
Page 8
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven decision making
We have more evidence about what students know and can do
than ever before - their achievements, behaviours, environmental
factors that influence learning
We should
• draw on all our knowledge about the learning environment to
improve student achievement
• explore what lies behind patterns of achievement
• decide what changes will make a difference
Page 9
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
What evidence does a school have?
•
•
•
•
•
Page 10
Demographics
Student achievement
Perceptions
School processes
Other practice
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Demographics
What data do we have now to provide a profile of our school?
What other data could we create?
• School
• Students
• Staff
• Parents/caregivers and community
Page 11
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Student achievement
What evidence do we have now about student achievement?
What other evidence could we collect?
• National assessment results
• Standardised assessment results administered internally
• Other in-school assessments
• Student work
Page 12
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Perceptions
What evidence do we have now about what students, staff and
others think about the school?
Are there other potential sources?
• Self appraisal
• Formal and informal observations made by teachers
• Structured interactions
• Externally generated reports
• Student voice
• Other informal sources
Page 13
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
School processes
What evidence do we have about how our school is organised
and operates?
• Timetable
• Classes
• Resources
• Finance
• Staffing
Page 14
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Other practice
How we can find out about what has worked in other schools?
• Documented research
• Experiences of other schools
Page 15
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 16
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 17
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Data indicate a
possible issue that
could impact on
student
achievement
Speculate
A teacher has a
hunch about a
problem or a
possible action
Explore
Check data and
evidence to
explore the issue
Reflect
on what has been
learned, how
practice will change
Evaluate the
impact on the
intervention
Question
Clarify the issue
and ask a question
Assemble Decide
what data and
evidence might be
useful
Act
Carry out the
intervention
Intervene
Plan an action aimed at
improving student
achievement
Page 18
Interpret Insights
that answer your
question
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Analyse data and
evidence
The evidence-driven decision making cycle
SPECULATE
TRIGGER
EXPLORE
REFLECT
QUESTION
EVALUATE
ASSEMBLE
ACT
INTERVENE
Page 19
INTERPRET
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
ANALYSE
The evidence-driven decision making cycle
.Reflect
How will we
teach writing in
the future?
Trigger
Some of our
students are
poor at writing
A teacher has a
hunch - poor
writers might
spend little time
on homework
Explore data
Survey of
students
shows that
this is only
partially true
Evaluate Has
writing improved?
Intervene
Create multiple
opportunities for writing;
include topics that can
use sport as context;
connect speaking and
writing. PD for staff.
Page 20
Interpret
information
Poor writers
likely to play
sport, speak
well, read less,
do little HW
Analyse
NQF/NCEA
results by
standard
Analyse non
NQF/NCEA data
and evidence
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Question
What are the
characteristics
of students who
are poor at
writing?
Assemble
more data &
other
evidence:
asTTle reading,
homework,
extracurric,
Attendance, etc
Evaluate and reflect
• Summative evaluation – assess how successful the intervention
was; decide how our practice will change; report to board
• Formative evaluation - at every stage in the cycle we reflect and
evaluate
Are we are on the right track?
Do we need to fine-tune?
Do we actually need to complete this?
Page 21
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Types of analysis
We can compare achievement data by subject or across
subjects for
• an individual student
• groups of students
• whole cohorts
The type of analysis we use depends on the question we want
to answer
Page 22
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Trigger questions
•
•
•
•
•
•
•
•
•
•
How good/poor is …?
What aspects of … are good/poor?
Is … actually changing?
How is … changing?
Is … better than last year?
How can … be improved?
Why is … good/poor?
What targets are reasonable for …?
What factors influence the situation for …?
What would happen if we …?
Formative or summative?
Page 23
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Questions about policy
We have been running 60-minute periods for 5-years now.
What effect has the change had?
Page 24
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Questions from hunches
• I suspect this poor performance is being caused by …
Is this true?
• We reckon results will improve if we put more effort into ...
Is this likely?
• I think we’d get better results from this module if we added …
Is there any evidence to support this idea?
Page 25
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Questions with purpose
What do we know about reported bullying incidents for year 10
students?
MAY BE BETTER AS
Who has been bullying whom? Where?
What are students telling us?
What does pastoral care data tell us? Were some interventions
more effective with some groups of students than others?
Page 26
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Professional decision making
We have evidence-based information that we see as reliable and
valid
What do we do about it?
If the information indicates a need for action, we use our
collective experience to make a professional decision
Page 27
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Professionals making decisions
Do any particular groups of year 11 students attend less
regularly than average for the whole cohort?
The analysis identified two groups – so I need to think about
how to deal with irregular attendance for each group.
How will I do that?
Page 28
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Professionals making decisions
You asked what factors are related to poor student
performance in formal writing.
The analysis suggested that poor homework habits have
a significant impact on student writing.
You make some professional judgements and decide
• Students who do little homework don’t write enough
• You could take action to improve homework habits - but
you’ve tried that before and the success rate is low
• You have more control over other factors - like how much
time you give students to write in class
So you conclude - the real need is to get students to
write more often
Page 29
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Deciding on an action
Information will often suggest a number of options for action.
How do we decide which action to choose?
We need to consider
• what control we have over the action
• the likely impact of the action
• the resources needed
Page 30
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Planning for evaluation
• What evidence do we need to collect before we start?
• Do we need to collect evidence along the way, or just at the
end?
• How can we be sure that any assessment at the end of the
process will be comparable with assessment at the outset?
• How will we monitor any unintended effects?
Don’t forget evidence such as timetables, student opinions, teacher
observations …
Page 31
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evaluate the impact of our action
Did the intervention improve the situation that triggered the
process?
If the aim was to improve student achievement, did that
happen?
Page 32
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evaluate the impact of our action
Was any change in student achievement significant?
What else happened that we didn’t expect?
How do our results compare with other similar studies we can
find?
Does the result give us the confidence to make the
change permanent?
Page 33
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Future practice
• What aspects of the intervention will we build into future
practice?
• What aspects of the intervention will have the greatest impact?
• What aspects of the intervention can we maintain over time?
• What changes can we build into the way we do things in our
school?
• Would there be any side-effects?
Page 34
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Future directions
• What professional learning is needed? Who would most benefit
from it?
• Do we have the expertise we need in-house or do we need
external help?
• What other resources do we need?
• What disadvantages could there be?
• When will we evaluate this change again?
Page 35
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven strategic planning
If we use evidence-driven decision making to improve student
achievement and enhance teaching practice …
… it follows that strategic planning across the school should also
be evidence-driven.
Page 36
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven strategic planning
INDICATORS
FROM DATA
.
asTTle scores
show a high
proportion of
Yr 9 achieving
below
curriculum level
NCEA results
show high nonachievement in
transactional
writing
STRATEGIC GOAL
ANNUAL PLAN
YEAR TARGET
To raise the levels of writing
across the school
Develop and implement
a plan to raise levels of
Writing at year 9
Raise writing asTTle
results Yr 9boys from
3B to 3A
Strategic action
Development plan to be
based on an analysis of
all available data and to
include a range of
shared strategies
etc.
Develop a writing
development plan which
addresses writing across
subjects and levels ,
including targets,
professional development
and other resourcing needs
etc.
etc.
Poor results in
other language
NCEA
standards
EVALUATION
DATA
Appraisal
asTTle writing
results improve by
…
Perception data
from year 9 staff
indicates …
Evaluation of
effectiveness of
range of shared
strategies, barriers
and enablers …
etc.
PD
Self
review
etc
School
charter
Page 37
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
What now?
How can we apply this model in our school?
Page 38
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.