Transcript Exploring Assessment for Learning

Consider the Evidence
Evidence-driven decision making
for secondary schools
A resource to assist schools
to review their use of data and other evidence
5
Getting Started
Page 1
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven decision making
Today we will think about
• the data and other evidence we can use to improve teaching,
learning and student achievement
• how to get started on a decision-making process by asking
purposeful questions
Page 2
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
.Reflect
How will we
teach writing in
the future?
Trigger
Significant
numbers not
achieving well
in writing
A teacher has a
hunch - poor
writers might
spend little time
on homework
Explore data
Survey of
students
shows that
this is only
partially true
Evaluate Has
writing improved?
Intervene
Create multiple
opportunities for writing;
include topics that can
use sport as context;
connect speaking and
writing. PD for staff.
Page 3
Interpret
information
Poor writers
likely to play
sport, speak
well, read less,
do little HW
Analyse
NQF/NCEA
results by
standard
Analyse non
NQF/NCEA data
and evidence
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Question
What are the
characteristics
of students who
are poor at
writing?
Assemble
more data &
other
evidence:
asTTle reading,
homework,
extracurric,
Attendance, etc
Evidence-driven decision making
Getting started
Trigger
Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 4
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven eating
You need to buy lunch. Before you decide what to buy you
consider a number of factors:
• how much money do you have?
• what do you feel like eating?
• what will you be having for dinner?
• how far do you need to go to buy food?
• how much time do you have?
• where are you going to eat it?
Page 5
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven teaching
I had a hunch that Ana wasn’t doing as well as she could in her
research assignments, a major part of the history course. What
made me think this?
Ana’s general work (especially her writing) was fine. She made
perceptive comments in class, contributed well in groups and had
good results overall last year, especially in English.
How did I decide what to do about it?
Page 6
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven teaching (cont…)
I looked more closely at her other work. I watched her working in
the library one day to see if it was her reading, her use of
resources, her note taking, her planning, or what. At morning tea I
asked one of Ana’s other teachers about Ana’s approach to similar
tasks. I asked Ana if she knew why her research results weren’t as
good as her other results, and what her plans were for the next
assignment.
I thought about all of this and planned a course of action. I gave
her help with using indexes, searching, note taking and planning
and linking the various stages of her research.
Page 7
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Consider the Evidence
A resource to assist schools
to review their use of data and other evidence
What is meant by ‘data and other evidence’?
Page 8
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence
Any facts, circumstances or perceptions that can be used as an
input for an analysis or decision
• how classes are compiled, how classes are allocated to
teachers, test results, teachers’ observations, attendance
data, portfolios of work, student opinions …
Data are one form of evidence
Page 9
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Data
Known facts or measurements, probably expressed in some
systematic or symbolic way (e.g. as numbers)
• assessment results, gender, attendance, ethnicity …
Data are one form of evidence
Page 10
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Which factors are data?
Evidence to consider before buying lunch
• how much money you have
• what you feel like eating
• what you’ll be having for dinner
• how far you need to go to buy food
• how much time you have
• where you’re going to eat
• what your diet allows
Page 11
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven decision making
We have more evidence about what students know and can do
than ever before – their achievements, behaviours, environmental
factors that influence learning
We should
• draw on all our knowledge about the learning environment to
improve student achievement
• explore what lies behind patterns of achievement
• decide what changes will make a difference
Page 12
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
What evidence does a school have?
•
•
•
•
•
Page 13
Demographics
Student achievement
Perceptions
School processes
Other practice
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Demographics
What data do we have now to provide a profile of our school?
What other data could we create?
• School
• Students
• Staff
• Parents/caregivers and community
Page 14
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Demographics
Data that provides a profile of our school
• School - decile, roll size, urban/rural, single sex or coeducational, teaching spaces …
• Students - ethnicity, gender, age, year level, attendance,
lateness, suspension and other disciplinary data, previous
school, part-time employment …
• Staff - gender, age, years of experience, qualifications,
teaching areas, involvement in national curriculum and
assessment, turnover rate …
• Parents/caregivers and community - socio-economic factors,
breadth of school catchment, occupations …
Page 15
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Student achievement
What evidence do we have now about student achievement?
What other evidence could we collect?
• National assessment results
• Standardised assessment results administered internally
• Other in-school assessments
• Student work
Page 16
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Student achievement
Evidence about student achievement
• National assessment results - NCEA, NZ Scholarship - details like
credits above and below year levels, breadth of subjects
entered…
• Standardised assessment results administered internally - PAT,
asTTle …
• Other in-school assessments - most non-standardised but some,
especially within departments, will be consistent across classes includes data from previous schools, primary/intermediate
• Student work - work completion rates, internal assessment
completion patterns, exercise books, notes, drafts of material these can provide useful supplementary evidence
Page 17
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Perceptions
What evidence do we have now about what students, staff and
others think about the school?
Are there other potential sources?
• Self appraisal
• Formal and informal observations made by teachers
• Structured interactions
• Externally generated reports
• Student voice
• Other informal sources
Page 18
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Perceptions
Evidence about what students, staff, parents and the
community think about the school
• Self appraisal - student perceptions of their own abilities,
potential, achievements, attitudes …
• Formal and informal observations made by teachers - peer
interactions, behaviour, attitudes, engagement, studentteacher relationships, learning styles, classroom dynamics …
• Structured interactions - records from student interviews,
parent interviews, staff conferences on students …
• Externally generated reports - from ERO and NZQA (these
contain data but also perceptions) …
• Student voice - student surveys, student council
submissions…
• Other informal sources – views about the school environment,
staff and student morale, Board perceptions, conversations
among teachers …
Page 19
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
School processes
What evidence do we have about how our school is organised
and operates?
• Timetable
• Classes
• Resources
• Finance
• Staffing
Page 20
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
School processes
Evidence about how our school is organised and operates
• School processes - evidence and data about how your school
is organised and operates, including:
• Timetable –structure, period length, placement of breaks,
subjects offered, student choices, tertiary and workforce
factors, etc
• Classes - how they are compiled, their characteristics, effect of
timetable choices, etc
• Resources - access to libraries, text books, ICT, special
equipment, etc
• Finance - how the school budget is allocated, how funds are
used within departments, expenditure on professional
development
• Staffing - policies and procedures for employing staff,
allocating responsibility, special roles, workload, subjects and
classes
Page 21
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Other practice
How can we find out about what has worked (or not) in other
schools?
Page 22
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
22
Other practice
How we can find out about what has worked in other schools?
• Documented research – university and other publications,
Ministry of Education’s Best Evidence Syntheses, NZCER,
NZARE, overseas equivalents …
• Experiences of other schools – informal contacts, local
clusters, advisory services, TKI LeadSpace …
Page 23
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
What can we do with evidence?
Shane’s story
A history HOD wants to see whether history students are
performing to their potential.
She prints the latest internally assessed NCEA records for history
students across all of their subjects. As a group, history students
seem to be doing as well in history as they are in other subjects.
Then she notices that Shane is doing very well in English and only
reasonably well in history. She wonders why, especially as both are
language-rich subjects with many similarities.
The HOD speaks with the history teacher, who says Shane is
attentive, catches on quickly and usually does all work required. He
mentions that Shane is regularly late for class, especially on
Monday and Thursday. So he often misses important information or
takes time to settle in. He has heard there are ‘problems at home’
so has overlooked it, especially as the student is doing reasonably
well in history.
Page 24
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Shane’s story (cont...)
The HOD looks at the timetable and discovers that history is period
1 on Monday and Thursday. She speaks to Shane’s form teacher
who says that she suspects Shane is actually late to school virtually
every day. They look at centralised records. Shane has excellent
attendance but frequent lateness to period 1 classes.
The HOD speaks to the dean who explains that Shane has to take
his younger sister to school each morning. He had raised the issue
with Shane but he said this was helping the household get over a
difficult period and claimed he could handle it.
The staff involved agree that Shane’s regular lateness is having a
demonstrable impact on his achievement, probably beyond history
but not so obviously.
The dean undertakes to speak to the student, history teacher, and
possibly the parents to find a remedy for the situation.
Page 25
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Thinking about Shane’s story
What were the key factors in the scenario about Shane?
What types of data and other evidence were used?
What questions did the HOD ask?
What happened in this case that wouldn’t necessarily happen
in some schools?
Page 26
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Shane’s story - keys to success
The history HOD looked at achievement data in English and
history.
She looked for something significant across the two data sets, not
just low achievement.
Then she asked a simple question: Why is there such a disparity
between in these two subjects for that student?
She sought information and comments (perceptions evidence
and data) from all relevant staff.
The school had centralised attendance and punctuality records
(demographic data) that form teacher could access easily.
The action was based on all available evidence and designed to
achieve a clear aim.
Page 27
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Data indicate a
possible issue that
could impact on
student
achievement
Speculate
A teacher has a
hunch about a
problem or a
possible action
Explore
Check data and
evidence to
explore the issue
Reflect
on what has been
learned, how
practice will change
Evaluate the
impact on the
intervention
Question
Clarify the issue
and ask a question
Assemble Decide
what data and
evidence might be
useful
Act
Carry out the
intervention
Intervene
Plan an action aimed at
improving student
achievement
Page 28
Interpret Insights
that answer your
question
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Analyse data and
evidence
The evidence-driven decision making cycle
SPECULATE
TRIGGER
EXPLORE
REFLECT
QUESTION
EVALUATE
ASSEMBLE
ACT
INTERVENE
Page 29
INTERPRET
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
ANALYSE
Evaluate and reflect
• Summative evaluation – assess how successful the intervention
was; decide how our practice will change; report to board
• Formative evaluation – at every stage in the cycle we reflect
and evaluate
Are we are on the right track?
Do we need to fine-tune?
Do we actually need to complete this?
Page 30
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
> Trigger
Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 31
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Asking questions
Evidence-driven decision making
starts with asking good questions
You can tell whether a man is clever by his answers. You can
tell whether he is wise by his questions.
Nobel Prize winner, Naguib Mahfouz
Page 32
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Trigger questions
•
•
•
•
•
•
•
•
•
•
How good/poor is …?
What aspects of … are good/poor?
Is … actually changing?
How is … changing?
Is … better than last year?
How can … be improved?
Why is … good/poor?
What targets are reasonable for …?
What factors influence the situation for …?
What would happen if we …?
Formative or summative?
Page 33
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Summative questions
A target in the school’s annual plan is for all year 10 boys to
improve their writing level by at least one level using asTTle
(e.g. from 4B to 4A).
Have all year 10 boys improved by at least one asTTle level in
writing?
Page 34
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Questions about policy
We have been running 60-minute periods for 5 years now.
What effect has the change had?
Page 35
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Formative questions from data
The data suggest our students are achieving well in A, but less
well in B.
What can we do about that?
Page 36
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Formative questions from data
A significant proportion of our school leavers enrol in vocational
programmes at polytechnic or on-job.
How well do our school programmes prepare those students?
Page 37
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Questions from hunches
• I suspect this poor performance is being caused by …
Is this true?
• We reckon results will improve if we put more effort into ...
Is this likely?
• I think we’d get better results from this module if we added …
Is there any evidence to support this idea?
Page 38
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Hunches from raw data
2.1
Pamela N
1
A
Lee
2
E
Manu
3
Keisha N
4
E
Bron
5
M
Deane
6
N
Slane
7
A
Sam
8
M
Sione
9
A
10 Oran
E
11 Shirin
E
12 Hanna
E
13 Val
N
14 Liam
15 Morgan M
N
16 Hone
A
17 Mahi
Page 39
2.2
A
A
E
A
M
M
A
A
M
A
E
E
E
A
M
A
A
2.3
N
A
E
N
M
E
N
N
N
A
E
M
E
M
M
N
N
2.4*
N
N
E
N
N
M
N
A
N
A
E
M
E
M
M
N
A
2.5*
N
N
N
N
N
N
N
A
N
A
A
A
N
N
N
N
A
2.6* ABS DET
6
20
N
0
12
A
4
18
E
8
7
N
0
3
A
1
2
A
8
22
N
8
12
A
2
2
N
0
7
A
0
6
E
1
0
M
0
0
E
2
10
M
0
15
M
4
17
N
0
10
A
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Hunches from raw data
• Is the class as a whole doing better in internally assessed
standards than in externally assessed standards? If so, why?
• Are the better students (with many Excellence results) not doing
as well in external assessments as in internal? If so, why?
• Is there any relationship between absences and achievement
levels? It seems not, but it’s worth analysing the data to be
sure.
Page 40
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
> Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 41
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Question – Explore – Question
It looks like our students are doing well in A but not in B. What
can we do about it?
EXPLORE … what else should we be asking?
Is this actually the case?
Is there anything in the data to suggest what could we do about
it?
Page 42
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Question – Explore – Question
We have been running 60-minute periods for a year now. Did
the change achieve the desired effects?
EXPLORE … what else should we be asking?
How has the change impacted on student achievement?
Has the change has had other effects?
Is there more truancy?
Is more time being spent in class on assignments, rather than
as homework?
Page 43
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
> Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 44
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
A very good question
• Specific and with a clear purpose
• Able to be investigated through looking at data and other
evidence
• Likely to lead to information on which we can act
Page 45
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Questions with purpose
What do we know about reported bullying incidents for year 10
students?
MAY BE BETTER AS
Who has been bullying whom? Where?
What are students telling us?
What does pastoral care data tell us? Were some interventions
more effective with some groups of students than others?
Page 46
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Write more purposeful questions
1. What are the attendance rates for year 11 students?
2. What has been the effect of the new 6-day x 50-min period
structure?
3. How well are boys performing in formal writing in year 9?
4. What has been the effect of shifting the lunch break to after
period 4?
Page 47
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
More purposeful questions
1. How do year 11 attendance rates compare with other year
levels? Do any identifiable groups of year 11 students attend
less regularly than average?
2. Is the new 6-day x 50-min period structure having any positive
effect on student engagement levels? Is it influencing
attendance patterns? What do students say?
3. Should we be concerned about boys’ writing? If so, what action
should we be taking to improve the writing of boys in terms of
the literacy requirements for NCEA Level 1?
4. The new timing of the lunch break was intended to improve
student engagement levels after lunch. Did it achieve this? If
so, did improvements in student engagement improve student
achievement? Do the benefits outweigh any disadvantages?
Page 48
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 49
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
SPECULATE
TRIGGER
EXPLORE
REFLECT
QUESTION
EVALUATE
ASSEMBLE
ACT
INTERVENE
Page 50
INTERPRET
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
ANALYSE