Transcript Exploring Assessment for Learning

Consider the Evidence
A resource to assist schools
to review their use of data and other evidence
2
What is meant by ‘data and other evidence’?
Page 1
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence
Any facts, circumstances or perceptions that can be used as an
input for an analysis or decision
• how classes are compiled, how classes are allocated to
teachers, test results, teachers’ observations, attendance data,
portfolios of work, student opinions …
Data are one form of evidence
Page 2
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Data
Known facts or measurements, probably expressed in some
systematic or symbolic way (e.g. as numbers)
• assessment results, gender, attendance, ethnicity …
Data are one form of evidence
Page 3
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
What evidence does a school have?
•
•
•
•
•
Page 4
Demographics
Student achievement
Perceptions
School processes
Other practice
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Demographics
Data that provides a profile of our school
• School - decile, roll size, urban/rural, single sex or coeducational, teaching spaces …
• Students - ethnicity, gender, age, year level, attendance,
lateness, suspension and other disciplinary data, previous
school, part-time employment …
• Staff - gender, age, years of experience, qualifications, teaching
areas, involvement in national curriculum and assessment,
turnover rate …
• Parents/caregivers and community - socio-economic factors,
breadth of school catchment, occupations …
Page 5
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Student achievement
Evidence about student achievement
• National assessment results - NCEA, NZ Scholarship - details like
credits above and below year levels, breadth of subjects entered …
• Standardised assessment results administered internally - PAT,
asTTle …
• Other in-school assessments - most non-standardised but some,
especially within departments, will be consistent across classes includes data from previous schools, primary/intermediate
• Student work - work completion rates, internal assessment
completion patterns, exercise books, notes, drafts of material these can provide useful supplementary evidence
Page 6
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Perceptions
Evidence about what students, staff, parents and the community think
about the school
• Self appraisal - student perceptions of their own abilities, potential,
achievements, attitudes …
• Formal and informal observations made by teachers - peer
interactions, behaviour, attitudes, engagement, student-teacher
relationships, learning styles, classroom dynamics …
• Structured interactions - records from student interviews, parent
interviews, staff conferences on students …
• Externally generated reports - from ERO and NZQA (these contain
data but also perceptions) …
• Student voice - student surveys, student council submissions …
• Other informal sources – views about the school environment, staff
and student morale, Board perceptions, conversations among
teachers …
Page 7
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
School processes
Evidence about how our school is organised and operates
• School processes - evidence and data about how your school is
organised and operates, including:
• Timetable –structure, period length, placement of breaks,
subjects offered, student choices, tertiary and workforce factors,
etc
• Classes - how they are compiled, their characteristics, effect of
timetable choices, etc
• Resources - access to libraries, text books, ICT, special
equipment, etc
• Finance - how the school budget is allocated, how funds are
used within departments, expenditure on professional
development
• Staffing - policies and procedures for employing staff, allocating
responsibility, special roles, workload, subjects and classes
Page 8
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Other practice
How we can find out about what has worked in other schools?
• Documented research – university and other publications,
Ministry of Education’s Best Evidence Syntheses, NZCER,
NZARE, overseas equivalents …
• Experiences of other schools – informal contacts, local clusters,
advisory services, TKI LeadSpace …
Page 9
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven strategic planning
INDICATORS
FROM DATA
.
asTTle scores
show a high
proportion of
Yr 9 achieving
below
curriculum level
NCEA results
show high nonachievement in
transactional
writing
STRATEGIC GOAL
ANNUAL PLAN
YEAR TARGET
To raise the levels of writing
across the school
Develop and implement
a plan to raise levels of
Writing at Year 9
Raise writing asTTle
results Yr 9 boys from
3B to 3A
Strategic action
Development plan to be
based on an analysis of
all available data and to
include a range of
shared strategies
etc
Develop a writing
development plan which
addresses writing across
subjects and levels ,
including targets,
professional development
and other resourcing needs
etc
etc
Poor results in
other language
NCEA
standards
EVALUATION
DATA
Appraisal
asTTle writing
results improve by
…
Perception data
from Yr 9 staff
indicates …
PD
Evaluation of
effectiveness of
range of shared
strategies, barriers
and enablers …
etc
Self
review
etc
School
charter
Page 10
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 11
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
REFLECT
on what has
been learned,
how practice
will change
TRIGGER
Data indicate a
possible issue
that could
impact on
student
achievement
SPECULATE
A teacher has a
hunch about a
problem or a
possible action
EXPLORE
Check data
and evidence
to explore the
issue
EVALUATE
the impact on
the
intervention
QUESTION
Clarify the
issue and ask a
question
ASSEMBLE
Decide what
data and
evidence might
be useful
ACT
Carry out the
intervention
INTERVENE
Plan action to
improve student
achievement
Page 12
INTERPRET
Insights that
answer your
question
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
ANALYSE data
and evidence
The evidence-driven decision making cycle
Reflect
How will we
teach writing in
the future?
Trigger
Some of our
students are
poor at writing
A teacher has a
hunch - poor
writers might
spend little time
on homework
Explore data
Survey of
students
shows that
this is only
partially true
Evaluate Has
writing improved?
Intervene
Create multiple
opportunities for writing;
include topics that can
use sport as context;
connect speaking and
writing. PD for staff.
Page 13
Interpret
information
Poor writers
likely to play
sport, speak
well, read less,
do little HW
Analyse
NQF/NCEA
results by
standard
Analyse non
NQF/NCEA data
and evidence
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Question
What are the
characteristics
of students who
are poor at
writing?
Assemble
more data &
other
evidence:
asTTle reading,
homework,
extracurric,
Attendance, etc
Types of analysis
We can compare achievement data by subject or across subjects for
• an individual student
• groups of students
• whole cohorts
The type of analysis we use depends on the question we want to
answer
Page 14
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Inter-subject analysis
• Have my students not achieved a particular history standard
because they have poor formal writing skills, rather than poor
history knowledge?
Page 15
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Intra-subject analysis
• What are the areas of strength and weakness in my own
teaching of this class?
Page 16
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Longitudinal analysis
• Are we producing better results over time in year 11 biology?
Page 17
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
> Trigger
Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 18
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Asking questions
Evidence-driven decision making
starts with asking good questions
You can tell whether a man is clever by his answers. You can
tell whether he is wise by his questions.
Nobel Prize winner, Naguib Mahfouz
Page 19
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Trigger questions
•
•
•
•
•
•
•
•
•
•
Page 20
How good/poor is …?
What aspects of … are good/poor?
Is … actually changing?
How is … changing?
Is … better than last year?
How can … be improved?
Why is … good/poor?
What targets are reasonable for …?
What factors influence the situation for …?
What would happen if we …?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Questions from hunches
• I suspect this poor performance is being caused by …
Is this true?
• We reckon results will improve if we put more effort into ...
Is this likely?
• I think we’d get better results from this module if we added …
Is there any evidence to support this idea?
Page 21
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
> Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 22
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Question – Explore – Question
It looks like our students are doing well in A but not in B. What can
we do about it?
EXPLORE … what else should we be asking?
Is this actually the case?
Is there anything in the data to suggest what we could do about it?
Page 23
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Question – Explore – Question
We have been running 60-minute periods for a year now. Did the
change achieve the desired effects?
EXPLORE … what else should we be asking?
How has the change impacted on student achievement?
Has the change has had other effects?
Is there more truancy?
Is more time being spent in class on assignments, rather than as
homework?
Page 24
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
> Question
Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 25
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
A very good question
• Specific and with a clear purpose
• Able to be investigated through looking at data and other
evidence
• Likely to lead to information on which we can act
Page 26
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Questions with purpose
What do we know about reported bullying incidents for year 10
students?
MAY BE BETTER AS
Who has been bullying whom? Where?
What are students telling us?
What does pastoral care data tell us? Were some interventions
more effective with some groups of students than others?
Page 27
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
> Assemble
Analyse
Interpret
Intervene
Evaluate
Reflect
Page 28
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Assembling the evidence
• We want to know if our senior students are doing better in one
area of NCEA biology than another.
So … we need NCEA results for our cohort.
• It could be that all biology students do better in this area than
others.
So … we also need data about national differences across the two
areas.
Page 29
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Think critically about data
• Was the assessment that created this data assessing exactly
what we are looking for?
• Was the assessment set at an appropriate level for this group of
students?
• Was the assessment properly administered?
• Are we comparing data for matched groups?
Page 30
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
Assemble
> Analyse
Interpret
Intervene
Evaluate
Reflect
Page 31
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Analysing data and other evidence
• Schools need some staff members who are responsible for
leading data analysis
• Schools have access to electronic tools to process data into
graphs and tables
• All teachers do data analysis
• Data is not an end in itself - it’s one of the many stages along
the way to evidence-driven decision making
Page 32
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Making sense of the results
Think about significance and confidence
How significant are any apparent trends?
How much confidence can we have in the information?
Page 33
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
Assemble
Analyse
> Interpret
Intervene
Evaluate
Reflect
Page 34
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do we have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Information
Knowledge gained from analysing data and making
meaning from evidence.
Information is knowledge (or understanding) that can inform your
decisions.
How certain you will be about this knowledge depends on a
number of factors: where your data came from, how reliable it
was, how rigorous your analysis was.
So the information you get from analysing data could be a
conclusion, a trend, a possibility.
Page 35
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Words, words, words …
Information can … establish, indicate, confirm, reinforce, back up,
stress, highlight, state, imply, suggest, hint at, cast doubt on,
refute …
• Does this confirm that …?
• What does this suggest?
• What are the implications of …?
• How confident are we about this conclusion?
Page 36
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Interrogate the information
• Is this the sort of result we envisaged? If not, why?
• How does this information compare with the results of other
research or the experiences of other schools?
• Are there other variables that could account for this result?
• Should we set this information alongside other data or evidence
to give us richer information?
• What new questions arise from this information?
Page 37
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
Assemble
Analyse
Interpret
> Intervene
Evaluate
Reflect
Page 38
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Professional decision making
We have evidence-based information that we see as reliable and
valid
What do we do about it?
If the information indicates a need for action, we use our collective
experience to make a professional decision
Page 39
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Deciding on an action
Information will often suggest a number of options for action. How
do we decide which action to choose?
We need to consider
• what control we have over the action
• the likely impact of the action
• the resources needed
Page 40
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Planning for evaluation
• What evidence do we need to collect before we start?
• Do we need to collect evidence along the way, or just at the
end?
• How can we be sure that any assessment at the end of the
process will be comparable with assessment at the outset?
• How will we monitor any unintended effects?
Don’t forget evidence such as timetables, student opinions, teacher
observations …
Page 41
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
Assemble
Analyse
Interpret
Intervene
> Evaluate
Reflect
Page 42
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evaluate the impact of our action
Did the intervention improve the situation that triggered the
process?
If the aim was to improve student achievement, did that happen?
Page 43
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evaluate the impact of our action
Was any change in student achievement significant?
What else happened that we didn’t expect?
How do our results compare with other similar studies we can find?
Does the result give us the confidence to make the change
permanent?
Page 44
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
Trigger
Explore
Question
Assemble
Analyse
Interpret
Intervene
Evaluate
> Reflect
Page 45
Clues found in data, hunches
Is there really an issue?
What do you want to know?
Get all useful evidence together
Process data and other evidence
What information do you have?
Design and carry out action
What was the impact?
What will we change?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Future practice
•
•
•
•
•
Page 46
What aspects of the intervention will we embed in future
practice?
What aspects of the intervention will have the greatest impact?
What aspects of the intervention can we maintain over time?
What changes can we build into the way we do things in our
school?
Would there be any side-effects?
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Future directions
• What professional learning is needed? Who would most benefit
from it?
• Do we have the expertise we need in-house or do we need
external help?
• What other resources do we need?
• What disadvantages could there be?
• When will we evaluate this change again?
Page 47
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
What now?
How can we apply this model in our school?
Page 48
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.