Transcript Classroom aggregation technologies: third generation pedagogy

Learning, assessment and
technology: in that order
Keynote address to AMEE conference
September 2009; Malaga, Spain
Dylan Wiliam
Institute of Education, University of London
www.dylanwiliam.net
Overview of presentation
Some theoretical precepts
 About learning
 About teaching
• Pedagogies of engagement
• Pedagogies of contingency
• Pedagogies of formation
The role of technology
 Supporting, rather than replacing, teachers
 Classroom aggregation technologies
Some thoughts about supporting teachers in changing practice
How do we improve
learning?
What gets learnt?
(Denvir & Brown, 1986)
Key insights from C20th psychology
What gets learned as a result of a particular sequence of
instructional activities is impossible to predict, but
Student errors are not random
Conclusions
 Assessment is the bridge between teaching and learning
 Teaching is interesting because learners are so different, but only
possible because they are so similar
Learning power environments
Key concept:
 Teachers do not create learning
 Learners create learning
Teaching as the engineering of learning environments
Key features:
 Create student engagement (pedagogies of engagement)
 Well-regulated (pedagogies of contingency)
 Develop habits of mind (pedagogies of formation)
Why pedagogies of engagement?
Intelligence is partly inherited
 So what?
Intelligence is partly environmental
 Environment creates intelligence
 Intelligence creates environment
Learning environments
 High cognitive demand
 Inclusive
 Obligatory
Engagement and feedback
 264 low and high ability grade 6 students in 12 classes in 4
schools; analysis of 132 students at top and bottom of each class
 Same teaching, same aims, same teachers, same class work
 Three kinds of feedback: scores, comments, scores+comments
Scores
Comments
Achievement
Attitude
no gain
High scorers: positive
Low scorers: negative
30% gain
High scorers : positive
Low scorers : positive
[Butler(1988) Br. J. Educ. Psychol., 58 1-14]
Responses
Scores
Comments
Achievement
Attitude
no gain
High scorers :
positive
Low scorers: negative
30% gain
High scorers :
positive
Low scorers : positive
What do you think happened for the students given both
scores and comments?
A.
B.
C.
D.
E.
Gain: 30%; Attitude: all positive
Gain: 30%; Attitude: high scorers positive, low scorers negative
Gain: 0%; Attitude: all positive
Gain: 0%; Attitude: high scorers positive, low scorers negative
Something else
[Butler(1988) Br. J. Educ. Psychol., 58 1-14]
Kinds of feedback (Nyquist, 2003)
Weaker feedback only
 Knowledge of results (KoR)
Feedback only
 KoR + clear goals or knowledge of correct results (KCR)
Weak formative assessment
 KCR+ explanation (KCR+e)
Moderate formative assessment
 (KCR+e) + specific actions for gap reduction
Strong formative assessment
 (KCR+e) + activity
Effect of formative assessment (HE)
N
Effect size
Weaker feedback only
31
0.14
Feedback only
48
0.36
Weaker formative assessment
49
0.26
Moderate formative assessment
41
0.39
Strong formative assessment
16
0.56
(Nyquist, 2003; revised values)
Effects of feedback
Kluger & DeNisi (1996) review of 3000 research reports
 Excluding poorly designed studies left 131 reports, 607 effect
sizes, involving 12652 individuals
On average, feedback increases achievement, but
 Effect sizes highly variable
 38% (50 out of 131) of effect sizes were negative
Engagement in learning
Attribution (Dweck, 2000)
 Personalization (internal vs. external)
 Permanence (stable vs. unstable)
 Good learners attribute failure and success to internal, unstable
causes. (It’s down to you, and you can do something about it.)
Views of ‘ability’
 Fixed (IQ)
 Incremental (untapped potential)
 Essential that teachers inculcate in students a view that ‘ability’ is
incremental rather than fixed (by working, you’re getting smarter).
Dual-pathway theory (Boekaerts, 2006)
Long-term learning goals are translated into short-term
learning intentions
Dynamic comparisons of task and situational demands with
personal resources
Resulting activation of energy along one of two pathways:
 Well-being
 Growth
Motivation: cause or effect?
high
arousal
Flow
anxiety
challenge
control
worry
relaxation
apathy
boredom
low
low
competence
high
(Csikszentmihalyi, 1990)
Why pedagogies of contingency?
Fuchs & Fuchs (1986)
Natriello (1987)
Crooks (1988)
Banger-Drowns, et al. (1991)
Kluger & DeNisi (1996)
Black & Wiliam (1998)
Nyquist (2003)
Dempster (1991, 1992)
Elshout-Mohr (1994)
Brookhart (2004)
Allal & Lopez (2005)
Köller (2005)
Brookhart (2007)
Wiliam (2007)
Hattie & Timperley (2007)
Shute (2008)
Hinge-point question
An experimental study of problem-based learning in
undergraduate medical education reports that a result was
significant (p<0.05). This means that:
A. The experimental group out-performed the control group by at least
5%
B. There is only a 5% chance that the experimental group did not outperform the control group
C. There is a 5% chance that there is no difference between the
experimental group and the control group
D. There is only a 5% chance that the observed result would have
happened if the experimental and control groups had the same
achievement
Other supports for contingency
All-student response systems
 ABCD cards
 “Exit-pass” questions
Exit-pass question
Summarize the key principles of the following schools of
psychology on the appropriate coloured card
 Associationism (blue)
 Information processing (orange)
 Constructivism (red)
 Situated approaches (green)
Pedagogies of formation
Instilling disciplinary habits of mind
 History
 Philosophy
 Statistics
Instilling critical perspectives
 Values
Three generations of pedagogy
First generation
 Traditional pedagogy
 Negligible contingency
Second generation
 All student response systems
 Contingency dependent entirely on teacher skill
Third generation
 Automated aggregation technologies
 Contingency supported by technology
Evidence-centred design
Quality in assessment is essentially a matter of validity
 Validity is a property of inferences, not of instruments
 Assessments should be designed “backwards” from the intended
inferences
Four-process architecture for assessment design
 Task selection
 Task presentation
 Evidence identification
 Evidence accumulation
Almond, Steinberg and Mislevy (2002)
Task selection/
Task presentation
Hinge-point question
Which of the following is the most important difference
between the theories of Piaget and Vygotsky?
A. Piaget places greater importance on the role of
conservation in cognitive development
B. Vygotsky places greater importance on the role of cultural
artifacts in cognitive development.
C. Vygotsky did not believe in distinct stages of cognitive
development.
D. Piaget was a social constructivist while Vygotsky placed
greater emphasis on cultural-historical activity theory
Evidence
identification
Evidence identification
Single student response systems
All-student response systems
 Flash-cards/dry erase boards
 Classroom ‘clickers’
 Traditional keyboards (wired/wireless)
 Anoto pens
Anoto pen
Wireless pen
Special coated paper
Pen ‘knows where it is’
Palm with wireless keyboard
Text-based input
Limited task-presentation
capability
Portable
Classroom ‘clickers’ (and their progeny)
Discourse®
www.ets.org/discourse
Evidence identification
Automated essay scoring (e-rater)
Paraphrase analysers (c-rater)
Graphical and equation analysers (m-rater)
Evidence identification technologies
automated
aggregation
clickers
teachermediated
m-rater c-rater
e-rater
latent
semantic
analysis
ABCD
cards
dry-erase
boards
structured
Discourse®
unstructured
evidence structure
Evidence
accumulation
Evidence accumulation
Unidimensional student models
 Useful for summative purposes
 Almost useless for formative purposes
Multidimensional student models
Evidence-centred design
 Bayesian inference networks
•
•
•
•
Proficiency model
Task model
Evidence model
Student model
Four-process architecture for ECD
Task selection
Evidence accumulation
Task presentation
Evidence identification
Mislevy, Almond and Lukas (2003)
Evidence utilization
Whole-class
Sub-groups
 Homogenous
 Heterogenous
Individualization
Actually, the technology is the easy part
What’s hard is changing practice
Telling teachers what to do doesn’t work
Context always intrudes…
37
The ancient yogis used logs of wood, stones, and ropes to help them practise asanas
effectively. Extending this principle, Yogacharya Iyengar invented props which allow
asanas to be held easily, and for a longer duration without strain.
Yogacharya Iyengar in setubandha sarvangasana
This version of the posture requires considerable strength in the neck, shoulders, and
back requiring years of practice to achieve it. It should not be attempted without
38
supervision.
39
A model for teacher learning
Content, then process
Content (what we want teachers to change)
 Evidence
 Ideas (strategies and techniques)
Process (how to go about change)
 Choice
 Flexibility
 Small steps
 Accountability
 Support
Summary
Learning has to be done by the learner, not for the learner
Teaching as engineering effective learning environments
Features of effective learning environments
 Pedagogies of engagement
 Pedagogies of contingency
• All-student response systems
• Classroom aggregation technologies
 Pedagogies of formation
Learning milieu focused on growth, rather than well-being
Teachers supported to improve practice continually