Developing Potentials for Learning:

Evidence, assessment, and progress

John Hattie Visible Learning Lab University of Auckland New Zealand EARLI 2007

Three parts to the story

1.

To identify major influences on achievement (using 700+ meta-analyses) 2.

To outline an assessment model based on these influences 3.

To demonstrate the power and joy of Educational Psychology

Influences on Achievement ?

Decreased 0 Zero Enhanced

Reducing Class Size on Achievement?

What is the effect of reducing class size Hundreds of evaluations of reducing class size ….

0 Decreased Zero Enhanced

Effect on Achievement over time?

Reducing Class Size 0 .20

1.0

Decreased Zero

An effect-size of advancing achievement % improving rate of learning r variable & achievement % of students with treatment exceeding those not treated .20

9 mths 10% .10

8

Enhanced

1.0

3 yrs 45% .45

34

The typical influence on achievement

So what is the typical effect across



750+ meta-analysis



50,000 studies, and



200+ million students

Effect on Achievement over time?

Typical Effect Size 0 .20

.40

Zero 1.0

Enhanced Decreased

Distribution of effects

Influences on Achievement

0 .15

REVERSE .30

.40

.50

.60

ZONE OF DESIRED EFFECTS .70

.80

.90

1.0

Rank these 11 effects:

          

Reducing disruptive behavior in the class Feedback Acceleration of gifted students Reading Recovery Integrated curriculum programs Homework Individualized instruction Ability grouping Open vs. traditional classes Retention (holding back a year) Shifting schools

(from 1 = highest effect to 11 = lowest effect)

Rank these 11 effects: Answers

          

Reducing disruptive behavior in the class Feedback Acceleration of gifted students Reading Recovery Integrated curriculum programs Homework Individualized instruction Ability grouping Open vs. traditional classes Retention (hold back a year) Shifting schools .86

.72

.60

.50

.40

.30

.20

.10

.00

-.16

-.34

Major domains of interest



Curricula



Home



School



Student



Teacher



Teaching

Rank 100 99 98 97 96 95 94 93 92 91

The Disasters ...

Category Influence Mobility (shifting schools) Retention Television Summer vacation Open vs. traditional Multi-grade/age classes Inductive teaching Reading: Whole language Perceptual-motor programs Out of school experiences Studies 181 207 31 39 315 94 24 64 180 52 Effects 540 2675 235 62 333 72 24 197 637 50 ES -.34

-.16

-.14

-.09

.01

.04

.06

.06

.08

.09

Rank

The Well belows...

Category Influence 90 89 Distance education Web based learning 88 87 86 85 84 83 82 81 Ability grouping Teacher training Diet on achievement Teacher subject matter knowledge Gender (girls – boys) Multi-media methods Problem based learning Home-school programmes Studies 788 10 494 53 23 27 2926 244 203 14 Effects 1545 10 1363 286 125 64 6051 133 345 14 ES .09

.09

.11

.11

.12

.12

.12

.15

.15

.16

Rank

Not Worth it yet ...

Category Influence 80 Extra-curricula programs 79 78 77 76 75 74 73 72 71 Family structure Co-/team teaching Learning hierarchies Aptitude/treatment interactions Individualised instruction Charter schools Religious schools Class size Teaching test taking Studies 96 660 136 24 61 581 18 56 96 267 Effects 68 846 47 24 340 1030 18 56 785 364 ES .17

.18

.19

.19

.19

.20

.20

.20

.21

.22

Rank 70 69 68 67 66 65 64 63 62 61

Typical “average teacher” territory ...

Category Influence Finances Summer school Competitive learning Programmed instruction Within class grouping Mainstreaming Desegregation Exercise/relaxation Audio-based teaching Home visiting by teachers Studies 189 105 831 464 148 150 335 227 146 71 Effects 681 600 203 362 297 370 723 1971 48 52 ES .23

.23

.24

.24

.25

.28

.28

.28

.28

.29

58 57 56 55 54 53 52 51 Rank 60

Close to average

Category Influence Reducing anxiety 59 Principals/school leaders on student achievement Ability grouping for gifted students Homework Inquiry based teaching Simulations and gaming Reading: Exposure to reading Bilingual programs Teacher positive expectations Computer assisted instruction Studies 69 344 125 261 205 342 145 128 635 4481 Effects 904 1008 202 275 420 449 324 666 745 8079 ES .30

.30

.30

.31

.31

.32

.36

.37

.37

.37

Rank 50 49 48 47 46 45 44 43 42 41

Average …

Category Influence Enrichment on gifted Integrated curriculum programs Adjunct aids Hypermedia instruction Behavioral organisers/adjunct questions Self-concept on achievement Frequent/effects of testing Early intervention Motivation on learning Small group learning Studies 214 61 138 46 Effects 543 80 323 143 ES .39

.39

.41

.41

577 324 323 1627 322 78 1933 2113 1077 9050 979 155 .41

.43

.46

.47

.48

.49

Rank 40 39 38 37 36 35 34 33 32 31

Getting there …

Category Influence Questioning Cooperative learning Reading: Second/third chance programs Play programs Visual based/audio-visual teaching Outdoor programs Concept mapping Peer influences Keller's mastery learning program Reading: Phonics instruction Studies 214 2285 52 70 468 187 91 12 263 407 Effects 342 1519 1395 70 3860 429 105 122 162 5950 ES .49

.49

.50

.50

.51

.52

.52

.53

.53

.53

Rank

Let’s have them ....

Category Influence 30 Reading: Visual-perception programs 25 24 23 22 29 28 27 26 21 Parental Involvement Peer tutoring Goals - challenging Mastery learning Social skills programs Socio-economic status Home environment Providing worked examples Reading: Comprehension programs Studies 762 694 767 454 369 540 499 35 62 365 Effects 5244 1761 1200 671 284 3068 957 109 151 2416 ES .55

.55

.55

.56

.57

.57

.57

.57

.57

.58

Exciting ….

Rank 20 19 18 17 16 15 14 13 12 11 Category Influence Direct instruction Time on task Study skills Acceleration of gifted Problem solving teaching Teacher professional development on student achievement Reading: Repeated reading programs Reading: Vocabulary programs Meta-cognition strategies Teaching students self verbalisation Studies 304 64 656 60 221 450 54 301 43 92 Effects 597 100 2446 412 719 1790 ES .59

.59

.59

.60

.61

.64

156 800 123 1061 .67

.67

.67

.67

7 8 3 4 5 6 9 10 Rank 1 2

The Winners ...

Category Influence Self-report grades Absence of disruptive students Classroom behavioural Quality of teaching Reciprocal teaching Prior achievement Teacher-student relationships Feedback Providing formative evaluation to teachers Creativity programs Studies 209 Effects 305 ES 1.44

140 160 141 38 3387 229 1276 21 658 315 942 195 53 8758 1450 1928 21 814 .86

.80

.77

.74

.73

.72

.72

.70

.70

Various Influences

Teacher Curricula Teaching Student Home School Average Metas 29 135 344 133 31 96 768 Studies 2,052 6,892 24,906 10,735 1,998 4,019 50,602 People .5m

7m 52m 7m 10m 4m 82/241m* Effects 5,379 29,476 50,953 37,308 3,968 13,609 140,693 ES .50

.45

.43

.39

.35

.23

.40

se .05

.07

.07

.04

.06

.07

.06

Identifying what matters

Percentage of Achievement Variance Teachers Home Peers Schools Principal Students

Visible teaching & Visible learning



What some teachers do!



In active, calculated and meaningful ways



Providing multiple opportunities & alternatives



Teaching learning strategies



Around surface and deep learning



That leads to students constructing learning

Visible Teaching – Visible Learning

Teachers



Clear learning intentions



Challenging success criteria



Range of learning strategies



Know when students are not progressing



Providing feedback



Visibly learns themselves

Students …



Understand learning intentions



Are challenged by success criteria



Develop a range of learning strategies



Know when they are not progressing



Seek feedback



Visibly teach themselves

The Contrast



An active teacher, passionate for their subject and for learning, a change agent OR



A facilitative, inquiry or discovery based provider of engaging activities

Activator or Facilitator ?

An Activator

Reciprocal teaching Feedback Teaching students self-verbalization Meta-cognition strategies Direct Instruction Mastery learning Goals - challenging Frequent/ Effects of testing Behavioral organizers

A Facilitator

Simulations and gaming Inquiry based teaching Smaller class sizes Individualized instruction Problem-based learning Different teaching for boys & girls Web-based learning Whole Language Reading Inductive teaching

Activator or Facilitator ?

An Activator ES

Reciprocal teaching Feedback Teaching students self-verbalization Meta-cognition strategies Direct Instruction Mastery learning Goals - challenging Frequent/ Effects of testing Behavioral organizers

ACTIVATOR

.74 .72 .67 .67 .59 .57 .56 .46 .41

.60 A Facilitator ES

Simulations and gaming Inquiry based teaching Smaller class sizes .32 .31 .21 Individualized instruction Problem-based learning .20 .15 Different teaching for boys & girls .12 Web-based learning Whole Language Reading Inductive teaching .09 .06 .06

FACILITATOR .17

Teaching or Working Conditions?

Teaching

Quality of teaching Reciprocal teaching Teacher-student relationships Providing feedback Teaching student self-verbalization Meta-cognition strategies Direct Instruction Mastery learning

AVERAGE

ES

.77 .74 .72 .72 .67 .67 .59 .57

.68

Structural/Working Conditions

Within class grouping Adding more finances Reducing class size Ability grouping Multi-grade/age classes Open vs. Traditional classes Summer vacation classes Retention .28 .23 .21 .11 .04 .01 -.09 -.16

.08

An emphasis on learning strategies

Creativity Programs NS 658 Teaching student self-verbalization 92 Meta-cognition strategies 43 Problem solving teaching Study skills Concept mapping Motivation on learning 221 656 91 322 NE ES 814 .

70 1061 .67

123 719 2446 105 979 .67

.61

.59

.52

.48

Message for Learning

   

Balance of surface, deep, & constructed knowing Teachers preach deep, students see surface!

Learning strategies



To reduce cognitive load



To use when stuck (welcome error!) Requires deliberative practice



Builds expectations of “can do”

 

Thrives on challenge Requires feedback

Self Regulation



Need surface and deep to maximize meta-cognition



Involves questioning our constructions of knowledge & knowing



Requires highly structured or direct teaching



Is the heart of Visible learning and Visible teaching

Some worrying details

Influence No. of Effect Effects Size

Teacher training Teacher subject knowledge Gender (Girls - Boys) Principals on achievement Instructional vs.

Transformation leadership 53 27 2926 344 0.11

0.12

0.12

0.3

0.44

0.22

Assessment and Feedback

Assessment…

Influence

Self-report grades Feedback Providing formative evaluation to teachers Frequent/ Effects of testing Teaching test taking skills

Effect Size

1.44

.72

.70

.46

.22

Teaching/Teachers needs to be evidence based

        Whenever we test in classes it is primarily to help teachers know:

Whether their teaching methods have been successful or not Whether their learning intentions are worthwhile & challenging Whether students are attaining their desired success criteria Which students have learnt or not learnt Where teachers can capitalize on student strengths & minimize gaps Where students are on the learning ladder Whether they have a shared conception of progress What is optimal to teach next

Priority to maximize FEEDBACK to THE TEACHER

 Feedback is information provided by an agent (e.g., teacher, peer, book, parent, self/experience) regarding aspects of one’s performance or understanding.

Purpose of feedback

      provide alternative strategies to understand material increase effort, motivation or engagement confirm that the responses are correct or incorrect indicate that more information is available or needed point to directions that could be pursued to restructure understandings

Feedback is evidence about:

   Where am I going?

How am I going?

Where to next?

Gap

Enhancing Learning

How ???

Levels

Summary of effect-sizes relating to types of feedback (adapted from Kluger & DeNisi, 1996)

Moderator Correct feedback ‘Tis correct ‘Tis incorrect No.

114 197 ES .43

.25

Summary of effect-sizes relating to types of feedback (adapted from Kluger & DeNisi, 1996)

Moderator Correct feedback FB about changes from previous trials ‘Tis correct ‘Tis incorrect Yes No No.

114 197 ES .43

.25

50 380 .55

.28

Summary of effect-sizes relating to types of feedback (adapted from Kluger & DeNisi, 1996)

Moderator Correct feedback FB about changes from previous trials FB designed to discourage student ‘Tis correct ‘Tis incorrect Yes No Yes No No.

114 197 50 380 49 388 ES .43

.25

.55

.28

-.14

.33

Summary of effect-sizes relating to types of feedback (adapted from Kluger & DeNisi, 1996)

Moderator Correct feedback FB about changes from previous trials FB designed to discourage the student Praise FB ‘Tis correct ‘Tis incorrect Yes No Yes No Yes No No.

114 197 50 380 49 388 80 358 ES .43

.25

.55

.28

-.14

.33

.09

.34

Summary of effect-sizes relating to types of feedback (adapted from Kluger & DeNisi, 1996)

Moderator Correct feedback FT about changes from previous trials FT designed to discourage the student Praise FT No. of times FT was provided ‘ Tis correct ‘Tis incorrect Yes No Yes No Yes No Lots Little No.

114 197 50 380 49 388 80 358 97 171 .43

.25

ES .55

.28

-.14

.33

.09

.34

.32

.39

Summary of effect-sizes relating to types of feedback (adapted from Kluger & DeNisi, 1996)

Moderator Correct feedback FB about changes from previous trials FB designed to discourage the student Praise FB No. of times FB was provided Goal setting ‘ Tis correct ‘Tis incorrect Yes No Yes No Yes No Lots Little Difficult goals Easy, do your best goals No.

114 197 50 380 49 388 80 358 97 171 37 373 .43

ES .25

.55

.28

-.14

.33

.09

.34

.32

.39

.51

.30

Thus …



Provide correct feedback



About previous attempts



Related to more difficult goals



That does not discourage



or threaten their self-esteem .43

.55

.51

.33

.47

Assessment and Feedback: asTTle

Learning) (Assessment Tools for Teaching and

What is Visible Learning?

  

Where are we going?

 related to both the

rich ideas underlying the curricula, and to the desired levels of proficiency as expected by the ‘National’ Curricula;

How are we going ?

 current status of performance comparable to appropriate (similar sub-) groups, and to expected performance;

Where to next?

 directions related to future teaching, learning, curriculum innovations, and system policies.

Visible Learning is about Feedback

What is Visible Learning ?

Critical Infrastructure for the Knowledge Economy 53

What is Visible Learning ?

A language for progression

What is Visible Learning ?

Empowerment of Teachers in collaborative ways

Welcome screen for Teachers

Customize a test

Choose Curriculum Strands

Choose difficulty

Choose difficulty

Paper and Pencil Administered On screen Administered Computer adaptive On screen

Create a test

Welcome screen for Teachers

Individual Learning Pathways

Group Learning Pathway

Curriculum Level Report

Progress Report

What Next Report

Diagnostic advancement and appropriately targeted online references

Student Welcome Screen

Student Test

Other research options

         

Computer adaptive sequential testing Interactive video for assessing reading efficacy Target Setting Integrated with school student management system Evaluating teachers’ skill in estimating difficulty Progression maps in curricula The Human Computer Interactions – mode effects Linear programming Open-ended scoring combining technology and teachers How students read screens, effects of scrolling

 

Available (free and voluntary) for all NZ schools Used in 1000 + elementary & secondary schools

If teachers talk about the technology, we have failed If teachers talk to each other about teaching, we have succeeded

Developing Potentials for Learning: Evidence, assessment, and progress 1.

Identify major influences on achievement:

Feedback, Challenge, and Learning Strategies All effects are relative to the .40 typical value

2.

Outline an assessment model based on these influences:

An emphasis of feedback to the teacher

3.

To demonstrate the power and joy of Educational Psychology [email protected]

Developing Potentials for Learning:

Evidence, assessment, and progress John Hattie Visible Learning Lab University of Auckland New Zealand EARLI 2007