Transcript Growing Beyond a Pilot: Product Integrity in Environments

Formative Assessment and Student
Achievement: Two Years of
Implementation of the
Keeping Learning on Track® Program
Courtney Bell (ETS)
Jonathan Steinberg (ETS)
Dylan Wiliam (Institute of Education, University of London)
Caroline Wylie (ETS)
Keeping Learning on Track ® Program

Developed from research-basis around formative
assessment (Crooks, 1997; Natriello, 1998; Kluger & DeNisi; 1996;
Black & Wiliam, 1998c; Nyquist, 2003; Brookhart, 2005) and the power
of teacher learning communities (Grossman, Wineburg, &
Woolworth 2001; Thompson, & Goe, 2006)

Five research-based strategies (Leahy et al., 2005)





Engineering effective classroom discussions, questions, and
learning tasks that elicit evidence of student learning;
Clarifying and sharing learning intentions and criteria for success;
Providing feedback that moves learners forward;
Activating students as the owners of their own learning;
Activating students as instructional resources for one another.
KLT Components



2-day introductory workshop
2-day TLC leader workshop
Monthly school-based TLC meetings


Structured and supported by TLC Leader
modules
Sufficient content for two years of monthly
meetings
Working in Cleveland
2005-06
 Work began with 10 schools (Cohort A)
2006-07
 Nine of the original schools (1 closed) continued
 Additional 5 new schools (Cohort B)
All Schools
 All in third or fourth year of failing to make Adequate
Yearly Progress
Details of sample
Grade
3
4
5
6
7
8
Mathematics
200520062006
2007
3093
72
3073
3073
3162
3068
3442
3156
3703
3445
69
3728
Reading
200520062006
2007
3093
72
3074
3058
3158
3069
3450
3155
3717
3454
71
3670
Numbers of students in each grade in CMSD, 2005-2006 and 2006-2007
Data Used

2005-06 and 2006-07 Ohio Achievement Tests for Grades 3-8

Reliability



CMSD scores well below state mean





0.65 to 0.83 sd below state mean in math
0.60 to 0.71 sd below state mean in reading
High student mobility


Math: 0.85 to 0.90
Reading: 0.87 to 0.89
4400 of 16500 students changed school from 05-06 to 06-07
400 students were retained in grade or were promoted by more
than one grade from 05-06 to 06-07
Analysis excluded these students
No data available on:


Teacher assignment
Fidelity of implementation
Data analysis

Univariate HLM






Dependent variable: 2006-2007 test scores
Students nested in schools, schools nested in
treatment
2005-2006 scores used as a covariate
No significant treatment effect
p-values typically well over 0.5
Suggests weak effects rather than low power
Exploratory analyses

Cleveland school data from the OH DOE website to identify
matched schools
 Number of students
 Proportion of LEP students
 % of longevity (measure of teacher turnover)
 % of African American students
 Accountability designation in 06-07 (defined by state
accountability system)
 Year of improvement status (defined by state accountability
system)
 % of core courses not taught by a highly qualified teacher
 If a tie-breaker was needed, the final category was % of special
education students
Exploratory results: math
KLT
Grade
1 year
Non-KLT
2 year
All
1 year 2 year
p-value
4
1.45
0.26
0.03
0.59
0.63
0.89
5
1.31
1.45
0.07
0.14
0.07
0.60
6
0.23
0.16
0.84
0.84
0.06
0.84
7
0.52
0.14
0.19
0.09
0.90
0.89
8
0.87
0.24
0.63
1.74
0.00
0.84
Model coefficients for 2006-2007 math scores, in standard deviations (positive/negative)
Exploratory results: reading
KLT
Grade
1 year
Non-KLT
2 year
All
1 year 2 year
p-value
4
0.77
0.80
0.54
1.47
0.87
0.52
5
1.11
1.49
0.87
0.66
0.59
0.10
6
0.37
0.69
0.20
0.53
0.37
0.82
7
0.59
0.63
0.04
0.59
0.59
0.75
8
0.79
0.29
1.29
1.49
0.29
0.08
Model coefficients for 2006-2007 reading scores, in standard deviations (positive/negative)
Comparison of matched schools
Grade
4
5
6
7
8
1 year
0.70
0.45
0.90
0.00
2.28
2 year
0.07
2.08
1.06
0.04
0.00
KLT schools - matched non-KLT schools (Coefficients for
reading scores in standard deviations)
Discussion


No statistically significant effect of KLT
Exploratory analyses



Limitations




Math: no evidence of effect
Reading: some evidence in earlier grades
Fidelity of implementation
Small effects
Low statistical power
Next steps

Research designs based on “theory of action” of KLT
 Impact of training on teacher behaviors
 Impact of changes in teacher behaviors on students:


Attitudes
Outcomes
For More Information
To view, download and print the papers:
http://www.ets.org/research/aerancme08.html
Conference files will be posted on the web for one
month