Transcript faculty.missouri.edu

Longitudinal Analysis of MAP Achievement Growth:
Preliminary Estimates of School Effects
Sept. 15-16, 2010
Mark Ehlert
Cory Koedel
Michael Podgursky
Department of Economics, MU
CALDER, NCPI
Kansas City Area Education Research Consortium
Prepared for Missouri Technical Advisory Committee meeting. Kansas City, MO.
September 15-16, 2010
1
Overview
• Examination of emerging MOSIS data
system
• Patterns of Scale Score growth in MAP
• A Simple VAM for School Effects
– Model and results
– Covariates or not?
• Estimation of Teacher Program
Effectiveness
• Future directions
2
• Missouri is developing a sophisticated P20 data system
• IES State Longitudinal Data Grant
• Ranks very favorably compared to other
states
• Data quality is high
3
Data
• Matched Spring 2006-2009 student MAP
scores using MOSIS ID
• Exclusions
– Bad/duplicate values of MOSIS
– Students retained in grade
– Special districts
• Match rate for 4 years roughly 85% - 87%;
match rate for 1 year regularly at 95%
4
Math MAP Testing Regime
Grade
3
2006
a
2007
b
2008
c
2009
d
4
5
6
a
a
a
a
a
a
b
a
a
c
b
a
7
8
a
a
a
a
a
a
a
a
10
EoC
a
a
5
MAP Math: Average Performance
MAP Math: Average Performance Growth
By
Cohort
by Cohort
720
Average Scale Score
700
680
660
640
620
3
4
5
Grade06 = 3
6
Grade06 = 4
7
8
Grade06 = 5
6
MAP Com. Arts : Average Performance
MAP Com Arts: Average Performance Growth
By Cohort
by Cohort
720
Average Scale Score
700
680
660
640
620
3
4
5
Grade06 = 3
6
Grade06 = 4
7
8
Grade06 = 5
7
MAP Math: 2008-2009 Average Gain Score
By Grade and Decile of 2008 Performance
8
MAP Com Arts: 2008-2009 Average Gain Score
By Grade and Decile of 2008 Performance
9
Value-added models
• Why “value-added?”
• Traditional economic definition
– Business or firm value-added
• Value of output – value of inputs
• VAT
• Education analogy
– Control for initial (pre-treatment) performance
– Estimate the effect of contemporaneous inputs on education
outcomes
10
• We want to identify causal effect of inputs
– “what works”
• Treatment and control / comparison
groups
– Example: teacher training programs and
teacher effectiveness
– Class size
– Teacher credentials
11
A Simple VAM
Lagged or baseline
performance
Student
Characteristics
random
error
A i j t = f (A i t – k , S i , SCH j ) + ε i j t
Educational outcome
(e.g., test score, graduation, college attendance)
School / classroom
inputs or treatment
i – th student
j –th school or classroom
t – th year or grade
12
Gain Score
Lagged Test Scores in both subjects
A i g - A i g-1 = f (Ai g-1 (m, ca), student char, grade, year)
+ school effects + ε i t
Average Effect by school (state mean = 0)
Model estimated over all Missouri students, grades 3-8
Schools included if n > 20 student gain scores
3 gain scores x multiple grades per school
13
Appendix B – Regression Coefficients and Related Statistical Estimates
Dependent Variable
Math
Coefficient
Past Scores:
First-Year Scale Score in Math
Squared First-Year Scale Score
in Math
First-Year Scale Score in Com
Arts
Squared First-Year Scale Score
in Com Arts
Indicators for Student
Characteristics:
American-Indian
Asian/Pacific Islander
Black
Hispanic
Female
Special Education
Limited English Proficiency
Free/Reduced Price Lunch
Eligibility
In the School Less Than a Full
School Year
Indicators for Grade and Year:
Terminal-year Grade 4
Terminal-year Grade 5
Terminal-year Grade 6
Terminal-year Grade 7
Dummy for 2006-2007
Dummy for 2007-2008
Com Arts
T-Statistic
Coefficient
T-Statistic
- 0.34194**
- 0.00003**
- 30.81
- 3.37
0.19145**
0.00001
18.91
0.98
- 0.48245**
- 38.19
- 0.54505**
- 47.30
0.00054**
55.42
0.00007**
7.84
2.72
26.53
47.43
6.46
48.82
86.27
1.38
46.78
0.08011
1.49822**
1.64919**
0.09035
3.60541**
8.07978**
2.12545**
2.38827**
0.26
9.38
- 20.99
- 0.71
89.18
-123.90
- 13.32
- 51.61
- 0.92858**
4.64526**
- 4.08531**
- 0.89970**
- 2.16402**
- 6.16846**
0.24188
- 2.37397**
-
- 3.33210**
- 29.97
- 2.63134**
- 25.95
-19.46713**
-18.22465**
-13.95812**
-17.73426**
- 0.45133**
- 0.60767**
2
R = 0.251
Number of School Effects = 1,773 (except the reference school)
Sample Size(Number of Gainscores) = 926,358
** denotes that coefficient is significant at 1% level.
-157.16
-161.81
-170.22
-254.94
- 8.40
- 11.47
- 8.14487**
- 6.14417**
-16.47484**
-12.02678**
- 1.45330**
- 0.24555**
2
R = 0.308
- 72.10
- 59.81
-220.29
-189.57
- 29.67
- 5.08
-
-
14
Standardized School Effects on MAP Math Performance
vs. Percent Eligible for Free-Reduced Priced Lunch
1
Others
District X: Significant
District X: Insignificant
Significant: Significantly different from the statewide average of school effects.
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Eligible for Free/Reduced-Priced Lunch in School
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
15
Standardized School Effects on MAP Math Performance
vs. Percent Minorities
1
Others
District X: Significant
District X: Insignificant
Significant: Significantly different from the statewide average of school effects.
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Minorities
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
16
Standardized School Effects on MAP Math Performance
In Rank Order
1
Others
District X: Significant
District X: Insignificant
Significant: Significantly different from the statewide average of school effects.
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percentile Rank
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
17
Standardized School Effects on MAP Com Arts Performance
vs. Percent Eligible for Free-Reduced Priced Lunch
1
Others
District X: Significant
District X: Insignificant
Significant: Significantly different from the statewide average of school effects.
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Eligible for Free/Reduced-Priced Lunch in School
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
18
Standardized School Effects on MAP Com Arts Performance
vs. Percent Minorities
1
Others
District X: Significant
District X: Insignificant
Significant: Significantly different from the statewide average of school effects.
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Minorities
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
19
Standardized School Effects on MAP Com Arts Performance
In Rank Order
1
Others
District X: Significant
District X: Insignificant
Significant: Significantly different from the statewide average of school effects.
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percentile Rank
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
20
Standardized School Effects on MAP Performance
Com Arts vs. Math
1
Standardized School Effect for Com Arts*
Others
District X
0.5
0
-0.5
Coefficients of Correlation(ComArts, Math):
Pearson  = 0.75271
Spearman  = 0.70142
-1
-1
-0.5
0
0.5
1
Standardized School Effect for Math*
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
21
Effect of Covariates
(math results)
Model 1 = student covariates
Model 2 = no student covariates
22
Standardized School Effects on MAP Math Performance
Model 1 vs. Model 2
1
Standardized School Effect* from Model 2
Model 1. WITH student covariates
Model 2. WITHOUT student covariates
0.5
0
-0.5
Coefficients of Correlation (Model 1, Model 2):
Pearson  = 0.97989
Spearman  = 0.97581
-1
-1
-0.5
0
0.5
1
Standardized School Effect* from Model 1
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
23
Standardized School Effects on MAP Math Performance
vs. Percent Eligible for Free/Reduced-Priced Lunch
1
Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-12, School Dummies, Grade, Year, Student level covariates)
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Eligible for Free/Reduced-Priced Lunch in School
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
24
Standardized School Effects on MAP Math Performance
vs. Percent Eligible for Free/Reduced-Priced Lunch
1
Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-12, School Dummies, Grade, Year)
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Eligible for Free/Reduced-Priced Lunch in School
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
25
Standardized School Effects on MAP Math Performance
vs. Percent Minorities
1
Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-12, School Dummies, Grade, Year, Student level covariates)
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Minorities in School
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
26
Standardized School Effects on MAP Math Performance
vs. Percent Minorities
1
Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-12, School Dummies, Grade, Year)
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Minorities in School
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
27
Standardized School Effects on MAP Math Performance
vs. Percent Eligible for Free/Reduced-Priced Lunch
District X
1
Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-12 , School Dummies, Grade, Year, Student level covariates)
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Eligible for Free/Reduced-Priced Lunch in School
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
28
Standardized School Effects on MAP Math Performance
vs. Percent Eligible for Free/Reduced-Priced Lunch
District X
1
Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-12 , School Dummies, Grade, Year)
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Eligible for Free/Reduced-Priced Lunch in School
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
29
Standardized School Effects on MAP Math Performance
vs. Percent Minorities
District X
1
Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-12, School Dummies, Grade, Year, Student level covariates)
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Minorities in School
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
30
Standardized School Effects on MAP Math Performance
vs. Percent Minorities
District X
1
Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-12, School Dummies, Grade, Year)
Standardized School Effect*
0.5
0
-0.5
-1
0
20
40
60
80
100
Percent Minorities in School
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores
(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
31
Work Under Way
• Teacher training program effects
– New teachers
• Retirement system effects
– Effectiveness of teachers x retirement
behavior
32
A i g - A i g-1 = f (Ai g-1 (m, ca), student char, grade, year)
+ school effects + teacher effects + ε i t
Within school
Model estimated over all Missouri students, grades 3-8
33
A i g - A i g-1 = f (Ai g-1 (m, ca), student char, grade, year)
+ school effects + teacher char + ε i t
Within school
Model estimated over all Missouri students, grades 3-8
34
Comparative Effectiveness of
Teacher Preparation Programs
35
37 Teacher Training
Programs
36
Schools with
at least one new
teacher graduate:
Fall, 2005 – Fall, 2009
37
38
39
40
41
42
43
37 x 37 cross placement of program grads x school, gr. 4-8
PSI1
591
156
122
45
91
96
90
30
51
192
46
139
40
33
28
34
10
13
11
23
143
82
102
19
18
10
12
10
6
11
4
10
13
9
4
11
0
PSI2
156
501
193
59
41
53
38
61
28
23
201
43
15
68
160
32
68
17
11
30
37
7
7
8
19
12
40
11
4
65
2
62
0
6
6
0
2
PSI3
122
193
352
53
139
130
60
64
34
12
80
14
50
29
37
77
15
20
26
57
14
2
3
49
14
22
21
24
16
17
6
19
12
3
21
0
0
PSI4
45
59
53
157
47
32
102
13
70
20
18
11
12
10
3
20
4
9
14
12
11
2
4
6
4
0
4
2
14
5
4
3
4
12
2
0
0
PSI5
91
41
139
47
365
222
124
61
109
4
12
2
107
9
4
10
3
2
75
2
2
0
8
70
0
4
12
22
4
2
4
2
44
5
24
0
0
PSI6
96
53
130
32
222
241
88
48
77
12
11
11
62
0
13
7
6
12
52
8
2
0
6
39
2
6
0
21
20
0
3
2
22
2
2
0
0
PSI7
90
38
60
102
124
88
338
15
86
10
4
14
40
0
10
5
6
2
26
6
0
6
7
24
6
0
2
2
37
2
7
0
23
2
2
0
0
PSI8
30
61
64
13
61
48
15
97
18
4
21
4
26
10
10
8
6
10
21
6
0
4
0
18
6
2
5
12
6
6
2
0
2
0
10
0
0
PSI9
51
28
34
70
109
77
86
18
124
4
0
5
25
5
0
2
4
6
30
4
7
2
3
20
2
6
0
8
12
2
4
0
11
4
0
0
0
PSI10
192
23
12
20
4
12
10
4
4
165
4
46
5
4
8
15
2
2
2
15
20
34
30
0
6
4
2
0
0
0
0
4
0
15
0
4
0
PSI11
46
201
80
18
12
11
4
21
0
4
273
17
5
113
76
9
29
18
2
6
21
4
5
0
10
9
33
0
11
34
2
26
0
0
0
0
0
PSI12
139
43
14
11
2
11
14
4
5
46
17
115
4
4
20
6
4
2
0
6
23
11
6
0
2
5
4
0
0
2
0
5
0
3
0
0
0
PSI13
40
15
50
12
107
62
40
26
25
5
5
4
79
9
4
6
0
4
30
2
0
5
2
18
0
4
0
10
6
0
0
0
10
2
7
0
0
PSI14
33
68
29
10
9
0
0
10
5
4
113
4
9
127
20
8
6
4
2
0
8
7
3
2
4
0
18
0
10
6
0
15
0
2
0
0
0
PSI15
28
160
37
3
4
13
10
10
0
8
76
20
4
20
113
7
29
0
2
2
9
4
2
0
2
3
15
0
0
26
0
22
0
0
0
0
2
PSI16
34
32
77
20
10
7
5
8
2
15
9
6
6
8
7
65
0
0
2
26
6
0
0
3
10
13
0
0
6
0
4
0
0
2
0
0
0
PSI17
10
68
15
4
3
6
6
6
4
2
29
4
0
6
29
0
40
0
2
4
2
0
0
2
2
0
2
0
0
12
0
9
0
0
0
0
0
PSI18
13
17
20
9
2
12
2
10
6
2
18
2
4
4
0
0
0
41
0
6
0
5
0
2
2
2
0
2
24
0
2
0
0
0
0
0
0
PSI19
11
11
26
14
75
52
26
21
30
2
2
0
30
2
2
2
2
0
63
0
2
0
0
10
0
0
0
5
0
0
2
0
8
0
5
0
0
PSI20
23
30
57
12
2
8
6
6
4
15
6
6
2
0
2
26
4
6
0
65
0
0
0
0
0
2
0
2
0
2
4
0
0
0
2
0
0
PSI21
143
37
14
11
2
2
0
0
7
20
21
23
0
8
9
6
2
0
2
0
161
11
10
2
6
2
7
0
0
0
0
0
0
0
0
0
0
PSI22
82
7
2
2
0
0
6
4
2
34
4
11
5
7
4
0
0
5
0
0
11
63
10
0
2
0
0
0
10
0
0
0
0
5
0
2
0
PSI23
102
7
3
4
8
6
7
0
3
30
5
6
2
3
2
0
0
0
0
0
10
10
48
2
3
0
4
2
0
0
0
0
0
0
0
0
0
PSI24
19
8
49
6
70
39
24
18
20
0
0
0
18
2
0
3
2
2
10
0
2
0
2
54
0
0
0
13
0
0
0
0
6
0
2
0
0
PSI25
18
19
14
4
0
2
6
6
2
6
10
2
0
4
2
10
2
2
0
0
6
2
3
0
39
0
2
0
0
0
0
0
0
0
0
0
0
PSI26
10
12
22
0
4
6
0
2
6
4
9
5
4
0
3
13
0
2
0
2
2
0
0
0
0
17
0
2
0
0
2
2
0
0
0
0
0
PSI27
12
40
21
4
12
0
2
5
0
2
33
4
0
18
15
0
2
0
0
0
7
0
4
0
2
0
43
0
0
4
0
16
0
0
0
0
0
PSI28
10
11
24
2
22
21
2
12
8
0
0
0
10
0
0
0
0
2
5
2
0
0
2
13
0
2
0
20
2
0
0
0
2
0
0
0
0
PSI29
6
4
16
14
4
20
37
6
12
0
11
0
6
10
0
6
0
24
0
0
0
10
0
0
0
0
0
2
49
0
0
0
2
0
0
0
0
PSI30
11
65
17
5
2
0
2
6
2
0
34
2
0
6
26
0
12
0
0
2
0
0
0
0
0
0
4
0
0
28
0
5
0
0
0
0
0
PSI31
4
2
6
4
4
3
7
2
4
0
2
0
0
0
0
4
0
2
2
4
0
0
0
0
0
2
0
0
0
0
10
0
2
0
0
0
0
PSI32
10
62
19
3
2
2
0
0
0
4
26
5
0
15
22
0
9
0
0
0
0
0
0
0
0
2
16
0
0
5
0
37
0
0
0
0
0
PSI33
13
0
12
4
44
22
23
2
11
0
0
0
10
0
0
0
0
0
8
0
0
0
0
6
0
0
0
2
2
0
2
0
32
0
0
0
0
PSI34
9
6
3
12
5
2
2
0
4
15
0
3
2
2
0
2
0
0
0
0
0
5
0
0
0
0
0
0
0
0
0
0
0
13
0
0
0
PSI35
4
6
21
2
24
2
2
10
0
0
0
0
7
0
0
0
0
0
5
2
0
0
0
2
0
0
0
0
0
0
0
0
0
0
19
0
0
PSI36
11
0
0
0
0
0
0
0
0
4
0
0
0
0
0
0
0
0
0
0
0
2
0
0
0
0
0
0
0
0
0
0
0
0
0
5
0
PSI37
0
2
0
0
0
0
0
0
0
0
0
0
0
0
2
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
4
44
Other research
Teacher Pension Effects
How do pension rules affect workforce quality?
45