Transcript Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council Dr. Robert H.

Value-Added Systems
Presentation to the ISBE Performance
Evaluation Advisory Council
Dr. Robert H. Meyer
Research Professor and Director
Value-Added Research Center
University of Wisconsin-Madison
February 25, 2011
Attainment and Gain
• Attainment – a “point in time” measure of student
proficiency
– compares the measured proficiency rate with a
predefined proficiency goal.
• Gain – measures average gain in student scores
from one year to the next
Attainment versus Gain
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Growth: Starting Point Matters
Reading results of a cohort of students at two schools
School
2006 Grade 4
Scale Score Avg.
2007 Grade 5
Scale Score Avg.
Average
Scale Score Gain
A
455
465
10
B
425*
455*
30
Grade 4 Proficient Cutoff 438
Grade 5 Proficient Cutoff 463
*Scale Score Average is below Proficient
Example assumes beginning of year testing
4
Value-Added
• A kind of growth model that measures the
contribution of schooling to student performance on
the standardized tests
• Uses statistical techniques to separate the impact of
schooling from other factors that may influence
growth
• Focuses on how much students improve on the
tests from one year to the next as measured in scale
score points
Value-Added Model Definition
• A value-added model (VAM) is a quasi-experimental
statistical model that yields estimates of the contribution of
schools, classrooms, teachers, or other educational units to
student achievement, controlling for non-school sources of
student achievement growth, including prior student
achievement and student and family characteristics.
• A VAM produces estimates of productivity under the
counterfactual assumption that all schools serve the same
group of students. This facilitates apples-to-apples school
comparisons rather than apples-to-oranges comparisons.
• The objective is to facilitate valid and fair comparisons of
productivity with respect to student outcomes, given that
schools may serve very different student populations.
A More Transparent (and Useful)
Definition of VA
• Value-added productivity is the difference
between actual student achievement and
predicted student achievement.
• Or, value-added productivity is the difference
between actual student achievement and the
average achievement of a comparable group of
students (where comparability is defined by a set
of characteristics such a prior achievement,
poverty and ELL status).
In English
Posttest =
Post-on-Pre
x Pretest
Link
Student
School
Unobserved
+
+
+
Characteristics
Effects
Factors
Value
Added
VARC Philosophy
• Development and implementation of a valueadded system should be structured as a
continuous improvement process that allows for
full participation of stakeholders
• Model Co-Build; Complete customization
– Analysis
– Reporting
• Value–added is one tool in a toolbox with
multiple indicators
VARC Value-Added Partners
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Design of Wisconsin State Value-Added System (1989)
Minneapolis (1992)
Milwaukee (1996)
Madison (2008)
Wisconsin Value-Added System (2009)
Milwaukee Area Public and Private Schools (2009)
Racine (2009)
Chicago (2006)
Department of Education: Teacher Incentive Fund (TIF) (2006 and 2010)
New York City (2009)
Minnesota, North Dakota & South Dakota: Teacher Education Institutions and Districts
(2009)
Illinois (2010)
Hillsborough County , FL (2010)
Broward County, FL (2010)
Atlanta (2010)
Los Angeles (2010)
Tulsa (2010)
Districts and States working with VARC
Minneapolis
Milwaukee
Madison
Racine
Chicago
Los Angeles
New York City
Tulsa
Atlanta
Hillsborough
County
Broward
County
Measuring knowledge
• Many factors influence what a student learns
and how their knowledge is measured
• A variety of measures, including (but not limited
to) assessments, tell us what a student knows at
a point in time.
• What are some ways we measure knowledge?
Measuring knowledge
Large scale assessments
MAP
WKCE
Daily teacher assessments
Daily Journal
Unit Project
Local assessments used by the district
Diagnostic Test
End-of-course Exam
Observations
After-school Activities Hands-on Project
The Simple Logic of Value-Added Analysis
• School Value-Added Report
– School specific data
– Grade level value-added
• Comparison Value-Added Reports
– Compare a school to other schools in the district,
CESA, or state
– Also allows for grade level comparisons
• Tabular Data available for School Report and
Comparison Reports
Attainment and Value-Added
How complex should a value-added model be?
• Rule: "Simpler is better, unless it is wrong.“
• Implies need for “quality of indicator/ quality
of model” diagnostics.
Model Features
•
•
•
•
•
•
•
•
Demographics
Posttest on pretest link
Measurement error
Student mobility: dose model
Classroom vs. teacher: unit vs. agent
Differential effects
Selection bias mitigation: longitudinal data
Test property analysis
MAP vs. ISAT
• MAP dates: September, January, May
• MAP: uses Rasch equating
– ISAT: 3PL
• MAP: slightly higher reliability - ~0.96 in
math, ~0.94 in reading
– ISAT math ~0.93, reading ~0.9
• Cut scores on MAP are determined by
equipercentile equating to ISAT
Hillsborough FCAR Average Math Developmental Scale Scores
means
2100
2000
1900
1800
1700
1600
1500
1400
1300
1200
1100
1000
02
03
04
05
06
07
08
09
10
11
GRADE_LEVEL
cohort
Cohort year is fall of first grade
1996
2002
1997
2003
1998
2004
1999
2005
2000
2006
2001
2007
12
Hillsborough Average FCAT Reading Developmental Scale Scores
means
2000
1900
1800
1700
1600
1500
1400
1300
1200
1100
1000
900
800
700
02
03
04
05
06
07
08
09
10
11
GRADE_LEVEL
cohort
Cohort year is fall of first grade
1996
2002
1997
2003
1998
2004
1999
2005
2000
2006
2001
2007
12
Hillsborough Standard Deviation of FCAT Math Developmental Scale Scores
stdevs
340
330
320
310
300
290
280
270
260
250
240
230
220
210
200
190
180
170
160
02
03
04
05
06
07
08
09
10
11
GRADE_LEVEL
cohort
Cohort year is fall of first grade
1996
2002
1997
2003
1998
2004
1999
2005
2000
2006
2001
2007
12
Hillsborough Standard Deviation of FCAT Reading Developmental Scale Scores
stdevs
410
400
390
380
370
360
350
340
330
320
310
300
290
280
270
260
250
240
230
02
03
04
05
06
07
08
09
10
11
GRADE_LEVEL
cohort
Cohort year is fall of first grade
1996
2002
1997
2003
1998
2004
1999
2005
2000
2006
2001
2007
12
Minimal correlation between initial status and value-added
Grade-Level Statewide Results
Grade
Mean
Score
N
Reliability
of ValueAdded
SD of
Score
Subject
State
Math
MN
3
59460
200.0
13.9
0.901
Math
MN
4
58346
210.8
14.6
0.916
Math
MN
5
57053
219.9
16.2
0.907
Math
MN
6
52400
226.7
16.5
0.873
Math
MN
7
47985
232.1
17.4
0.883
Math
MN
8
44227
236.4
17.9
0.823
Math
MN
9
26512
238.8
18.2
0.826
Grade-Level Statewide Results
Grade
Mean
Score
N
Reliability
of ValueAdded
SD of
Score
Subject
State
Math
WI
3
43289
199.9
13.2
0.820
Math
WI
4
44140
209.3
13.7
0.842
Math
WI
5
43822
217.3
14.8
0.849
Math
WI
6
47004
222.7
15.2
0.836
Math
WI
7
44549
228.4
16.0
0.837
Math
WI
8
43246
233.1
16.8
0.865
Math
WI
9
26427
234.0
17.7
0.862
Grade-Level Statewide Results
Grade
Mean
Score
N
Reliability
of ValueAdded
SD of
Score
Subject
State
Reading
WI
3
43139
194.8
15.1
0.736
Reading
WI
4
43671
202.9
14.4
0.780
Reading
WI
5
43668
209.7
13.8
0.737
Reading
WI
6
46233
214.0
14.2
0.719
Reading
WI
7
44616
218.3
14.0
0.792
Reading
WI
8
43251
221.7
14.1
0.826
Reading
WI
9
28066
223.3
14.4
0.843
MPS and MMSD Value-Added compared to Wisconsin
6th to 7th Grade (Nov 2006 – Nov 2007) Mathematics – State VA Model School Effects
MPS
School Effects
MMSD
School Effects
School/District VA Productivity Parameters in WKCE Scale Score Units
(Relative to State)
Visit the VARC Website
http://varc.wceruw.org/
for more information about VARC and value-added