Transcript presentation3_bolt

Applying SGP to the STAR
Assessments
Daniel Bolt
Dept of Educational Psychology
University of Wisconsin, Madison
Some Unique Features of STAR
Assessments
• Same CAT assessment is administered within and across
years (possible to generate look-up tables for the calculation
of SGP)
• Multiple STAR administrations to a student are possible
throughout the year, and can occur at different times and/or
frequencies for different students
Why SGP may be Useful with STAR
Assessments
• Different STAR scores may be associated with different
amounts of measurement error (e.g., extremely low or
high STAR scores are sometimes of questionable
validity)
• There often exists more/less variability in growth
observed across students at different initial STAR score
levels
Some Practical Issues Related to
Administration of STAR Assessments
• How frequently and at what intervals should STAR be
administered to get reliable estimates of end-of-year
scores? Does the answer depend on the initial (fall)
score of a student?
• Is there practical value in the use of SGP for
answering this question (in contrast to alternative
approaches, such as ordinary least squares---OLS--regression methods)?
Goals of the Present Study
• Examine SGP as a methodology for quantifying growth
and for studying the precision of end-of-year growth
predictions using STAR Assessments
• Compare SGP against competing methodologies (OLS
regression) in terms of their reported precision of end-ofyear predictions
Sample Sizes (Students with Fall, Winter & Spring Assessments)
Grade
Math
Reading
1
99283
100000
Early
Literacy
96877
2
100000
100000
28657
3
100000
100000
5758
4
100000
100000
554
5
100000
100000
6
100000
100000
7
86975
100000
8
72599
100000
9
20835
61919
10
13810
37455
11
9796
26278
12
5155
12763
Evaluating the Precision of
Spring Score Predictions
• How well do winter assessments improve our
predictions of end-of-year outcomes?
• Are the winter assessments more/less useful
depending on the fall score obtained by the
student?
• In answering these questions, we find it useful
to examine changes in the confidence intervals
for spring scores defined by the SGP percentile
cuts
900
800
95 %ile
80 %ile
600
700
50 %ile
20 %ile
*
5 %ile
500
STAR Math Scaled Score
1000
Example of SGP Percentile Cuts, One Covariate
Fall
Spring
Academic Year
900
600
700
800
95 %ile
80 %ile
*
*
50 %ile
20 %ile
5 %ile
500
STAR Math Scaled Score
1000
Example of SGP Percentile Cuts, Two Covariates
Fall
Winter
Academic Year
Spring
Comparison of SGP & OLS Intervals,
Math Grade 1
According
to OLS
Standard
Error of
Prediction
According
to SGP
%ile cuts
250
350
450
0
400
800 1200
Fall Score
0
400
800
Fall Score
350
450
0 200
250
250
350
450
200
250
300
350
150
250
350
450
Size of 80% Conf. Interval
150
Size of 80% Conf. Interval
150
Size of 80% Conf. Interval
Grade 1
150
SGP
Size of 80% Conf. Interval
450
800
350
OLS
400
250
0
150
Size of 80% Conf. Interval
150
Size of 80% Conf. Interval
STAR Math, Grades 1-6
Grade 2
Grade 3
600
0
0
400
400
800
Fall Score
Fall Score
Fall Score
Grade 4
Grade 5
Grade 6
800
Fall Score
300
400
500
0
400
800
Fall Score
200
600
1200
Fall Score
600
Grade 10
500
SGP
400
0
300
350
450
250
350
450
250
350
450
Size of 80% Conf. Interval
150
150
250
Size of 80% Conf. Interval
150
Size of 80% Conf. Interval
Grade 7
200
800 1200
Size of 80% Conf. Interval
Fall Score
500
OLS
400
400
300
0
200
Size of 80% Conf. Interval
200
Size of 80% Conf. Interval
STAR Math, Grades 7-12
Grade 8
Grade 9
400 800
200
200
600
1000
Fall Score
Fall Score
Grade 11
Grade 12
600 1000
Fall Score
0
400 800
Fall Score
250
350
450
250
350
450
0
SGP
Grade 4
0
400
800
Fall Score
1400
200
400
600
Size of 80% Conf. Interval
150
Size of 80% Conf. Interval
150
Size of 80% Conf. Interval
Grade 1
200 250 300 350 400
400 800
Size of 80% Conf. Interval
400
Fall Score
350
OLS
300
0
250
Size of 80% Conf. Interval
200 250 300 350 400
Size of 80% Conf. Interval
STAR Reading, Grades 1-6
Grade 2
Grade 3
400 800
0
0
400
800
Fall Score
Fall Score
Grade 5
Grade 6
400 800
Fall Score
1400
0 100
300
500
0
400 800
Fall Score
1400
200
SGP
Grade 10
600
200 600
1200
Fall Score
400
300
Size of 80% Conf. Interval
100 200 300 400
-100 0 100
0
300
500
Grade 7
100 200 300 400 500
1400
Size of 80% Conf. Interval
Fall Score
500
400 800
300
0
100
200
Size of 80% Conf. Interval
100
Size of 80% Conf. Interval
OLS
0
Size of 80% Conf. Interval
-200
Size of 80% Conf. Interval
STAR Reading, Grades 7-12
Grade 8
Grade 9
1200
0
400 800
200 600
1400
Fall Score
Fall Score
Grade 11
Grade 12
1200
Fall Score
0
200
300
400
400
600
700
800
Fall Score
250
OLS
500
150
300
50
Size of 80% Conf. Interval
100
Size of 80% Conf. Interval
250
300
900
200
300
400
300
Fall Score
Grade 3
400
500
600
700
Fall Score
900
SGP
Fall Score
Grade 4
800
200
300
400
Grade K
100
Size of 80% Conf. Interval
100
Size of 80% Conf. Interval
200
Size of 80% Conf. Interval
STAR Early Literacy, Grades K-4
Grade 1
Grade 2
300
500
700
Fall Score
900
Adding Winter Scores as Covariates
with SGP
• Using 80% interval width curves as a baseline,
we can further examine how much the intervals are
reduced when adding a winter assessment
• The decline in the 80% interval can be used as
an indicator of the added precision provided by the
winter assessment
STAR Math Example
STAR Reading Example
STAR Early Literacy Example
Some Examples of STAR Score Patterns
Subject
Fall
Winter
Spring
MATH
501
625
3
2
392
376
984
928
835
777
328
306
32
91
833
846
891
915
1346
1345
315
312
858
829
531
790
852
771
589
551
1333
1255
832
840
823
851
READING
LITERACY
Change
in 80%
Interval
(+432)
(+321)
(-208)
(-209)
(+41)
(+40)
(-148)
(-256)
(+297)
(+282)
(-242)
(-256)
STAR Math Example
100
80
60
40
20
SGP
0
Decline in Size of 80% Conf. Interval
Grade 5
0
OLS
200
400
600
Fall Score
800
1000
40
60
80 100 120
0
400
800 1200
Fall Score
100
0 200
SGP
Grade 4
0
400
800
Fall Score
120
120
40
60
80 100 120
20
60
100
140
Decline in Size of 80% Conf. Interval
20
Decline in Size of 80% Conf. Interval
20 40 60 80
Decline in Size of 80% Conf. Interval
Grade 1
20 40 60 80
800
Decline in Size of 80% Conf. Interval
80
Fall Score
60
OLS
400
40
0
20
Decline in Size of 80% Conf. Interval
20
Decline in Size of 80% Conf. Interval
STAR Math, Grade 1-6
Grade 2
Grade 3
600
0
0
400
400
800
Fall Score
Fall Score
Grade 5
Grade 6
800
Fall Score
100 150 200 250
0
400 800
Fall Score
Grade 10
200
600
1200
Fall Score
150
Fall Score
100
0
50
120
40
60
80
100
20
40
60
80
100
Decline in Size of 80% Conf. Interval
20
Decline in Size of 80% Conf. Interval
20 40 60 80
Decline in Size of 80% Conf. Interval
Grade 7
0
800 1200
Decline in Size of 80% Conf. Interval
150
OLS
400
100
0
50
Decline in Size of 80% Conf. Interval
50
Decline in Size of 80% Conf. Interval
STAR Math, Grades 7-12
Grade 8
Grade 9
400 800
200
200
600
1000
SGP
Fall Score
Fall Score
Grade 11
Grade 12
600 1000
Fall Score
STAR Reading Example
SGP
OLS
0
400 800
Fall Score
Grade 4
0
400
800
Fall Score
1400
50
SGP
45
Fall Score
0
40
40
50
60
70
40
50
60
70
20
40
60
80
100 120
Decline in Size of 80% Conf. Interval
30
Decline in Size of 80% Conf. Interval
30
Decline in Size of 80% Conf. Interval
Grade 1
35
OLS
400 800
Decline in Size of 80% Conf. Interval
0
30 35 40 45 50 55
Decline in Size of 80% Conf. Interval
25 30 35 40 45 50
Decline in Size of 80% Conf. Interval
STAR Reading, Grades 1-6
Grade 2
Grade 3
400 800
0
0
400
800
Fall Score
Fall Score
Grade 5
Grade 6
400 800
Fall Score
1400
0
20
40
60
0
400 800
Fall Score
1400
50
60
SGP
Grade 10
200
600
200 600
1200
Fall Score
10
20
30
40
50
Decline in Size of 80% Conf. Interval
20 25 30 35 40 45 50
Decline in Size of 80% Conf. Interval
36 38 40 42 44 46 48
Decline in Size of 80% Conf. Interval
Grade 7
0 10 20 30 40 50 60
1400
Decline in Size of 80% Conf. Interval
40
Fall Score
30
OLS
400 800
20
0
10
Decline in Size of 80% Conf. Interval
-20
Decline in Size of 80% Conf. Interval
STAR Reading, Grades 7-12
Grade 8
Grade 9
1200
0
400 800
200 600
1400
Fall Score
Fall Score
Grade 11
Grade 12
1200
Fall Score
STAR Early Literacy Example
120
100
80
60
40
20
SGP
0
Decline in Size of 80% Conf. Interval
Grade 1
300
OLS
400
500
600
Fall Score
700
800
900
20
40
60
400
500
600
700
OLS
800
Fall Score
10 20 30 40 50
300
-10 0
Decline in Size of 80% Conf. Interval
0
Decline in Size of 80% Conf. Interval
900
120
300
Fall Score
Grade 3
400
500
600
700
Fall Score
900
Fall Score
SGP
Grade 4
800
120
Grade K
0 20 40 60 80
Decline in Size of 80% Conf. Interval
0 20 40 60 80
Decline in Size of 80% Conf. Interval
20 30 40 50 60 70 80
Decline in Size of 80% Conf. Interval
STAR Early Literacy, Grades K-4
Grade 1
Grade 2
300
500
700
Fall Score
900
Evaluating Predicted Spring Scores in Terms of
State Proficiency Thresholds
• The accuracy of SGP and OLS predictions can also
be compared against the thresholds associated with
state-specific proficiency categories
•By assuming normally distributed residuals (with
constant variance) for OLS, SGP and OLS can each
be used to define a probability that the spring score
will exceed a predefined threshold
900
600
700
800
95 %ile
80 %ile
*
*
50 %ile
20 %ile
5 %ile
500
STAR Math Scaled Score
1000
Example of SGP Percentile Cuts against StateDefined Proficiency Category
Fall
Winter
Academic Year
Spring
Proficiency
Threshold
Comparing SGP and OLS on Accuracy of
Proficiency Predictions
MATH
Grade
Proficiency
Cuta
Percentile
READING
OLS R2,b
SGP R2,b
Proficiency
Cuta
Percentile
OLS R2,b
SGP R2,b
3
617
48
.40
.41
445
48
.53
.66
4
709
61
.40
.40
531
52
.53
.68
5
758
60
.42
.42
592
49
.52
.69
6
810
65
.46
.46
647
42
.52
.70
7
823
74
.48
.49
825
51
.55
.74
8
859
68
.47
.48
933
53
.55
.73
aEstimated
STAR cutscores for the Tennessee Comprehensive Assessment Program (TCAP)
bEfron’s Pseudo R2
Extending SGP to Accommodate Multiple
Intermediate Assessments
• How can additional intermediate assessments be
used in SGP to further improve predicted spring
scores?
•Challenge: Handling varying-time point assessment
schedules
•One possible solution: Linear interpolation to fixed
node locations
Conclusions and Future Directions
• Our SGP analyses suggest substantial variability
in the precision of spring score predictions for
STAR Math, Reading and Early Literacy depending
on fall scores
• There is clear value in incorporating winter
assessments into SGP---the largest value occurs
for students with extreme fall scores in STAR Math,
intermediate fall scores in STAR Reading
• More experimentation needed to determine how
best to make use of multiple intermediate
assessments within SGP