Transcript Lincolnwood School District 74

Student Growth
Models for
Principal and Student
Evaluation
A Model Description By
Benjamin Ditkowsky, Ph.D.
It’s Not Always Easy To See What Our
Students Can And Can Not Do
In Addition, Sometimes Some Things Just
Don’t Match
Can We Just Use The Same Test At Two
Points In Time (Use Gain Scores)?
 There are some problems with gain scores
 All tests are made
up of a true score and
measurement error xi = xiTRUE + ei
 Error is always present, but we don’t know how
much error is present
 When we subtract one score from another (i.e., a gain
score) error in the final score is greater than in either
of the tests from which the error originates
 xgain = x2TRUE– x1TRUE + e2 + e1
(true differences are subtracted but error propagates)
Error! There Are Many Reasons Other Than The Test
That Student Scores Vary From One Test To Another
Complex Solutions Exist, But
with complexity comes other concerns
 Do Complex
Solutions improve teaching and learning?
 Do Complex
Solutions maintain a focus on the
connection on standards?
 Do Complex Solutions applied to our data violate basic
statistical assumptions?
 Do Complex
Solutions focus on details or big ideas?
 The overly complex statistical models have not
been shown to be substantially more effective than
less complex models
Accountability Should Be
 Trustworthy

For a solution to be trustworthy it should meet all of the
assumptions upon which it is based
 Useable

For a solution to be useable, it should help teachers and
principals adjust instruction and intervention in real time
 Accessible

For a solution to be accessible it should be transparent
We Will Use Student Data For Evaluation
We Have Lots Of Data
Although, we don’t necessarily have the same data for all students…
 Complex solutions
have complex
algorithms to estimate missing
information.
 For most students…
we have enough information
 we can infer the rest.

Complex Statistical Analysis Is Unnecessary
Given sufficient evidence, complex analysis is unnecessary
If it …
walks like a duck
quacks like a duck
it probably is a duck
looks like a duck
swims like a duck
We Know What We Expect
We Use Cut Scores To Categorize Level of Achievement, and Progress
 Cut Scores are Based on
Expected Performance
 Test performance
categorized as

Exceeding Standards,

Meeting Standards,

Below Standards,

Academic Warning
is
 Progress is defined by
our assignment of the
value or worth of
improvement in the
categorical performance
of test scores from one
time to the next
Cut Scores Separate Performance Levels
Benchmark assessments can divide score into performance categories

ISAT Scores are broken down into performance designations (Academic
Warning, Below Standards, Meets Standards, Exceeds Standards)

Typical Growth in Language Development on ACCESS has been
established (W.I.D.A. Consortium, 2009)is calculated based on entry
performance level and change in composite score (typical low, and high
range growth)

R-CBM Scores reliably predict state test scores (Silberglitt & Hintze,
2005), cut scores predicting outcomes have been established by
aimsweb and within Illinois (i.e., Below Basic, Questionable, Proficient,
confidently proficient)

M-CAP scores predict State Test scores, and typical performance is
established across the township
We Are Explicit About Our Values
Growth is defined as a change in performance category
from Time 1 to Time 2

Exceptional Growth (2) as well as maintenance of exceptional
performance is highly valued, thus any score-pair that moves up two or
more categories or ending in the highest performance category is worth
two points.

Proficient Growth (1) is defined as any score-pair that moves up one
category or is maintained i.e., scores growth at a rate commensurate with
the increasing expectations of the Meeting Expectations category.

Inadequate Growth (-1) is defined as a score-pair in which the
performance category at time 1 is higher than the performance category
at time 2, or growth from time 1 to time 2 is not sufficient to move a
student from the below expectations category into the meeting
expectations category.

Unsatisfactory Growth (-2) is defined as a score-pair in which the
performance category drops by two categories from time 1 to time 2 or
ends in the Academic Warning Category.
A Value Table Weights Growth
Time 2 Performance Level
Time 1 Performance
A. Warning
Below
Meets
Exceeds
Academic Warning
-2
1
2
2
Below Expectations
-2
0
1
2
Meets Expectations
-2
-1
1
2
Exceeds Expectations
-2
-2
-1
2
Miguel Is A 2nd Grade Student
The Data we have for him include:

Type I
 ACCESS from grade K and 1
 Aimsweb - Fall and Winter
 RCBM
 MCAP (not in example)

Type II
 Vocabulary Matching Fall and
Winter
 Avenues Pre-Post Assessments for
ELLs

Type III
 Pre-Post Classroom Assessments
Miguel Is A 2nd Grade Student
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
Grade
2
2
2
2
2
Grade K ACCESS Composite
R CBM
Vocabulary Matching
Level
K
1
Fall
Winter
Fall
Winter
1.8
209
297
15 (BB)
66 (PR)
2 (M)
7 (M)
3.9
287
343
32 (QS) 50 (QS)
3 (M)
3 (B)
2.9
262
307
40 (QS)
40 (BB)
0 (U)
8 (M)
3.5
294
345
45 (PR)
72 (PR)
2 (M)
12 (E)
1.8
204
248
12 (BB) 58 (QS)
1 (B)
8 (M)
Grade
2
2
2
2
2
Grade K ACCESS Composite
Growth
Value
R CBM
Vocabulary Matching
Level
K
1
Pattern
Fall
Winter
Fall
Winter
1.8
209
297
Proficient
1
15 (BB)
66 (PR)
2 (M)
7 (M)
3.9
287
343
Excellent
2
32 (QS) 50 (QS)
3 (M)
3 (B)
2.9
262
307
Needs to Improve
0
40 (QS)
40 (BB)
0 (U)
8 (M)
3.5
294
345
Proficient
1
45 (PR)
72 (PR)
2 (M)
12 (E)
1.8
204
248
Unsatisfactory
-1
12 (BB) 58 (QS)
1 (B)
8 (M)
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
Avenues
Pre
Post
1
2
4
5
3
5
2
4
1
3
Avenues
Each ACCESS pair is assigned
aPost
Pre
growth value based on14the 25
3 the 5
comparison of scores and
2
4
expectations for growth
1
3
Unsatisfactory
(WIDA, March 2009)
Miguel is in second grade.
• Baseline: In K his language proficiency level
was 1.8
• Expectations: The low to high range for growth
was 44 to 90
• Scores: 297 – 209 = 88
His gain was 88, less than 90
• Category designation: His growth level is
considered in the proficient range
Miguel Is A 2nd Grade Student
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
Grade
2
2
2
2
2
Grade
2
2
2
2
2
Grade K ACCESS Composite
R CBM
Level
K
1
Fall
Winter
1.8
209
297
15 (BB)
66 (PR)
3.9
287
343
32 (QS) 50 (QS)
2.9
262
307
40 (QS)
40 (BB)
3.5
294
345
45 (PR)
72 (PR)
1.8
204
248
12 (BB) 58 (QS)
Grade K ACCESS Composite
R CBM
Level
K
1
Fall
Winter
1.8
209
297
15 (BB)
66 (PR)
3.9
287
343
32 (QS) 50 (QS)
2.9
262
307
40 (QS)
40 (BB)
3.5
294
345
45 (PR)
72 (PR)
1.8
204
248
12 (BB) 58 (QS)
Fall performance
was Below Basic
Expectations
Vocabulary Matching
Fall
Winter
2 (M)
7 (M)
3 (M)
3 (B)
0 (U)
8 (M)
2 (M)
12 (E)
1 (B)
8 (M)
Growth
Value
Pattern
Excellent
2
Needs to Improve
0
Unsatisfactory
-1
Excellent
2
Proficient
1
Avenues
Pre
Post
1
2
4
5
3
5
2
4
1
3
Vocabulary Matching
Fall
Winter
2 (M)
7 (M)
3 (M)
3 (B)
0 (U)
8 (M)
2 (M)
12 (E)
1 (B)
8 (M)
Avenues
Pre
Post
1
2
4
5
3
5
2
4
1
3
In Winter, Miguel’s
performance was in the
proficient range
Miguel is in second grade.
• Baseline: In Fall his score 15 WRC indicated Below Basic Performance
• Expectations: Winter Proficient Score is 65 WRC
• Growth: In Winter, Miguel scored in the proficient range, his movement up two
categories (Below to Proficient) is considered excellent growth
• Category designation: His growth level is considered in the excellent range
Fall
Grade
2
Measure
R-CBM
Winter
Spring
Below Basic Proficient Below Basic Proficient Below Basic Proficient
30
45
55
65
MeasuredEffects.com, 2010 ISAT Cut Scores
70
90
Miguel Is A 2nd Grade Student
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
Grade
2
2
2
2
2
Grade
2
2
2
2
2
Grade K ACCESS Composite
R CBM
Vocabulary Matching
Level
K
1
Fall
Winter
Fall
Winter
1.8
209
297
15 (BB)
66 (PR)
2 (M)
7 (M)
3.9
287
343
32 (QS) 50 (QS)
3 (M)
3 (B)
2.9
262
307
40 (QS)
40 (BB)
0 (U)
8 (M)
3.5
294
345
45 (PR)
72 (PR)
2 (M)
12 (E)
1.8
204
248
12 (BB) 58 (QS)
1 (B)
8 (M)
Grade K ACCESS Composite
R CBM
Vocabulary Matching
Level
K
1
Fall
Winter
Fall
Winter
1.8
209
297
15 (BB)
66 (PR)
2 (M)
7 (M)
3.9
287
343
32 (QS) 50 (QS)
3 (M)
3 (B)
2.9
262
307
40 (QS)
40 (BB)
0 (U)
8 (M)
3.5
294
345
45 (PR)
72 (PR)
2 (M)
12 (E)
1.8
204
248
12 (BB) 58 (QS)
1 (B)
8 (M)
Relatively low scores are in
the proficient range in the
fall, Miguel’s score of 2 is
considered proficient
Avenues
Pre
Post
1
2
4
5
3
5
2
4
1
3
Growth Value
Pattern
Proficient 1
Excellent
-1
Proficient
2
Excellent
2
Proficient
1
Avenues
In
Winter,
expectations
Pre
Post
for
1 VM are
2 higher, and
4
5
Miguel’s
score
increased
3
5
sufficiently to remain on
2
4
target.
1
3
Miguel is in second grade.
• Baseline: In Fall his score 2 WRC indicated his performance was typical in the Meets
category
• Expectations: Winter Proficient range is from 7 to 10
• Growth: In Winter, Miguel scored in the proficient range, his performance indicated
that his growth was consistent with expectations for his grade level (Meets to Meets)
• Category designation: His growth level is considered in the proficient range
FALL
VM
Grade
Robust Percentile Rank
5
10 25
50
75
90
2
0
0 0
2
5
8
Warning Below
Meets
Exceeds
Local Normative Values
Winter
5
1
Warning
Robust Percentile Rank
10
25
50
75
2
4
7
10
Below
Meets
90
13
Exceeds
Miguel Is A 2nd Grade Student
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
Grade
2
2
2
2
2
Grade
2
2
2
2
2
Grade K ACCESS Composite
R CBM
Vocabulary Matching
Level
K
1
Fall
Winter
Fall
Winter
1.8
209
297
15 (BB)
66 (PR)
2 (M)
7 (M)
3.9
287
343
32 (QS) 50 (QS)
3 (M)
3 (B)
2.9
262
307
40 (QS)
40 (BB)
0 (U)
8 (M)
3.5
294
345
45 (PR)
72 (PR)
2 (M)
12 (E)
1.8
204
248
12 (BB) 58 (QS)
1 (B)
8 (M)
Grade K ACCESS Composite
R CBM
Vocabulary Matching
Level
K
1
Fall
Winter
Fall
Winter
1.8
209
297
15 (BB)
66 (PR)
2 (M)
7 (M)
3.9
287
343
32 (QS) 50 (QS)
3 (M)
3 (B)
2.9
262
307
40 (QS)
40 (BB)
0 (U)
8 (M)
3.5
294
345
45 (PR)
72 (PR)
2 (M)
12 (E)
1.8
204
248
12 (BB) 58 (QS)
1 (B)
8 (M)
The Avenue’s pre test is broken into 6
levels designated in 3 levels (Beginning,
Intermediate and Advanced)
Avenues
Pre
Post
1
2
4
5
3
5
2
4
1
3
Avenues
Pre
Post
1
2
4
5
3
5
2
4
1
3
Growth Value
Pattern
Proficient
1
Excellent
-1
Excellent
2
Excellent
2
Excellent
1
Miguel is in second grade.
• Baseline: In Fall his proficiency level was 1 indicating early beginning language
proficiency
• Expectations: Winter Intermediate language range is from 3 to 4
• Growth: In Winter, Miguel’s language level increased from level 1 to 2, though not up
to the Intermediate range, his scores demonstrated growth
• Category designation: His growth level is considered in the proficient range
A Demonstration Of Proficient Growth
ACCESS Composite
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
Grade
2
2
2
2
2
Pattern
Proficient
Excellent
Needs to Improve
Proficient
Unsatisfactory
Value
1
2
0
1
-1
R CBM
Vocabulary Matching
Growth
Pattern
Value
Excellent
2
Needs to Improve
0
Unsatisfactory
-1
Excellent
2
Proficient
1
Pattern
Proficient
Excellent
Proficient
Excellent
Proficient
Value
1
-1
2
2
1
Avenues
Pattern
Proficient
Excellent
Excellent
Excellent
Excellent
1
-1
2
2
1
Group Rating
j
Individual Growth
1 Proficient
-1 Unsatisfactory
1 Proficient
2 Excellent
1 Proficient
1 Proficient
• Each score, for each second grade student in the targeted ELL subgroup has been
examined and categorized based on Cut Scores.
• Each available score-pair has been reviewed and growth has been categorized and
weighted.
• Individual ratings were calculated
mMiguel = m[1,2,1,1] = 1
• The overall group rating was calculated
Rating = m[1, -1, 1, 2, 1] = 11
Proficient Growth
Alternate Illustration: Why Use More Than Two
Administrations Of The Same Test From One Source
It is possible that different tests indicate different patterns, without
changing the overall rating
ACCESS Composite
Name
Miguel
Augustine
Isabelle
Fariha
Nabiha
j
Grade
2
2
2
2
2
Pattern
Proficient
Excellent
Needs to Improve
Excellent
Excellent
Excellent
Value
1
2
0
2
2
2
While there may be some areas
where progress is worth further
investigation, for the purposes of
evaluating
and
R CBM
MCAPcategorizing;
Vocabulary Matching
Growth
Patternoverall,
Value across
Pattern measures,
Value
Pattern
Value
Excellent
2
Needs to Improve
0
growth
is occurring
Needsacademic
to Improve
0
Needs to Improve
0
Excellent
-1
Unsatisfactory
-1
Excellent
2
Excellent
2
at an
acceptable
rate.
Needs to Improve
0
Proficient
1
Needs to Improve
0
Avenues
Unsatisfactory
-1
Needs to Improve
0
Proficient
1
Pattern
Proficient
Unsatisfactory
Needs to Improve
Proficient
Proficient
Needs to Improve
0
Proficient
1
Needs to Improve
0
Proficient
1
-1
0
1
1
Individual Growth
1 Proficient
0 Unsatisfactory
0 Proficient
1 Excellent
1 Proficient
1
1 Proficient
The differences in patterns by test may be diagnostically important, but
insufficient for high stakes evaluative purposes
Timmy Is A 4th Grade Student
What Data Might We Use For Him?
The data we might have for him include:

Type I
 ISAT from grade 3 and 4
 Aimsweb - Fall and Winter
 RCBM
 MCAP

Type II
 Vocabulary Matching

Type III
 Pre-Post Classroom Assessments
Gertrude Is A 7th Grade Student
What Data Might We Use For Her?
The data we might have for her include:

Type I
 ISAT from grade 3 and 4
 Aimsweb - Fall and Winter
 RCBM
 MCAP

Type II
 Vocabulary Matching
 Prentice Hall Benchmark Assessments

Type III
 Pre-Post Classroom Assessments
Setting Student Data Goals For Principal Evaluation
Goals can be set to increase the amount
of progress made
By February 201x, given available Type I and Type II assessments
administered at two points in time school wide, students at school
name will demonstrate an increase in the proportion making
adequate progress from 35% to 40%.
Setting Student Data Goals For Principal Evaluation
Goals can be set to increase the
proportion of students making adequate
progress made for a particular subgroup
By February 201x, given available Type I and Type II assessments
administered at two points in time school wide, students identified
as define cohort at school name will demonstrate an increase in the
proportion making adequate progress from 68% to 80%.
Setting Student Data Goals For Principal Evaluation
Goals can be set to decrease the
proportion of students not making
adequate progress for a particular
subgroup
By February 201x, given available Type I and Type II assessments
administered at two points in time school wide, students identified
as define cohort at school name will demonstrate a decrease in the
proportion making unsatisfactory progress from 8% to 4%.
Setting Student Data Goals For Principal Evaluation
Goals can be set to achieve an overall
rating for the demonstration of student
growth
By February 201x, given available Type I and Type II assessments
administered at two points in time school wide, students at school
name will demonstrate proficient or excellent growth as defined by
the convergence and magnitude of data classified with district
defined value tables.