Transcript NCME 2008 Invited Symposium

Measuring Academic Gains
in Arkansas
Education Commission of States
July 2, 2008
Austin, Texas
Dr. Charity Smith, Assistant Commissioner
Arkansas Department of Education
US Department of Education
Growth Model Pilot Study
 7 core principles of state-proposed growth models
100% proficiency by 2014
 Establish appropriate growth targets at student level
 Separate accountability decisions for English Language Arts and
Mathematics
 Inclusion of all students
 State assessment system and methodology
 Tracking student progress
 Student participation rates and additional academic indicators
(USDOE, 2005)

Two Systems for Measuring Growth
 Arkansas Federal Growth
Model

Difference in Gain Scores
Model (DGSM)
 Arkansas Act 35 Gains
Model

Transition Matrix Model
(TMM)
Difference in Gain Scores Model
 Difference between a student's current score and the score that would
meet standards in a set number of years
 Dividing the difference by the number of years gives the annual gain
needed. A student's actual gain can be compared to the target growth to
see if the student is on track to meet standards.
 Different scores can be aggregated to the school or district level to
obtain a group growth measure.
(CCSSO, 2007)
Arkansas Model DGSM
 Arkansas (USDOE, 2007)


Scale Score and Proficient Level is grade and subject specific
Annual growth increment is a function of the following:
 Proficiency Scale Score Standard for Subsequent Grade
 Proficiency Scale Score Standard for Current Grade
 Proficiency Scale Score Standard at Grade 8
 Student’s current grade scale score
PSSSG  PSSCG
GI
PSSG 8  PSSCG
 PSSG 8 


 SSCG 
Technical Advisory Committees
(Act 35 Assessment)

William J. Brown, Jr., Brownstar, Inc.
 William Brown, Jr., Brownstar, Inc.,

Gregory J. Cizek, University of North
Carolina at Chapel Hill
 Thomas Fisher, Fisher Education
Huynh Huynh, University of South
Carolina, Chair
 Huynh Huynh, University of South
Robert W. Lissitz, University of
Maryland
 Eugene Kennedy, Louisiana State




William A. Mehrens, Michigan State
University (Emeritus
Professor)
E. Roger Trent, Trent Consulting Inc.
Chair
Consulting LLC
Carolina, Chair
University
 Robert Kennedy, University of
Arkansas for Medical Sciences
 E. Roger Trent, Trent Consulting
Inc.
Arkansas Act 35 Accountability Team
 Charity Smith, Assistant Commissioner
 Willie Morris, Associate Director
 Louis Ferren, Data Manager
 Kimberly Millins, Special Projects Director
 Oliver Dillingham, Program Manager
School Rating System
 The Department of Education shall prepare an annual
report which shall describe the school rating system. The
annual report shall designate two (2) category levels for
each school.
 The first category, annual performance, is based on
performance from the prior year on the criterion-referenced
test and end-of-course exams.
 The second category, growth, shall be based on the schools’
improvement gains tracked longitudinally and using valueadded calculations on the criterion-referenced test.
 This presentation will focus only on the second category.
Intention of Act 35
 Promoting student learning at all levels so that “all students have an
opportunity to demonstrate increased learning” and “meet the expected
academic standards.”
 Improving student achievement through school accountability and
recognition
 Requiring that each school be classified into “two (2) category levels” as
follows:
-- “Category One” for the “school’s improvement gains” tracked
longitudinally using value-added calculation known as the annual
improvement category level and
--
“Category Two” based on “performance from the prior year”
referred to as the annual performance category (or “status”).
Annual Improvement Category
 The annual improvement category for rating schools will report each
school’s improvement gains by tracking student’s longitudinal
achievement gains on the state’s augmented criterion-referenced tests.
 The base year for the growth model is 06-07. The first reports will be
available in December of 2008.
 A school’s annual improvement gain is based upon the changes in
student achievement from one year to the next.
 No value will be added if instruction does not move a student’s
achievement from a given performance category to a higher
performance category.
 If a student’s achievement moves to a lower performance category, then
value is lost.
Initial Design Considerations
 Growth (“Improvement Gain”) will be based on the tests
included in the Arkansas Comprehension Testing,
Assessment and Accountability Program (ACTAAP).
 An annual improvement gain model will be based on an
aggregation of student changes in literacy and mathematics
for grades 3-8 and for literacy in grade 11.
 So far work has been conducted using test data for Grades
3-8 only.
 The annual improvement gain model shall be designed with
the expectation that (a) students who are proficient or
higher will either maintain or improve their performance
classification, and (b) students who are basic or below basic
will reach proficiency.
Annual Improvement Category Levels
 Level 1: Schools in need of immediate improvement
 Level 2: Schools on alert
 Level 3: Schools meeting improvement standards
 Level 4: Schools exceeding improvement standards
 Level 5: Schools of excellence for improvement
Calculations for Annual Student Gains
 Student growth is based upon changes in student
performance levels across two adjacent years.
 To assess annual changes more precisely each student
performance level (Below Basic, Basic, Proficient, and
Advanced) is split into two sub-categories.
 The subcategories are: BB1, BB2, Basic1, Basic2, Prof1,
Prof2, Adv1, and Adv2.
 For example, if a student in third grade earns a achievement
rating of Proficient 2 and the following year the same
student in fourth grade receives an achievement rating of
advanced 1, he/she would be given a value-added score of
+0.5. Similar value-added scores would be calculated for
each child in each school.
Table 1: Student Performance
Sub-categories
Student Performance Sub-categories Points Assigned
Below Basic 1
1
Below Basic 2
1.5
Basic 1
2
Basic 2
2.5
Proficient 1
3
Proficient 2
3.5
Advanced 1
4
Advanced 2
4.5
Table 2: Value-added Point for Changes in
Student Achievement
Current Year
Previous
Year
1
1.5
2
2.5
3
3.5
4
4.5
1
0
.5
1
1.5
2
2.5
3
3.5
1.5
-.5
0
.5
1
1.5
2
2.5
3
2
-1
-.5
0
.5
1
1.5
2
2.5
2.5
-1.5
-1
-.5
0
.5
1
1.5
2
3
-2
-1.5
-1
-.5
0
.5
1
1.5
3.5
-2.5
-2
-1.5
-1
-.5
0
.5
1
4
-3
-2.5
-2
-1.5
-1
-.5
0
.5
4.5
-3.5
-3
-2.5
-2
-1.5
-1
-.5
0
Literacy Grade 4
Gain
Points
Count
Total Gains
1
-2.0*1 = -2
Math Grade 4
Count
Total Gains
-2.5
-2.0
-1.5•
-1.0
-0.5
In 2007,
improvement
gain index ranged from
2 the school -1.5*2
= -3
approximately
-0.50-1.0*3
to 0.50.
3
= -3
2
-1.0*2 = -2
4
-0.5*4 = -2
3
-0.5*3 = -1.5
• Conceptually,
all students
performing
5
0.0*5 = in
0 the highest
5
0.0*5schools
=0
up one sub-category.
0.5 moved 3
0.5*3 = 1.5
4
0.5*4 = 2
0
1.0
1.5•
2.0
Sum
3
1.0*3 = 3
3
1.0*3 = 3
21
-5.5
19
4.5
Conceptually, all students in the lowest
performing
2
1.5*2schools
=3
moved down one sub-category.
Sum of all total gains = -5.5 + 4.5 = -1.0 Sum of Counts = 21 + 19 = 40
School Improvement Gain Index = Sum of Total Points / Sum of Counts
= -1.0/40 = -0.025
How School Ratings Are Calculated
Multiply the number of students (5) by the points assigned to the category. For example
multiply (5) times (4) to get the points produced by the students in the Advanced category.
Add up the points for each category. Divide the total points for the school (285) by the
number of students (100) to get a rating score (2.85), which is meeting standards.
Number of Students
Categories
Points Assigned to Categories
Total
5
Below Basic 1
1
5
5
Below Basic 2
1.5
7.5
10
Basic 1
2
20
20
Basic 2
2.5
50
30
Proficient 1
3
90
20
Proficient 2
3.5
70
5
Advanced 1
4
20
5
Advanced 2
4.5
22.5
Total Points for the school for all categories
285
Computation of Annual Improvement
Gain Index
 For each school, the Annual Improvement Gain Index is
the average of all value-added points across subject areas
and grades.
 Potentially, the range of the Annual Improvement Gain
Index is from -3.5 to +3.5, but in practice the range is
much smaller.
 The value of zero indicates no growth, positive values
indicate improvement, and negative values indicate a
decline in achievement.
Stakeholder Field Review of Gain Index
Procedures for Setting Cut Scores for Annual
Improvement Category Levels (Gains)
 Schools will be classified into one of five Annual
Improvement Gain Levels. Therefore, four (4) cut scores
will need to be set.
 The steps are: (1) preliminary work by the Technical
Advisory Committee on Accountability, (2) advice by
Arkansas stakeholders at a meeting, and (3) adoption by
the Arkansas State Board of Education.
Preliminary Work by the Technical
Advisory Committee on Accountability
 The distribution of the School Annual Improvement
Gain Index was compiled based on the matched
Spring 2006 and Spring 2007 data.
 Improvement Gain index was computed only for
schools with complete data on Literacy and Math for
at least 40 students.
Preliminary Work by the Technical
Advisory Committee on Accountability
 The distribution of the School Annual Improvement
Gain Index was compiled based on the matched
Spring 2006 and Spring 2007 data.
 Improvement Gain index was computed only for
schools with complete data on Literacy and Math for
at least 40 students.
Schools with 40+ Students 2006/2007
School Mean Change
N
Mean
Median
Mode
Std Dev
Skewness
Min
Max
798
0.0277
0.0300
0.0900
0.1468
-0.0800
-0.512
0.504
Histogram
0.525+ *
.
.*
. **
. *****
. **********
. ****************
0.175+**********************
. *******************************
. **************************************
. ***********************************
. *********************************
. ******************************
. ******************
-0.175+*************
. ********
. ****
. ***
.*
.*
.
-0.525+*
----+----+----+----+----+----+----+---
#
2
Box plot
0
3
6
13
28
48
65
91
112
105
99
90
54
37
23
10
7
1
3
0
|
|
|
|
|
+-----+
|
|
*--+--*
|
|
+-----+
|
|
|
|
|
0
0
1
0
Understanding the School Improvement
Gain Index
 In 2007, the school improvement gain index ranges
from approximately -0.50 to 0.50.
 Conceptually, all students for a school at the lowest
index of -0.50 move down one sub-category.
 Conceptually, all students for a school at the highest
index of 0.50 move up one sub-category.
Technical Characteristics of School
Improvement Gain Index



Reliability assessed via the Spearman-Brown formula = 0.922
Global standard error of measurement = 0. 041
Conditional standard error of measurement (Rulon Method)
School Number of Mean Gain CSEM
Group
Schools
Index
1
79
-0.234
0.055
2
80
-0.123
0.038
3
80
-0.071
0.040
4
80
-0.030
0.048
5
80
0.012
0.043
6
80
0.050
0.036
7
80
0.087
0.032
8
80
0.123
0.035
9
80
0.181
0.037
10
79
0.282
0.039
TAC Rationale for Setting Preliminary
Cut Scores
 Four cut scores are needed. A cut score is the lowest mean change for a
school in each school level.
 Cut Score 1 for Level 1/Level 2: 25% of the students move down one sub-
category and the other 75% remain in the same performance sub-category.
 Cut Score 2 for Level 2/Level 3: all students remain in the sub-category
where they were in the previous year. This cut reflects no growth on the
school improvement scale.
 Cut Score 3 for Level 3/Level 4: 25% of the students move up one sub-
category and the other 75% remain in the same performance sub-category.
 Cut Score 4 for Level 4/Level 5: 50% of the students move up one sub-
category and the other 50% remain in the same performance sub-category.
 TAC preliminary cut scores are: Cut score 1 = -0.12; Cut score 2 = 0.01; Cut
Score 3 = +0.13; and Cut Score 4 = +0.25.
Creating Ordered School Booklet
 The global standard error of measurement is about 0.04.
 The TAC preliminary lowest cut score is -0.12 and highest cut
score is +0.25, with a range from -0.12 to +0.25.
 In creating the Ordered School Booklet (OSB), a decision was
made to consider to extend this range by two SEMs on both
ends.
 All schools with a mean rating between -0.21 to +0.32 were
candidates from the OSB.
 Schools were stratified using mean rating (rounded to 0.01).
 One school was selected to represent each stratum.
 There are 54 representative schools in the OSB.
Base Year School Improvement
Gain Index Definitions
• In 2007, the school improvement gain index ranged from
approximately -0.50 to 0.50.
• Conceptually, all students in the highest performing schools
moved up one sub-category.
• Conceptually, all students in the lowest performing schools
moved down one sub-category.
Data for School Profile
• Information provided:
– Type of School
– Grade range
– 2007 enrollment
– Racial composition
– Mean of school improvement gain
– Performance status in 2006 and 2007
– Percentage of students on free or reduced price lunch,
with limited English proficiency, or in special education
– Percentage of mobile and migrant students
School Profile in Each Preliminary Level
• Level 1: “Schools in need of immediate improvement”
(116 schools)
• Level 2: “Schools on alert” (228)
• Level 3: “Schools meeting standards” (261)
• Level 4: “Schools exceeding standards” (140)
• Level 5: “Schools of excellence” (53)
Note: This preliminary level analysis includes high schools.
Initial Considerations for
Preliminary Cut Scores: Conceptual
Framework
• Spread of mean changes by level:
–
–
–
–
–
Level 1:
Level 2:
Level 3:
Level 4:
Level 5:
-0.51 to -0.13
-0.12 to 0.00
0.01 to 0.12
0.13 to 0.24
0.25 to 0.50
Panelists set cut points where they felt comfortable.
Composition of the Panel
• Facilitators: black female, black male, Hispanic male, white
female, and white male
• Panelists also racially and geographically diverse: PTA,
business, AAEA, AEA, ASBA
• Each group named 12 representatives, for a total of 60
Panelists (a small number were unable to participate)
• Monitored by TAC/Accountability
Procedures for Setting Cut Scores for Annual
Improvement Category Levels (Gains)
 Schools will be classified into one of five Annual
Improvement Gain Levels. Therefore, four (4) cut
scores will need to be set.
 The steps are: (1) preliminary work by the Technical
Advisory Committee on Accountability, (2) advice by
Arkansas stakeholders at a meeting, and (3)
adoption by the Arkansas State Board of Education..
Advice by Arkansas Stakeholders
 A meeting of Arkansas stakeholders was convened to
recommend Annual Improvement Gain cut scores to the
Arkansas Department of Education February 7, 2008 at
the Crowne Plaza Hotel in Little Rock.
Arkansas PTA
Louis Ferren
Arkansas Education Association
Dr. Andre’ Guerrero
Arkansas Association of Educational
Administrators
Willie Morris
Economics Arkansas
Dr. Charity Smith
Arkansas School Boards Association
Janine Riggs
Standard Setting Recommendations
Stakeholder Advisory Panels
Cut Scores
Cut 1/2
Cut 2/3
Cut 3/4
Cut 4/5
Administrators
-0.20
-0.01
0.10
0.20
Teachers
-0.15
0.00
0.11
0.24
Business
-0.12
0.01
0.13
0.25
Parents
-0.09
0.01
0.15
0.27
School Board
-0.12
0.01
0.13
0.25
Median
-0.12
0.01
0.13
0.25
Recommended Cut Scores by Arkansas
Stakeholder Advisory Team
Recommended cut scores are:
• Cut score 1 = -0.12
• Cut score 2 = 0.01
• Cut Score 3 = +0.13
• Cut Score 4 = +0.25
Note: Recommended cuts are the median scores from all
five stakeholder groups.