Transcript Stricom 1.21.9 C

C R E S S T / CU
Measuring Adequate Yearly
Progress
Robert L. Linn
University of Colorado at Boulder
National Center for Research on Evaluation, Standards, and Student Testing
CRESST Conference, UCLA, September 10-11, 2002
C R E S S T / CU
Adequate Yearly Progress
(AYP)
 Central
to the Accountability System of
the No Child Left Behind (NCLB) Act of
2001
 States
required to define AYP for the
State, school districts, and schools in a
way that enables all children to meet
the States student achievement
standards by 2014
C R E S S T / CU
Some Key Criteria for State’s
AYP Definitions
 Same
high standards of academic
achievement to all public elementary
school and secondary school
students in the State
 Statistically
 Results
valid and reliable
in continuous and substantial
academic improvement for all
students
C R E S S T / CU
Some Key Criteria for State’s
AYP Definitions (Continued)
 Annual
measurable AYP

Mathematics

Reading/language arts

Substantial and continual progress
 All
students considered as a whole
 Subgroups
of Students
C R E S S T / CU
Subgroups of Students
Identified for AYP
 Economically
 Major
disadvantaged students
racial and ethnic groups
 Students
with disabilities
 Students
with limited English proficiency
C R E S S T / CU
AYP Starting Point

Starting point defined in 2001-2002

The larger of either
1.
The percentage of students in the lowest
scoring subgroup who achieve the proficient
level or higher, or
2.
The percentage proficient or higher in the
school at the 20th percentile, based on
enrollment, among all schools ranked by the
percentage of students at the proficient level
or higher.
C R E S S T / CU
Annual Measurable
Objectives
 Separate
for mathematics and
reading/language arts
 Straight
line increase from starting
points in 2221-2002 to 100% in 2013-2014
 May
combine across grade levels within
a subject for school, district, or State
C R E S S T / CU
Illustrations of AYP Targets

Starting points 52% proficient or above in
reading/language arts and 40% proficient or
above in mathematics

Annual gains required


Reading/language arts: 4% = (100% - 52%)/12

Mathematics: 5% = (100% - 40%)/12
2003-2004 targets:

Reading/language arts: 60% = 52% + 2(4%)

Mathematics: 50% = 40% + 2(5%)
C R E S S T / CU
Requirements for School to Meet
AYP Target

Meet measurable objectives in reading/language
arts and mathematics for all students considered
as a whole

Meet objects for subgroups:

Economically disadvantaged

Major race and ethnic groups

Students with disabilities

Students with limited English proficiency
C R E S S T / CU
“Safe Harbor” Exception

If one subgroup fall short of AYP target
school can still avoid being placed in needs
improvement category if
1.
The percentage of students who score
below the proficient level is decreased by at
least 10% from the year before, and
2.
There is improvement for that subgroup on
other indicators
C R E S S T / CU
Statistically Reliable Results
Disaggregated reporting is required only if
1.
Results are statistically reliable, and
2.
The identity of individual students will
not be revealed
Statistical reliability requirement leads to
need to establish minimum number of
students for subgroup reporting
C R E S S T / CU
Minimum Number of Students

There is no number below which results have
zero statistical reliability and above which the
statistical reliability is good

Statistical reliability increases as a function of
the square root of the number of students

Relevant statistic is the standard error of the
difference between percentages for two
independent samples
C R E S S T / CU
The Standard Error of the Difference Between Percentages for Two
Independent Samples as a Function of the Number of Students in
Each Sample When the Average Percentage is 50
Number of Students in Each
Sample
Standard Error of Difference
in Percentages
10
22.4
20
15.8
30
12.9
40
11.2
50
10.0
C R E S S T / CU
Minimum Number of
Students: Implications

The number of students needed each year to
have a standard error as small as 10% is 5O.

Even with 50 students in a category each year
about 1 time in 6 the percentage of students in
the sample who are proficient would be no
larger in year 2 than in year 1 event when the
instruction had improved enough to increase
the percentage proficient for an indefinitely
large number of students by 10%.
C R E S S T / CU
Minimum Number of
Students: Implications
(Continued)
If the minimum number of students in a
category were set at 50, the number of
groups that would qualify for
disaggregated reporting would be
relatively small at most schools.
C R E S S T / CU
Minimum Number of Students:
Implications (Continued)
Tradeoff between competing goals
1.
More disaggregated reporting, and
2.
Improved statistical reliability
Compromise between competing goals
may be best solution, e.g., minimum n
of, say, 25, rather than 10 or 50
C R E S S T / CU
Variability in Stringency of Progress
Requirements for Different Schools: 20032004 AYP Target is 50%
Percentage of students
proficient or above in
2001-2002
Needed Increases in
percent proficient or
above by 2003-2004

School A: 30%

School A: 20%

School B: 45%

School B: 5%

School C: 75%

School C: could
decline by up to 25%
C R E S S T / CU
Scatterplot of Percent Proficient or Above for Schools on Colorado
Grade 4 Reading Assessments in 1997 and 1999
120
100
80
60
40
AYP Status in 1999
20
Met AYP
0
Failed to Meet AYP
0
20
40
60
80
100
Percent Proficient or Above in 1997
120
C R E S S T / CU
Cross Tabs of Schools Percent
Proficient or Above Standing in 1997
vs. 1999
Column 1999
0 to 49.99%
50 to 100%
Total
126
23
148
(84.5%)
(15.5%)
90
504
(15.2%
(84.8)
215
527
Row 1997
0 to 39.99%
40 to 100%
Total
594
742
C R E S S T / CU
Index Scores
 Proficient
level vs. below only credits
changes across proficient level cut score
 Improvements
in partially proficient
region not recognized by dichotomous
system
 Index
scores can credit improvements
below proficient level and still be
compatible with NCLB goals
C R E S S T / CU
Definitions of AYP Based on
Longitudinal Data
Advantages:

Individual student growth used as basis of
measuring progress

Past achievement taken into account without
assumptions needed in successive groups
approach

Only students who attend school a full year
contribute to school accountability measure
C R E S S T / CU
Definitions of AYP Based on
Longitudinal Data (Continued)
Challenges and disadvantages:

Matched student records more difficult to track for
mobile students than less mobile students

Need to take account of students with only a single
assessment result for the the year AYP is assessed

Quasi-longitudinal approach as an alternative
C R E S S T / CU
Secretary Paige
“The purpose of the statute, for both
assessments and accountability, is to build
on high quality accountability systems that
States already have in place, not to require
every State to start from scratch. Therefore, I
want to assure you that the Department will
work with States so that they nave the tools
they need to implement definitions of AYP
that meet the requirements of the statute and
maintain high standards” (Secretary Paige,
www.ed.gov/News/Letters/020724.html).