Transcript Slide 1 Snapshots: What We Know about State Assessment Practices Martha L. Thurlow National Center on Educational Outcomes [email protected] www.nceo.info National Center on Educational Outcomes (NCEO)

Slide 1
Snapshots:
What We Know about State
Assessment Practices
Martha L. Thurlow
National Center on Educational Outcomes
[email protected]
www.nceo.info
National Center on Educational Outcomes (NCEO)
Slide 2
Assessment Options
• General Assessments
• General Assessments with Accommodations
• Alternate Assessments based on GradeLevel Achievement Standards
• Alternate Assessments based on Alternate
Achievement Standards
• Alternate Assessments based on Modified
Achievement Standards
National Center on Educational Outcomes (NCEO)
Slide 3
General Assessments
•
•
•
•
Participation
Performance
Assessment Development
Narrowing the Achievement Gap
National Center on Educational Outcomes (NCEO)
Slide 4
Participation
• Public reporting of participation provides
one view – most often NCLB view
(students count only if in school for one
year)
• Reporting to the Secretary of Education,
as required by law gives another view –
based on total population of students with
IEPs
National Center on Educational Outcomes (NCEO)
Reading Assessment Participation Rates in Elementary School:
Percent Participation is of IEP Enrollment (Includes Regular and Alternate Assessment)
98
100
100
2005-2006
99
100
100
99
99
md
99
100
md
99
99
100
98
99
99
95
99
100
92
98
99
md
99
99
99
100
98
99
99
107
99
99
100
Slide 5
99
100
100
100
98
99
99
100
100
100
99
94
AS
95
BIE
100
CNMI
95
DC
100
98
98
Key
md
FSM
96
GU
> 105% (n=1 regular states and 0 unique states)
100
Palau
95% - 105% (n=45 regular states and 7 unique states)
99
< 95% (n=1 regular states and 2 unique states)
md =missing data (n=3 regular state and 1
unique state)
100
84
PR
RMI
VI
National Center on Educational Outcomes (NCEO)
From
NCEO
analysis of
APR data –
Thurlow et
al., 2008
Reading Assessment Participation Rates in High School:
Percent Participation is of IEP Enrollment (Includes Regular and Alternate Assessment)
91
100
98
2005-2006
98
97
89
97
98
93
96
md
96
97
md
99
84
98
95
86
94
97
91
98
97
96
96
90
98
84
95
99
96
98
96
98
93
Slide 6
97
100
96
100
92
100
97
98
97
97
100
100
AS
98
62
BIE
75
CNMI
72
DC
46
FSM
84
GU
100
Palau
89
87
Key
> 105% (n=0 regular states and 0 unique states)
95% - 105% (n=35 regular states and 3 unique states)
96
PR
< 95% (n=13 regular states and 6 unique states)
md
RMI
md =missing data (n=2 regular states and 1
80
VI
unique state)
National Center on Educational Outcomes (NCEO)
From
NCEO
analysis of
APR data –
Thurlow et
al., 2008
Slide 7
Participation Reporting Practices Changing
Situations Under Which Students are Counted
as Nonparticipants (# of States):
2007
Did not take assessment
Did not complete enough items to score
Accommodations produced invalid results
Tested at a lower grade than enrollment
47
17
18
16
2005
35
6
8
2
Based on data from the 2007 NCEO Survey of States
National Center on Educational Outcomes (NCEO)
Performance
Slide 8
• Public reporting of performance provides
one view – most often NCLB view
(students count only if in school for one
year)
• Reporting to the Secretary of Education,
as required by law gives another view –
based on total population of students with
IEPs
National Center on Educational Outcomes (NCEO)
Reading Assessment Proficiency Rates in Elementary School: Percent Proficient of
IEP Enrollment (Regular and Alternate Assessment)
46
61●
45
2005-2006
19
70
33*
62
51
65●
53●
48●
31
md
60
50
32
md
37
69
26
65
44
35
51●
47
21
54
70●
23
27
23●
27
50
50
51
49
Slide 9
29
21
55
65
21
20
53●
58●
31
45
58
59
6
AS
17
BIE
30
CNMI
17
DC
38●
14
Key – Percent Proficient
* State had >105%
participation on
assessments
 State had >1%
proficient on
alternate
assessment based
on alternate
achievement
standards
md
FSM
2
GU
40% – 49% (n=7 regular states and 0 unique states)
0
Palau
30% – 39% (n=7 regular states and 2 unique states)
33
20% – 29% (n=10 regular states and 0 unique states)
0
RMI
16
VI
50% – 100% (n=22 regular states and 0 unique states)
10% – 19% (n=2 regular states and 3 unique states)
0% –9% (n=0 regular states and 4 unique states)
md = missing data (2 regular states and 1 unique state
PR
National Center on Educational Outcomes (NCEO)
From
NCEO
analysis of
APR data –
Thurlow et
al., 2008
Reading Assessment Proficiency Rates in High School: Percent Proficient of IEP
Enrollment (Regular and Alternate Assessment)
39
44●
29
2005-2006
15
11
45
20
42
17
32
55●
19
40
md
33
26*
44
32
58
31
50
24
20
56●
9
25
19
17
8
38
24●
15
81
9
32
24
45
68●
14
20
40
26
12
39
Slide 10
72●
21
16
AS
13
BIE
19
CNMI
7
DC
16
FSM
2
GU
40% – 49% (n=7 regular states and 0 unique states)
0
Palau
30% – 39% (n=9 regular states and 0 unique states)
19
57●
30●
16●
5
* State had >105%
participation on
assessments
 State had >1%
proficient on
alternate
assessment based
on alternate
achievement
standards
Key – Percent Proficient
50% – 100% (n=8 regular states and 0 unique states)
20% – 29% (n=11 regular states and 0 unique states)
10% – 19% (n=10 regular states and 5 unique states)
0% –9% (n=4 regular states and 4 unique states)
md
8
PR
RMI
VI
md =missing data (n=1 regular state and 1 unique state
National Center on Educational Outcomes (NCEO)
From
NCEO
analysis of
APR data –
Thurlow et
al., 2008
Slide 11
Assessment Development
How States Addressed Universal Design in
General Assessment Development Process
(# of States):
Included in RFP for test development
Expert review
Think aloud methods as part of field test
Statistical analyses
“UDA” NOT addressed in test development
39
27
2
10
1
Based on data from the 2007 NCEO Survey of States
National Center on Educational Outcomes (NCEO)
Slide 12
Assessment Development
Bias/Sensitivity Reviews – Representation of
Disability Perspective
Content Review
Bias Review
Disability
Representation
31
33
No Disability
Representation
3
2
Based on data from the 2007 NCEO Survey of States
National Center on Educational Outcomes (NCEO)
Slide 13
Narrowing the Achievement Gap
Perceived Factors Related to Positive Achievement
Based on data from the 2007 NCEO Survey of States
National Center on Educational Outcomes (NCEO)
Slide 14
Positive Trends in Performance
Data Availability for Positive Trend Factors
Based on data from the 2007 NCEO Survey of States
National Center on Educational Outcomes (NCEO)
Slide 15
General Assessments with
Accommodations
• Numbers Using Accommodations
• Accommodations Policies
• Hints & Tips for States
National Center on Educational Outcomes (NCEO)
Slide 16
Accommodations
• States are required to report on the
number of students using
accommodations during the general
assessment
• Not all states yet report these data; some
report data for different types of
accommodations than others
National Center on Educational Outcomes (NCEO)
Reading Assessment Accommodation Rates in Elementary School: Percentage of
Students with IEPs Taking the Regular Reading Assessment with Accommodations
32
54
63
2005-2006
40
8
72
47
55
md
52
22
40
52
49
63
md
71
42
41
39
53
47
52
36
59
69
61
11
md
77
79
61
80
24
71
67
Slide 17
52
35
33
66
57
47
45
20
69
62
1
94
AS
59
BIE
73
CNMI
71
DC
73
64
65
Key
md
≥ 75% (n=3 regular states and 1 unique states)
50% - 74% (n=25 regular states and 5 unique states)
26% - 49% (n=13 regular states and 1 unique states)
< 25% (n=6 regular states and 0 unique states)
md =missing data (n=3 regular states and
3 unique states)
67
md
59
md
43
FSM
GU
Palau
PR
RMI
VI
National Center on Educational Outcomes (NCEO)
From
NCEO
analysis of
APR data –
Thurlow et
al., 2008
Reading Assessment Accommodation Rates in High School: Percentage of Students
with IEPs Taking the Regular Reading Assessment with Accommodations
18
53
43
2005-2006
36
5
51
44
53
md
47
md
37
57
md
23
41
35
42
53
73
27
59
65
5
43
77
76
38
79
46
59
66
md
70
45
49
Slide 18
56
22
0
46
81
50
62
0
61
59
100
0
AS
59
22
BIE
42
CNMI
48
DC
md
FSM
61
GU
39
28
Key
≥ 75% (n=4 regular states and 1 unique state)
50% - 74% (n=17 regular states and 2 unique states)
26% - 49% (n=17 regular states and 3 unique states)
md
46
Palau
PR
< 25% (n=8 regular states and 1 unique state)
md
RMI
md =missing data (n=4 regular states and
3 unique states)
53
VI
National Center on Educational Outcomes (NCEO)
From
NCEO
analysis of
APR data –
Thurlow et
al., 2008
Slide 19
Accommodation Policies Change
States with Read Aloud Question Changes in
Policy – comparing 2003 to 2005 and 2007
Number of States
No changes since 2003
One change since 2003
Two changes since 2003
19
19
12
From NCEO Data Viewer at http://data.nceo.info
National Center on Educational Outcomes (NCEO)
Slide 20
Alternate Assessments based on
Grade-Level Achievement
Standards (AA-GLAS)
• Number of States Providing This Option
• Issue – A Misunderstood Option?
National Center on Educational Outcomes (NCEO)
Percentage of Students with IEPs Participating
in AA-GLAS
Reading
Math
High
Elementary
Middle
School
Slide 21
High
Elementary
Middle
School
Kansas
18%
21%
19%
15%
22%
19%
Louisiana
5%
10%
5%
5%
10%
5%
Massachusettsa
0%a
0%a
0%a
0%a
0%a
1%
Minnesota
8%
7%
6%
7%
7%
8%
Mississippia
0%a
0%a
0%a
0%a
0%a
0%
North Carolina
18%
16%
2%
15%
15%
2%
Texas
19%
20%
13%
22%
21%
14%
Virginia
10%
12%
0%a
8%
11%
0%a
Wisconsina
1%
0%a
0%a
1%
0%a
0%a
a Small
percentage (< 0.5) participated n the grade level(s) and content area(s) represented as 0%.
From NCEO analysis of APR data – Thurlow et al., 2008
National Center on Educational Outcomes (NCEO)
Slide 22
Issue: AA-GLAS Misunderstood
2005-06
2006-07*
Kansas
Louisiana
Massachusetts
Minnesota
Mississippi
North Carolina
Texas
Virginia
Wisconsin
Indiana
Massachusetts
Montana
North Carolina
Virginia
Washington
West Virginia
National Center on Educational Outcomes (NCEO)
Slide 23
Alternate Assessments based on
Alternate Achievement Standards
• Participation
• Performance
National Center on Educational Outcomes (NCEO)
Reading Assessment Based on Alternate Achievement Standards Participation Rates
in Elementary School: Percent Participation is of IEP Enrollment (both Alternate and
Out-of-level)
Slide 24
12
20
6
2005-2006
3
34
8
6
21
7
12
23
6
5
md
6
6
6
md
5
9
8
7
10
5
6
5
6
9
8
7
3
8
6
5
6
4
6
5
4
8
8
18
12
7
6
6
0
AS
4
BIE
22
CNMI
7
DC
md
FSM
8
GU
>9% (n=10 regular states and 2 unique states)
18
Palau
0% − 9% (n=38 regular states and 7 unique states)
4
50
4
6
4
Key
md =missing data (n=2 regular states and
1 unique state)
PR
0
RMI
6
VI
National Center on Educational Outcomes (NCEO)
From
NCEO
analysis of
APR data –
Thurlow et
al., 2008
Reading Assessment Based on Alternate Achievement Standards Participation Rates in
High School: Percent Participation is of IEP Enrollment (both Alternate and Out-oflevel)
31
19
8
2005-2006
4
8
5
11
22
9
8
22
7
8
8
10
6
4
9
5
Slide 25
6
8
7
23
8
7
7
6
0
3
6
5
11
8
11
6
9
6
7
20
4
10
5
0
10
15
11
0
AS
3
BIE
6
CNMI
4
DC
0
FSM
2
GU
>9% (n=17 regular states and 0 unique states)
0
Palau
0% − 9% (n=33 regular states and 9 unique states)
9
47
24
13
2
Key
md =missing data (n=0 regular states and
1 unique state)
PR
md
RMI
9
VI
National Center on Educational Outcomes (NCEO)
From
NCEO
analysis of
APR data –
Thurlow et
al., 2008
Slide 26
States Exceeding the 1% Cap
Delaware*
Florida
Georgia
Louisiana*
Massachusetts**
Michigan
Mississippi
North Dakota
Ohio
Oregon**
Texas
Virginia
Wisconsin**
* High School Only
** Elementary School Only
From NCEO analysis of APR data – Thurlow et al., 2008
National Center on Educational Outcomes (NCEO)
Slide 27
Alternate Assessments based on
Modified Achievement Standards
• Number of States Providing This Option
• Criteria for Participation
• Nature of Assessments
National
Outcomes
(NCEO)
NationalCenter
CenterononEducational
Educational
Outcomes
(NCEO)
Number of States with AA-MAS
Slide 28
July 2007 – 5 states indicated that they had an AA-MAS
(all were developed prior to the April, 2007 regulation:
• Kansas
• Louisiana
• North Carolina
• North Dakota
• Oklahoma
In addition, Maryland provided information on its Web
site about the development a AA-MAS.
From NCEO Synthesis Report 67 – Lazarus et al., 2007
National Center on Educational Outcomes (NCEO)
Slide 29
States Considering AA-MAS
Based on data from the 2007 NCEO Survey of States
National Center on Educational Outcomes (NCEO)
Slide 30
Criteria for Participation
Eligibility Criteria in 6 States with AA-MAS*
Criteria
No. of States
Student has IEP
6
Multiple years behind grade level expectations
4
Not based on categorical label
4
Does not have significant cognitive disabilities
Not due to excessive absences or to social, cultural,
environmental, or economic factors
4
Learning grade level content
3
Not based on placement setting
2
Previously scored at unsatisfactory level
2
4
* Includes Maryland
From NCEO Synthesis Report 67 – Lazarus et al., 2007
National Center on Educational Outcomes (NCEO)
NCEO Survey Showed States Identifying Many
Categories As Most Likely to Participate in AA-MAS
Slide 31
Based on data from the 2007 NCEO Survey of States
National Center on Educational Outcomes (NCEO)
Slide 32
Nature of Assessments
Type of Assessment/Questions in 6 States with AA-MAS
State
Portfolio
Multiple
Constructed
Writing
Choice
Response
Prompt
KS
LA
MD
NC
ND
OK
Total
X
X
X
X
X
X
1
X
2
X
1
X
5
From NCEO Synthesis Report 67 – Lazarus et al., 2007
National Center on Educational Outcomes (NCEO)
Slide 33
Nature of Changes to Items in 6 States with AA-MAS
Change
No. of States
Distractor Removed
4
Fewer Items
4
Simplified Language
4
Shorter Passages
3
Segmentation of Passages
1
From NCEO Synthesis Report 67 – Lazarus et al., 2007
National Center on Educational Outcomes (NCEO)
Slide 34
Accommodations Incorporated into Design in 6 States with
AA-MAS
Accommodation
No. of States
Fewer items/page
3
Larger font size
3
Calculator
2
Breaks as needed
2
Key text underlined/bolded
1
From NCEO Synthesis Report 67 – Lazarus et al., 2007
National Center on Educational Outcomes (NCEO)
Slide 35
Characteristics of Assessment in 40 Planning to Develop and
AA-MAS
Characteristic
Number of States
Include only multiple choice questions
Reduce total # of items or time to complete
Simplify vocabulary
Use non-traditional items or formats
Use shorter or fewer reading passages
Use fewer items or fewer answer choices
11
25
26
16
29
23
Based on data from the 2007 NCEO Survey of States
National Center on Educational Outcomes (NCEO)
Slide 36
Stay Tuned for More Snapshots
Visit NCEO’s Web site for more and
continuing information on
assessment options:
www.nceo.info
National Center on Educational Outcomes (NCEO)