Transcript Jason Altman – NCEO Mari Quenemoen – NAAC TASH Annual Conference – Nov.

Jason Altman – NCEO
Mari Quenemoen – NAAC
TASH Annual Conference – Nov. 19, 2009
1
Introduction
• It is important that state large-scale
assessment systems be both inclusive of and
fair to all populations including students with
disabilities
• A 2009 survey explored these issues as a
whole and paid special attention to state
practices in assessing students who may
participate in an alternate assessment option
2
Survey
• All 50 states returned the 12th survey of state
directors of special education and state directors
of assessment by the National Center on
Educational Outcomes (NCEO)
• The survey was disseminated to respondents
electronically in the winter of 2009
• NCEO returned surveys for verification and, upon
receipt, entered, reviewed, and analyzed the data
• Results of the eleventh survey were presented at
the 2008 TASH annual conference
3
Presentation Purpose
• The purpose of this presentation is to provide
a snapshot of new initiatives, trends,
accomplishments, and emerging issues in the
large scale assessment of students with
disabilities
• Presenters will pay special attention to issues
surrounding the testing of students with
disabilities who may be assessed using an
alternate assessment
4
Apples to Oranges or
Apples to Apples
General Issues
• States reported that assessment validity and test
design/content were areas of much success
• Issues related to English language learners with
disabilities and the performance of urban schools
were often selected as challenging
• States appear to have a very mixed viewpoint on
the AA-MAS (7 respondents in each category)
• On the other hand, respondents appear to be
strongly entrenched on the successful side of the
ledger as related to reporting and monitoring
6
Regular Assessment Issues
Accommodations
• More than four in five states reported monitoring
accommodations use in 2009
0
Number of States
10
15
5
Directly observing test administrations
Sending teams into schools on a scheduled basis
14
1
13
0
Sending teams into schools on a random basis
10
3
Sending teams into schools on a targeted basis
0
Completing online record reviews
0
10
6
Other
1
We do not monitor accommodations use
1
Regular States
15
2
Conducting desk audits
25
22
7
Interviewing students, teachers, and administrators
20
10
9
Unique States
7
Regular Assessment Issues
Accommodations
• Most respondents also reported that
accommodations cause some difficulty on test day in
schools and districts within their state (n = 41)
0
2
4
Arranging for trained readers/interpreters
6
14
16
18
17
4
Scheduling substitute providers of accommodations
3
Training proctors in providing accommodations
3
Ensuring the ordering of special test editions
3
Ensuring that accommodations are recorded
20
19
3
Ensuring proctors give correct accommodations
14
12
11
6
2
Arranging for and checking equipment
4
5
1
1
Ensuring accommodations for makeup assessments
Other
We have ensured accommodations are carried out
Number of States
8
10
12
3
9
0
Regular States
9
Unique States
8
Alternate Assessments Based on Modified
Academic Achievement Standards
• In April 2007, new No Child Left Behind regulations on AAMAS were finalized
• States have the option of developing alternate assessments
based on modified achievement standards (AA-MAS)
• At the time of the 2007 survey of states we found that five
states already had an assessment in place, 32 states were
considering modifying an existing grade-level assessment
and another twenty-five were considering developing a
new assessment
• In the past two years, some states have further refined
their motivations for moving in this assessment direction,
while other states were instead making efforts to improve
the assessments that they already offer
9
Alternate Assessments Based on Modified
Academic Achievement Standards
• Since 2007, 14 states have decided not to develop of AA-MAS, while 12
states were still in the process of making their decision
• Of the 24 that have developed, or were developing, an AA-MAS, nine have
already given the assessment and analyzed the data, one for the first time
in 2008-09
We have given the
assessment, and analyzed
data
8
14
1
We will give this assessment
for the first time in 2008-09
We are currently in the
process of developing an
AA-MAS
15
12
We are currently
researching the need for an
AA-MAS
We have decided not to
develop at this time
10
Alternate Assessments Based on Modified
Academic Achievement Standards
• States that were developing or have developed their AA-MAS
were three times more likely to modify an existing grade-level
test rather than design an entirely new test
2
5
Design an entirely
new test
Modify an existing
grade-level test
15
Other, please explain
below
Researching the Need
or Not Developing
26
No Response
2
11
Alternate Assessments Based on Modified
Academic Achievement Standards
• Respondents have been asked on the past two surveys about the changes
they envisioned making when modifying existing tests
• Findings were similar to those found in 2007 save for a less frequent
response to use of non-traditional items (keeping in mind that many more
states have decided not to develop the AA-MAS at this time
0
5
Number of States
15
20
10
16
Simplify vocabulary
12
Use fewer answer choices
6
Include only multiple choice questions
3
Use non-traditional items or formats
2007
35
26
13
Use shortened or fewer reading passages
30
26
14
Reduce the number of total test items or time
2009
25
29
23
11
14
12
Alternate Assessments Based on Modified
Academic Achievement Standards
• States reported a variety of strategies and
methods in determining content targets and
blue prints for their AA-MAS
0
5
10
Number of States
15
Same specifications as for regular assessment
7
Special education curriculum literature
6
Teacher survey
3
Test company provided
3
1
2
Researching the Need or Not Developing
No Response
30
11
Review of literature on teaching at-risk learners
Other, please explain below
25
17
Stakeholder panels
Consultant developed
20
26
2
13
Alternate Assessments Based on Alternate
Achievement Standards
• Seven states (HI, ID, MS, NE,
NH, NV, and UT) and three
unique states (Guam, PR, and
RMI) are currently in the
process of revising their AAAAS
• Trend analysis from 2005
and 2003 data
14
AA-AAS Topics
•
•
•
•
•
•
Test formats
Content alignment
Scoring methods
Rubric analysis
Methods for determining achievement levels
Scorers
15
AA-AAS Formats
•40% of states use a
portfolio or body of
evidence (40% of these
are standardized)
•36% use a standardized
set of performance
events/tasks (44% of
which require the
submission of evidence)
•16% use a multiple
choice test
16
AA-AAS Content Alignment
• Complete transition to
alignment with academics
• Extended/expanded
academic content standards
(27 states) or grade level
academic content standards
(16)
• No states align AA-AAS to
functional skills, and IEP
teams no longer determine
the content of the test for
individual students
17
AA-AAS Scoring Procedures
0%
10%
Percentage of States
30%
40%
50%
60%
20%
70%
80%
90%
80%
Rubric
74%
66%
16%
Points Assigned on a Rating Scale
32%
16%
10%
Number of Items Correct
24%
38%
4%
Reading Rate or Accuracy
8%
4%
2003
2005
2009
18
Outcomes Measured by Rubrics
• Primarily skill/competence, level of
assistance, and alignment to academic
content
• Trend away from scoring other factors
19
Outcomes Measured by Rubrics (2005 and 2009)
0
5
2005
2009
Number of States
10
15
20
25
30
Level of Assistance
Skill/Competence
Alignment with Academic Content Standards
Ability to Generalize
Number/Variety of Settings
Degree of Progress
Appropriateness
Participation in General Education Settings
Support
Staff Support
Self Determination
Social Relationships
Parent Satisfaction
20
Outcomes Measured by Rubrics ( States and
Unique States 2009)
0
5
Level of Assistance
10
2
10
1
Degree of Progress
2
Appropriateness
2
6
8
5
Participation in General Education Settings
Support
6
1
5
Staff Support
4
Self Determination
3
Social Relationships
Parent Satisfaction
15
4
Ability to Generalize
30
25
5
Alignment with Academic Content Standards
25
24
3
Skill/Competence
Number/Variety of Settings
Number of States or Unique States
10
15
20
1
States
Unique States
21
Methods for Determining
Achievement Levels
0%
10%
20%
70%
8%
37%
4%
58%
Reasoned judgment
17%
23%
Bookmarking
Other
60%
20%
Body of work
Modified Angoff
50%
12%
Contrasting groups
Judgmental policy capturing
Percentage of States
30%
40%
29%
12%
10%
2005
2009
22
Who Scores the AA-AAS?
0
5
Number of States or Unique States
10
15
25
21
test company contractor
student's special education teacher
1
teachers from other districts
1
state education agency
1
teachers in the same district as student
1
members of student's IEP team
1
12
10
4
4
4
3
university contractor
other
20
1
currently in revision
3
3
States
7
Unique States
23
Thank you! For more information…
National Alternate Assessment
Center
University of Kentucky
859-257-7672
http://www.naacpartners.org
National Center on Educational
Outcomes
University of Minnesota
612-626-1530
http://www.nceo.info
Mari Quenemoen:
[email protected]
Jason Altman:
[email protected]
24