Transcript Designing More Accessible Achievement Tests for All Students

Designing More Accessible
Achievement Tests for All Students
Stephen N. Elliott
Learning Sciences Institute
and
Department of Special Education
Vanderbilt University
CCSSO 2009
National Conference on Student Assessment
Projects & Partners
• CAAVES: Consortium for Alternate Assessment
Validity and Experimental Studies
– USDE funded; 2006-2009
– Partners: AZ, HI, ID, & IN + Vanderbilt Measurement Group +
Discovery Education Assessment
• CMAADI: Consortium for Modified Alternate
Assessment Development and Implementation
– USDE funded; 2007-2010
– Arizona Dept. of Education
– Indiana Dept. of Education
• Visit Websites for Resources Discussed Today
–
–
–
http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml
http://peabody.vanderbilt.edu/LSI_Projects/C-MAADI_Project_Home.xml
http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home/TAMI_Project.xml
CCSSO NSA Conference / Elliott
2009
2
Inclusive Testing & Better Results
• NCLB Act 2007
Amendments on AA-MAS
• Students with disabilities
who exhibit persistent
academic difficulties.
–
–
–
–
Inattention
Organizational difficulties
Poor reading fluency
History of below proficient
test performances
– Low self-efficacy with
testing
CCSSO NSA Conference / Elliott
2009
3
Key Terms
• Access: the opportunity for test-takers to demonstrate proficiency on
the target construct of a test or a test item. In essence, complete access
is manifest when a test-taker is able show the degree to which he/she
knows the tested content. Access, therefore, must be understood as an
interaction between individual test-taker characteristics and features of
the test itself.
• Accommodation: widely recognized in state testing guidelines as
individualized changes to the setting, scheduling, presentation format,
or response format of an assessment.
• Modification: alterations or adjustments of test items to facilitate
access for virtually all test takers. Appropriate modifications ….
–
–
–
–
Remove extraneous material,
Maintain the same depth of knowledge (DOK),
Do NOT change the grade-level construct being measured, and
Increase the validity of the inference from the test score.
CCSSO NSA Conference / Elliott
2009
4
Anatomy of an Item
CCSSO NSA Conference / Elliott
2009
5
CAAVES Procedures
We completed the following…..
•
Modified a common set of existing reading and math items to
create items designed to be more accessible and still measure the
same grade-level content as the original items.
•
Conducted a cognitive lab study with a small sample of students
with and without disabilities to gain their insights into which item
modifications are preferred and most likely to improve test access
for students whose disability involves reading difficulties.
•
Conducted a cross-state experimental study to compare the effects
of tests with and without modified items on students’ test
performances and test score comparability.
•
Conducted post-assessment survey of all students concerning their
perceptions of item types and cognitive ease.
CCSSO NSA Conference / Elliott
2009
6
Guiding Theories & Research
• Evidenced-based model of test score
validity,
• Universal design principles,
• Cognitive Load Theory for designing
instructional materials, and
• Item writing research and practices.
CCSSO NSA Conference / Elliott
2009
7
Examples of Theory-Guided &
Data-Based Item Modifications
CCSSO NSA Conference / Elliott
2009
8
Example: Original to Modified Item
CCSSO NSA Conference / Elliott
2009
9
Overview of Results
Elliott, et al. (in press),
Exceptional Children
CCSSO NSA Conference / Elliott
2009
10
Modifications Benefited all Groups
Elliott, et al. (in press),
Exceptional Children
CCSSO NSA Conference / Elliott
2009
11
Item Summary Reports: An Example
CCSSO NSA Conference / Elliott
2009
12
12
Estimating
Impact
•Will AA-MASs result
in more students with
disabilities being
considered “proficient”
for AYP?
• We have explored the
impact of some
hypothetical cut scores
for the CAAVES
Reading and Math
Scores.
•An actual Standard
Setting is needed.
Elliott, et al. (in press),
Exceptional Children
CCSSO NSA Conference / Elliott
2009
13
Method for Documenting OTL
AZ Cog Lab
Study, 2008
CCSSO NSA Conference / Elliott
2009
14
Cognitive Labs
Excerpted from Kettler, Elliott, & Beddow,
in press Peabody Journal of Education
CCSSO NSA Conference / Elliott
2009
15
Post-Assessment Focus Groups
AZ CMAADI Pilot Study, 2009
CCSSO NSA Conference / Elliott
2009
16
Evolving Modification Paradigm
Step 1. Evaluate original item accessibility.
Step 2. Reduce sources of construct-irrelevant
variance in items.
Step 3. Document changes to items.
Step 4. Pilot test with student cognitive labs &
post-assessment focus groups.
Step 5. Field test with large sample of students.
Step 6. Conduct psychometric & related
analyses.
CCSSO NSA Conference / Elliott
2009
17
Characteristics of Appropriate Modifications
Desired Outcomes
Design Elements
• Simply words and text
structure
• Delete extraneous words
• Improve visuals and locate
within item
• Use bold text for
important words
• Eliminate least plausible
distractor so there are 3
answer choices
•
•
•
•
•
•
•
•
•
•
•
Increase accessibility
Decrease item difficulty
Increase item discrimination
Increase reliability estimates
Reduce readability level w/i grade
range
Maintain alignment w/ content stds.
Maintain DOK for all items
Increase validity of test scores
Reduce need for accommodations
Increase reading fluency
Improve students’ perceptions of tests
& motivation to engage in testing
CCSSO NSA Conference / Elliott
2009
18
Colleagues’ Presentations
• Quantifying and Improving Item & Test
Accessibility – Peter Beddow, Vanderbilt
• Using Students’ Insights to Influence Item &
Test Design – Andrew Roach, Georgia State
• Plausible Attractors & Item PsychometricsMichael Rodriguez, University of Minnesota
CCSSO NSA Conference / Elliott
2009
19
Thanks!
• Thank you very much for your time and
joining us for this session.
• Please provide follow-up questions and
suggestions in writing to:
[email protected]
http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml
http://peabody.vanderbilt.edu/LSI_Projects/C-MAADI_Project_Home.xml
CCSSO NSA Conference / Elliott
2009
20