Transcript Research on Universal Design of Assessments Making Assessments Accessible

Research on Universal
Design of Assessments
Making Assessments Accessible
and Valid for All Students
National Center on Educational Outcomes
Universal Design of Assessment:
The Whole Elephant
David Malouf
Office of Special Education
Programs
U.S. Department of Education
National Center on Educational Outcomes
Universal Design of Assessments
Designing assessments to be
inherently accessible and valid
for the widest possible range
of students.
National Center on Educational Outcomes
Series of Federal Laws
• Improving America’s Schools Act of
1994
• 1997 reauthorization of the Individuals
with Disabilities Education Act
• No Child Left Behind Act of 2001
• Upcoming reauthorization of the
Individuals with Disabilities Education
Act
National Center on Educational Outcomes
Improving America’s Schools Act
of 1994
• Under Title I, all students were to be
included in assessments, public
reporting, and accountability.
National Center on Educational Outcomes
Individuals with Disabilities
Education Act (1997 reauth.)
• Added requirements for students with
disabilities to be included in State and
district-wide assessments.
• Introduced alternate assessments
• Required public reporting of aggregated
and disaggregated data
National Center on Educational Outcomes
No Child Left Behind Act of 2001
• Increased assessment demands
• Increased requirements for including
students with disabilities and students
with limited English proficiency
• Accountability for subgroups
National Center on Educational Outcomes
From the Regulations for the
No Child Left Behind Act:
§200.2 State responsibilities for
assessment
(b) The assessment system required under
this section must meet the following
requirements:
.......
(2) Be designed to be valid and accessible
for use by the widest possible range of
students, including students with
disabilities and students with limited
English proficiency.
National Center on Educational Outcomes
President’s Commission on
Excellence in Special Education
• Created in 2001 to recommend reforms in
America’s special education system
• 13 Public hearings and meetings, plus written
input
• Reported in July 2002
National Center on Educational Outcomes
President’s Commission on
Excellence in Special Education
RECOMMENDATION—INCORPORATE
UNIVERSAL DESIGN IN ACCOUNTABILITY
TOOLS: Ensure all tools used to
assess students for accountability
and the assessment of progress are
designed to include any
accommodations and modifications
for students with disabilities.
National Center on Educational Outcomes
Upcoming reauthorization of the
Individuals with Disabilities Education
Act (IDEA)
National Center on Educational Outcomes
Senate Bill 1248
Universal design. The State
educational agency (or, in the case
of a districtwide assessment, the
local educational agency) shall, to
the extent possible, use universal
design principles in developing and
administering any assessments under
this paragraph.
§612(a)(16)(A)(ii)(E)
National Center on Educational Outcomes
The pressure is on...
Will we do quick retrofits
of tests and called them
“universally designed”?
Will we simply port tests
onto computers and call
them “universally
designed”?
National Center on Educational Outcomes
Inclusive Population
Constructs designed for accessibility
Accessible items
Amenable to
accommodations
Simple and clear
instructions and
procedures
Maximum readability
Maximum legibility
National Center on Educational Outcomes
Three Research Projects on
Universal Design of Assessment
• “Development Techniques for Universally
Designed Assessments”
– Sandra Thompson, University of Minnesota
• “Access to Assessment via Technology”
– Jacqueline Kearns, University of Kentucky
– Bob Dolan, CAST
• “Project MAP (Making Accommodations
Personalized”
– Gerald Tindal, University of Oregon
National Center on Educational Outcomes
Development
Techniques for
Universally Designed
Assessments
National Center on Educational
Outcomes
University of Minnesota
Sandy Thompson
http://education.umn.edu/nceo
National Center on Educational Outcomes
3 Studies
• Use “think aloud” to examine
student – end user perspectives
• Analyze items for differences
possibly due to design features
• Design training for item/test
reviewers
National Center on Educational Outcomes
“Think aloud”
• Recently interviewed 90
students using think aloud
protocol
• 4th and 8th grade
• Used multiple choice and
constructed response items
from state math test
National Center on Educational Outcomes
Logistics
•
•
•
•
•
10 researchers
5 days
Worked in pairs
All sessions videotaped
Primary accommodations included
oral administration and sign
language interpretation
National Center on Educational Outcomes
Student Characteristics
Grade 4 Grade 8
Learning Disability
10
10
Deaf/Hard of Hearing
10
10
Mild Cognitive Impairment
5
5
English Language Learner
10
10
No Disability
10
10
National Center on Educational Outcomes
Overall Observations
• Students who were confident of content
did not have problems with design
• Students who had no idea how to solve
the problem did not have problems with
design
• Students “in the middle” – not sure of
content, some reading difficulty, design
made a difference
National Center on Educational Outcomes
Examples of Student Perceptions
• Many students didn’t see one of the cities on
a map
• The name of one of the cities was
“Independence” - uncommon meaning
• Box between top and bottom of item – some
students did not read entire item
• Sign for parallel gave away the answer
• Some students read fraction 3 5/8 as “35
divided by 8”
• Students unfamiliar with settings – “Glee club
does number,” “fitness club”
National Center on Educational Outcomes
Other Observations
• Some students got a lot of “help” from sign
language interpreters and teachers who
wanted to make sure they understood the
problem – what happens on test day?
• Some student forms reported need for oral
administration when they could clearly read
the items independently – are some
accommodations inappropriate?
• For English language learners - is oral
administration in English an appropriate
accommodation on a Math test?
• Should sign language interpreters have a
script to follow so they don’t “give away”
some answers?
National Center on Educational Outcomes
Considerations for Item
Review
• Overall appearance is clean and
organized
• Clear format for text
• Clear format for pictures and graphics
(when essential to item)
• Concise and readable text
• Format allows for changes without
changing meaning or difficulty
• Meets criteria for measuring what it is
intended to measure
National Center on Educational Outcomes
Considerations for Test
Review
• Meets general criteria for measuring
what it is intended to measure
• Overall appearance is clean and
organized
• Instructions are necessary, clear, and
understandable
• Scoring criteria are appropriate
• Others?
National Center on Educational Outcomes
Alignment and Usability Need to
be Considered Together
Alignment
Usability
National Center on Educational Outcomes
Future Plans
• Interview students with vision impairments
– Partner with American Printing House for the Blind
• Explore differential item analyses
– Partner with researchers from CTB/McGraw-Hill
• Produce short awareness video
– Partner with Institute on Community Integration at
University of Minnesota
• Design and pilot training for item reviewers
and item developers
– Partner with Missouri Department of Education
National Center on Educational Outcomes
Universal Design of Assessment:
Applications of Technology
CCSSO National Conference on
Large-Scale Assessment
June 23, 2003
National Center on Educational Outcomes
Universal Design for Learning (1)
• Universal Design for Learning (UDL) extends
the concept of universal design from a
physical space to a cognitive space
• Based upon psychological and neuroscientific
theories of learning
• Relies on scaffolds, supports, &
accommodations which support students’
challenges and thus provide access to
learning
• Applies to design and development of goals,
methods, assessments, and materials
National Center on Educational Outcomes
Universal Design for Learning (2)
• To support individual differences,
students must be provided with multiple
means of interacting with curriculum
– Multiple means of representation
– Multiple means of interaction and
expression
– Multiple means of engagement
• Digital technology is neither necessary
nor sufficient for UDL, but it is an
enabling factor
National Center on Educational Outcomes
Kentucky
• Universal Design for Learning Expert Group
• Instructional Technology for Student Success
(ITSS) initiative
• Digital Text Network
• E-Text Schools program
• Senate Bill 243
– Requires any textbook offered for adoption in
Kentucky schools be made available by the
publisher in an accessible digital format
National Center on Educational Outcomes
KY CATS Online Assessment
• Web-based, individualized assessment for
qualifying students with disabilities:
– Students with IEP or 504 Plan that specifies need for
"reader" as an instructional and assessment
accommodation;
– Students who require and routinely use text-reader or
screen-reader technologies to access printed material in
classroom instruction and assessment;
– Students who have accessed and used the CATS
Online Practice Area.
• Based upon success of pilot studies, 16 districts,
31 Schools, & 204 students participated in “live”
CATS Online this spring
National Center on Educational Outcomes
CATS Online
National Center on Educational Outcomes
Maine
• Maine Learning Results emphasize inclusion
of diverse students by separating goals and
methods
• Maine Comprehensive Assessment System
uses variety of state and local components,
allowing flexibility and multiple measures of
learning to accommodate learner diversity
• State assessment (MEA) tightly aligned with
Learning Results
• Actively pursuing UDL as a means for
increasing access to general curriculum
National Center on Educational Outcomes
Overarching Research Question
How can digital technologies be used to
improve accessibility of large-scale
assessments in accordance with the
principles of Universal Design for
Learning?
National Center on Educational Outcomes
Research Question 1
What are the practical, procedural, and
political design and implementation
features of accessible computer-based
assessments?
National Center on Educational Outcomes
Research Question 2
What technology pre-requisite skills do
students need to use computer-based
assessments?
National Center on Educational Outcomes
Research Question 3
What is the impact of a computer-based
universally designed assessment on the
scores of students with disabilities?
National Center on Educational Outcomes
Research Question 4
To what extent does accessible
curriculum design impact student
assessment results?
National Center on Educational Outcomes
Research Question 5
Can the universally designed assessment
design features be replicated with new
content in a different state context?
National Center on Educational Outcomes
Research Methodology
• Qualitative Studies
– Interviews, observations, focus groups
• Quantitative Studies
– Survey, score analysis
• Critical Theory
– Constituent reflection
National Center on Educational Outcomes
Preliminary Findings To Date
• Interviewed approximately 40 students
from 7 schools who took CATS Online
• Most students used the ‘text-to-speech’
feature
• Students liked the independence that
the text reader provided, which allowed
them to re-read questions multiple times
National Center on Educational Outcomes
Features that Promote
Accessibility
• Multiple means of representation
–
–
–
–
Text-to-speech
Font features (e.g. size)
Screen layout options
Textual description of images
• Multiple means of expression and
interaction
– Word processing
– Flexible navigation
National Center on Educational Outcomes
Features that Impede
Accessibility
• Word processing – requires a set of
skills that some students may not have
• Text-to-speech – voice quality and
accuracy issues
• Layout issues – poetry, tables, etc.
National Center on Educational Outcomes
Preliminary Item Analysis
Positive Exemplars
National Center on Educational Outcomes
MEA Grade 11
Science and Technology
16. This diagram shows that the beams of light from two flashlights can pass
through each other and then continue on unaffected. This observation
illustrates which property or properties of light?
A. only particle
 B. only wave
C. both particle and wave
D. neither particle nor wave
Learning Results: H-1, Energy. Students will understand concepts of energy.
Students will be able to analyze the evidence that leads scientists to conclude that
light behaves somewhat like a wave and somewhat like a particle.
Consider
– Explicit reference to image
– Redundancy of information between image and text
National Center on Educational Outcomes
Scales of Justice
Use the illustration below to answer question 12.
CATS Grade 8 Social Studies
12. “Lady Justice” is a symbol of the United States justice system. As shown
in the illustration above, she wears a blindfold and carries a balance scale.
Explain how this symbol represents two characteristics of the U.S. justice
system.
Consider
– Explicit reference to image
– Redundancy of information between image and text
– Multiple opportunities for response (writing vs.
keyboarding)
National Center on Educational Outcomes
Preliminary Item Analysis
Negative Exemplars
National Center on Educational Outcomes
MEA Grade 11 Math
2. Which of.the following is the best estimate of the sum of
√26 + √78?
A. 10
 B. 14
C. 52
D. 104
Learning Results: B-1, Computation. Students will understand and demonstrate
computation skills. Students will be able to use various techniques to approximate
solutions, determine the reasonableness of answers, and justify the results.
Consider
– Ability of text-to-speech tools to recognize and
represent the square-root symbol
– Potential invalidation of item when square-root
symbol is “decoded” by human or digital reader
National Center on Educational Outcomes
Alex’s
Garden
CATS Grade 8 Math
30. Alex is watering part of his garden with a sprinkler that covers a circular area
shown in the diagram below.
a. What is the total area of Alex’s garden? Show your work.
b. What is the area of the part of the garden that is being watered by the
sprinkler? Express your answer to the nearest square foot. Show your work.
c. What percent of his garden is being watered by the sprinkler? Express your
answer to the nearest percent. Show your work.
BE SURE TO LABEL YOUR RESPONSES (a), (b), and (c).
Consider
– Transferring of stimulus to paper for response
– Use of text-to-speech with image labels
Nationalof
Center
on Educational
Outcomes
– Representation
image
to blind
students
Concerns & Conclusions
• Applying universal design to instruction
and assessment: which comes first?
(Chicken and egg problem)
• Practical considerations of use of
technology e.g. security, scalability, cost
• Need for seamless integration of
instructional and assessment
technologies to ensure valid, accurate,
and fair testing
National Center on Educational Outcomes