Transcript Developing Accessible Tests with Universal Design and

Developing Accessible Tests
with Universal Design and
Digital Technologies
Robert Dolan
Tracey Hall
Restatement of the Problem:
How to Standardize Assessment of
Nonstandard (i.e. All) Individuals?
• The challenge
Ensuring adequate individualization of test delivery
without compromising comparability
• Proposed solutions:
– Application of universal design principles
– Use of digital technologies
Universal Design
Originated in architecture as a means for
ensuring environments, and later products
and communications, be designed from
the start to consider the needs of the
widest possible array of users
(Ron Mace, 1996)
Matthew
Guggenheim Museum
Universal Design
• Not one-size-fits-all, but rather
provides users with alternatives.
• Designed in from the beginning,
not added on later.
• Increases access opportunities for
everyone.
Universal Design for Learning
• Universal Design for Learning (UDL)
extends universal design from a
physical space to a pedagogical space.
• New approach for teaching, learning,
and assessment that draws on new
brain research and new media
technologies to respond to individual
learner differences.
Universal Design for Learning
• UDL curricula, teaching practices, and
policies are inherently flexible so as to
support individual learning differences.
• Reduces the demand on educators to
develop and implement modifications
and accommodations.
Universal Design for Learning
• Supports individual learner differences
through use of:
– Multiple means of representation
– Multiple means of expression
– Multiple means of engagement
• Relies on scaffolds and accommodations
to support students’ challenges and thus
improve access to learning.
Multiple Means of Representation
To give learners various ways of
acquiring information and knowledge
•
•
•
•
Provide multiple examples
Highlight critical features
Provide multiple media and formats
Support background context
Multiple Means of Expression
To provide learners alternatives for
demonstrating what they know
• Provide flexible models of skilled performance
• Provide opportunities to practice with
supports
• Provide ongoing, relevant feedback
• Offer flexible opportunities for demonstrating
skill
Multiple Means of Engagement
To tap into learners' interests, challenge them
appropriately, and motivate them to learn
•
•
•
•
Offer
Offer
Offer
Offer
choices of content and tools
adjustable levels of challenge
choices of rewards
choices of learning context
Role of Digital Technology
New media, such as digital text,
sound, images, and the Web,
can be adjusted for different
individuals and can reduce
construct-irrelevant barriers,
thus opening doors to learning.
http://www.cast.org/teachingeverystudent/mlk
- Folktales Example -
Universal Design for Learning
and Assessment
Universal Design for Learning
principles apply globally across
standards, instruction, materials,
and assessments.
UDL & Instructional / Assessment Cycle
Initial Assessment
• Archival Assessment
• Formal Standardized Measures
• Diagnostic Assessments
Formative
Evaluation
• Analyze Errors
• Monitor Progress
• Modify Instruction
UDL
Instructional Design
Multiple means of:
• Representation
• Expression
• Engagement
• Determine Content
• Instructional Strategies
• Materials Selection and Sequence
• Select examples
• Provide for cumulative review
Instructional Delivery
• Access
• Participation and Engagement
• Monitor performance and pace
• Immediate Feedback
Universal Design & Accommodations
• Accommodations critical to reduce constructirrelevant variance.
• Universal design requires that accommodations
be considered from the start to ensure
effectiveness.
• Ideally accommodations should be seamlessly
integrated across instruction and assessment.
The goal is to reduce accommodations as retrofits.
Universal Design and Assessment
Current Efforts
Two complementary efforts:
– Improving current state of testing through
application of original Mace principles.
– Improving future of testing by increasing
flexibility and individualization using
digital technologies.
UDA Directed Research
Compare and contrast two states,
Kentucky and Maine,
with respect to universal design
and technology-based testing.
Kentucky Online System
Technology Skills Checklist
• Basic computer skills
• Keyboarding skills
• Word processing skills
• Text-reader / screen-reader skills
• Online assessment skills
Fleming, Kearns, Dethloff, Lewis & Dolan JSET 2006
Research on Universal Design
and Assessment
• Limited research to date.
• Accommodations research in
general shows mixed results.
• Technology-based testing
research in general shows mixed
results.
Keyboarding vs. Handwriting
• Study by Boston College Lynch School
of Education (Michael Russell & Walt
Haney) and Wellesley, MA, Public
Schools (Tom Plati).
• Significant improvement in MCAS
scores for typically-achieving 4th, 8th,
and 10th graders’ when using
keyboarding for open-response
questions.
ation Effect
Scores
14.8
Keyboarding
vs.
Handwriting
13.3
Mode of Administration Effect
Mean Writing Scores
Paper
Computer
15.8
16
Grade 10
Total Score
14
14.4
12.8
13.9
14.8
13.3
12
10
Paper
8
Computer
6
4
2
0
Grade 4
Grade 8
Grade 10
Study of Universally Designed
Technology-Based Testing
• Pilot study with high-school
students with learning disabilities.
• Test of civics and U.S. history.
• Paper-and-pencil vs. computerbased testing with optional
text-to-speech read-aloud
(CBT-TTS).
Prototype CBT System
Effect of CBT-TTS on Test Scores
Dolan, Hall, et
al. JTLA 2005
Research Take Home Points
• Both technology and universal
design promising for reducing
construct-irrelevant variance.
• Importance of matching
instruction and assessment
techniques and technologies.
• Additional research needed!
Future of Universal Design,
Technology, and Assessment
• Providing students with choice
during testing.
• Developing richer, more authentic
assessment situations.
• Progress monitoring.
Providing Students with
Choice During Testing
What is the effect of providing students
choice of equivalent questions /stimuli?
(i.e. multiple means of representation)
Mei Ling gave the following description of a three-dimensional figure.
• The solid has 6 faces
• The solid has 8 vertices.
• The solid has 12 edges.
8. Which of the following figures matches Mei Ling’s description?
Click on any figure to see it spin in 3D.
Developing Richer, More
Authentic Assessments
Can universal design and
technology be used to develop
assessment items that reduce not
only physical and sensory barriers
but cognitive barriers as well?
Progress Monitoring
Can technology improve
formative assessments through
real-time feedback and individualization
of instruction?
Commercial Tools available
for Progress Monitoring
• AIMSWEB
• DIBELS
Progress Monitoring Built in a Supported Writing Environment
Technology Limitations & Concerns
• Availability of technology and the
infrastructure to support it.
• Need to retain traditional administration
modes for use under certain situations.
• Potential for inequities between students.
• Comparability issues.
• Students’ ability to exercise choice wisely.
Conclusion
• Technology can help realize the
flexibility necessary to accurately
and fairly assess diverse students.
• The principles of Universal Design
for Learning can inform the
appropriate use of technology in
assessment.