Transcript Leveraging Evidence-Centered Design Within Scenario

The Critical Role of Design
Patterns in Large-Scale
Assessment
Robert J. Mislevy & Min Liu University of Maryland
Geneva Haertel SRI International
DR K-12 grant #0733172, “Application of Evidence-Centered Design to State
Large-Scale Science Assessment.”
This material is based upon work supported by the National Science Foundation under Grant No. DRL- 0733172. Any opinions, findings,
and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the
National Science Foundation.
Three design challenges
 Hard-to-assess standards in the domain of
interest
 e.g., inquiry science skills
 Efficient and valid design and development of
complex tasks
 e.g., scenarios, simulations
 Accessibility of tasks for diverse learners
 varying perceptual and expressive capabilities
 valid use of assistive technology, modifications,
alternative assessments
Design Patterns
 Design Patterns in Architecture
 Design Patterns in Software Engineering
 Design Patterns in Literature
Motivation for
Assessment Design Patterns
 In-between structure, to connect...
 Thinking about science learning & inquiry
 Technical elements of measurement & delivery
 Narrative, not technical, contents
 Some Design Patterns from PADI
 Model-Based Reasoning
 Model Formation; Evaluation; Model Revision; Use
 Observational & Experimental Investigations
 Systems Thinking
Motivation for
Assessment Design Patterns
 They lay out a design space for developers
 Choices, connections, examples
 Things to be aware of (e.g., research on
Universal Design for Learning)
 Can improve both Efficiency + Validity
 Attributes reflect assessment argument
structure
Assessment Arguments

What complex of knowledge,
skills, or other attributes
should be assessed?

What behaviors or
performances should reveal
those constructs?

What tasks or situations
should elicit those behaviors?
Messick, S. (1994). The interplay of evidence and
consequences in the validation of performance
assessments. Educational Researcher, 23(2), 13-23.
Student
Model
Evidence
Model
Task
Model
Mislevy, R.J., & Haertel, G. (2006).
Implications for evidence-centered design
for educational assessment. Educational
Measurement: Issues and Practice, 25, 6-20.
The Structure of Assessment Design Patterns
ATTRIBUTE
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
Rationale
How/why this DP addresses evidence
about focal KSAs.
Focal
Knowledge,
Skills, Abilities
The primary knowledge / skills /
abilities (KSAs) targeted by this design
pattern.
Additional
KSAs
Other knowledge/skills/abilities that
may be required by tasks.
Characteristic
features of
tasks
Aspects of assessment situations that
are needed to evoke evidence about
the focal KSAs.
Variable
features of
tasks
Aspects of assessment situations that
can be varied to shift difficulty or
focus.
Potential work
products
What students actually say, do, or
make, to produce evidence.
Potential
observations
Aspects of work products we might
identify and evaluate, as evidence
about students’ KSAs.
Potential
rubrics
Ways of evaluating work products to
produce values of observations.
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
Rationale
How/why this DP addresses evidence
about focal KSAs.
Focal
Knowledge,
Skills, Abilities
The primary knowledge / skills /
abilities (KSAs) targeted by this design
pattern.
The Rationale
background that
Additional provides
Other knowledge/skills/abilities
KSAs of the
may be
requiredKSAs,
by tasks.
into the nature
Focal
Characteristic
Aspects of
assessment
situations
and the
kinds of things
that
people
do that
features of
are needed to evoke evidence about
in what kinds
of
situations
tasks
the focal KSAs.that
evidence
it. E.g., Aspects
overview,
research
Variable
of assessment
situations that
features of
can be varied to shift difficulty or
links, examples.
tasks
focus.
Potential work
products
What students actually say, do, or
make, to produce evidence.
Potential
observations
Aspects of work products we might
identify and evaluate, as evidence
about students’ KSAs.
Potential
rubrics
Ways of evaluating work products to
produce values of observations.
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
DESCRIPTION
Rationale
How/why this DP addresses evidence
about focal KSAs.
Focal
Knowledge,
Skills, Abilities
The primary knowledge / skills /
abilities (KSAs) targeted by this design
pattern.
Additional
KSAs
Other knowledge/skills/abilities that
may be required by tasks.
The design patternCharacteristic
is organizedAspects of assessment situations that
Evidence
around Focal KSAs. features
They ofwill are
beneeded to evoke evidence about
tasks
the focal KSAs.
Model involved in the Student
Model,
Variable
Aspects of assessment situations that
features
can be varied to shift difficulty or
be
otherofKSAs
tasks
focus.
although there may
that are included in the target of
Potential work What students actually say, do, or
Taskinference (e.g., Model
Revision—
products
make, to produce evidence.
context?).
Potential
Aspects of work products we might
Modelbut what models, what
observations
identify and evaluate, as evidence
about students’ KSAs.
Associated with
Potential
Ways of evaluating work products to
Characteristic
Features of
rubrics
produce values of observations.
Tasks.
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
Rationale
How/why this DP addresses evidence
about focal KSAs.
Focal
Knowledge,
Skills, Abilities
The primary knowledge / skills /
abilities (KSAs) targeted by this design
pattern.
Additional
KSAs
Other knowledge/skills/abilities that
may be required by tasks.
Characteristic
features of
tasks
Aspects of assessment situations that
are needed to evoke evidence about
the focal KSAs.
Variable
Aspects of assessment situations that
Potential
rubrics
Ways of evaluating work products to
produce values of observations.
Additional
KSAs
play
multiple
features
of
can
be varied
to shift difficulty or
tasksneed to
focus.
roles. You
think about
work What students actually say, do, or
whichPotential
ones
you
really
DO want
products
make, to produce evidence.
to include
as targets
of inference
Potential
Aspects
of work products we might
observations
identifyones
and evaluate,
(validity)
and which
you as evidence
about students’ KSAs.
really DON’T (invalidity).
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
Rationale
How/why this DP addresses evidence
about focal KSAs.
Focal
Knowledge,
Skills, Abilities
The primary knowledge / skills /
abilities (KSAs) targeted by this design
pattern.
Additional
KSAs
Other knowledge/skills/abilities that
may be required by tasks.
Characteristic
features of
tasks
Aspects of assessment situations that
are needed to evoke evidence about
the focal KSAs.
Variable
Aspects of assessment situations that
Potential
rubrics
Ways of evaluating work products to
produce
valuesof
of observations.
Features
Tasks.
The Additional
DOto shift difficulty or
features KSAs
of
canyou
be varied
tasks as targets
focus. of
want to include
Potential work What students actually say, do, or
inference
are
part of make,
the to
claim.
products
produce evidence.
E.g., knowing
Mendel’s
laws as
Potential
Aspects of work products we might
observations
and evaluate,
well as being
able toidentify
formulate
a as evidence
about students’ KSAs.
model in an investigation.
Connected with Variable
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
Rationale
How/why this DP addresses evidence
about focal KSAs.
Focal
Knowledge,
Skills, Abilities
The primary knowledge / skills /
abilities (KSAs) targeted by this design
pattern.
Additional
KSAs
Other knowledge/skills/abilities that
may be required by tasks.
Characteristic Aspects of assessment situations that
features of
are needed to evoke evidence about
Additional
KSAs
tasks
the focalyou
KSAs.
The
Aspects of as
assessment situations that
DON’TVariable
want to include
of
can be varied to shift difficulty or
targetsfeatures
oftasks
inference
introduce
focus.
construct-irrelevant
reasons
for say, do, or
Potential work What
students actually
products
make,
to produce evidence.
poor performance.
(Especially
Potential
Aspects of work products we might
important
for
assessing
special
observations
identify and evaluate, as evidence
about
populations – UDL
&students’ KSAs.
Connected
with Variable
Potential
Ways of evaluating work products to
accommodations.)
Features of Tasks & Work
rubrics
produce values of observations.
Products.
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
Rationale
How/why this DP addresses evidence
about focal KSAs.
Focal
Knowledge,
Skills, Abilities
The primary knowledge / skills /
abilities (KSAs) targeted by this design
pattern.
Additional
KSAs
Other knowledge/skills/abilities that
may be required by tasks.
Characteristic
features of
tasks
Aspects of assessment situations that
are needed to evoke evidence about
the focal KSAs.
Variable
features of
tasks
Aspects of assessment situations that
can be varied to shift difficulty or
focus.
Features
PotentialThe
workCharacteristic
What students actually
say, do, or of
products
make, to produce evidence.
Tasks help you think about
Potential
Aspects of work products we might
critical
features
of the
tasks
observations
identify
and evaluate,
as evidence
about students’
KSAs. to get
situation
you need
Potential
Ways
of evaluating
work products
to
evidence
about
the Focal
KSAs.
rubrics
produce values of observations.
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
Rationale Features
How/why this
addresses
evidence
Variable
of DP
Tasks
also
about focal KSAs.
help
you think
about data
Focal
The primary knowledge / skills /
Knowledge,
(KSAs) targeted
by this design
concerningabilities
the situation
– but
Skills, Abilities pattern.
now to influence difficulty …
Additional
KSAs
Other knowledge/skills/abilities that
may
required
by reduce
tasks.
or
tobebring
in or
demand
Characteristic
features of
tasks
for
Additional
KSAs
to avoid
Aspects
of assessment
situations
that
are needed toexplanations.
evoke evidence about
alternative
Variable
features of
tasks
Aspects of assessment situations that
can be varied to shift difficulty or
focus.
Potential work
products
the focal KSAs.
What students actually say, do, or
make, to produce evidence.
Potential
observations
Aspects of work products we might
identify and evaluate, as evidence
about students’ KSAs.
Potential
rubrics
Ways of evaluating work products to
produce values of observations.
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
Rationale
How/why this DP addresses evidence
Some
Variable
Features of Tasks
about focal KSAs.
help
you match
features of tasks
Focal
The primary knowledge / skills /
Knowledge,
abilities (KSAs)
targeted by this
and background
/ knowledge
/ design
Skills, Abilities pattern.
characteristics of students:
Additional
Other knowledge/skills/abilities that
Interests,
familiarity,
KSAs
may be requiredprevious
by tasks.
instruction.Aspects of assessment situations that
Characteristic
features of
tasks
are needed to evoke evidence about
the focal KSAs.
Variable
features of
tasks
Aspects of assessment situations that
can be varied to shift difficulty or
focus.
Potential work
products
What students actually say, do, or
make, to produce evidence.
Potential
observations
Aspects of work products we might
identify and evaluate, as evidence
about students’ KSAs.
Potential
rubrics
Ways of evaluating work products to
produce values of observations.
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Rationale
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
How/why this DP addresses evidence
aboutProducts
focal KSAs. help you
Work
Potential
Focal
The primary knowledge / skills /
think
about
what
you want to
Knowledge,
abilities (KSAs) targeted by this design
Skills, Abilities
capture
frompattern.
a performance –
Additionalprocess,
Other knowledge/skills/abilities
that
product,
constructed
KSAs
may be required by tasks.
model, written explanation, etc.
Characteristic
features of
tasks
Aspects of assessment situations that
are needed to evoke evidence about
the focal KSAs.
Variable
features of
tasks
Aspects of assessment situations that
can be varied to shift difficulty or
focus.
Potential work
products
What students actually say, do, or
make, to produce evidence.
Potential
observations
Aspects of work products we might
and evaluate, as evidence
Canidentify
also
call
attention to demand
about students’ KSAs.
Potential
rubrics
for Additional KSAs, & avoid
Ways of evaluating work products to
alternative
explanations
for poor
produce values
of observations.
performance
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
Evidence
Model
Rationale
How/why this DP addresses evidence
about focal KSAs.
Focal
Knowledge,
Skills, Abilities
The primary knowledge / skills /
abilities (KSAs) targeted by this design
pattern.
Additional
KSAs
Other knowledge/skills/abilities that
may be required by tasks.
Potential Observations are
Characteristic Aspects of assessment situations that
possibilities
the
qualities
features
of
are for
needed
to evoke
evidenceof
about
tasks Products
the focal KSAs.
Work
– i.e., the data
Variable
Aspects
ofperformance.
assessment situations that
concerning
the
features of
can be varied to shift difficulty or
tasks
Task
Model
DESCRIPTION
Potential work
products
focus.
What students actually say, do, or
make, to produce evidence.
Potential
observations
Aspects of work products we might
identify and evaluate, as evidence
about students’ KSAs.
Potential
rubrics
Ways of evaluating work products to
produce values of observations.
How Design Patterns Support Thinking
about the Assessment Argument
ATTRIBUTE
Student
Model
Evidence
Model
Task
Model
DESCRIPTION
Rationale
How/why this DP addresses evidence
about focal KSAs.
Focal
Knowledge,
Skills, Abilities
The primary knowledge / skills /
abilities (KSAs) targeted by this design
pattern.
Additional
KSAs
Other knowledge/skills/abilities that
may be required by tasks.
Characteristic Aspects of assessment situations that
features of
are needed to evoke evidence about
tasksPotential
the focal
KSAs. are
And
Rubrics
Variable
Aspects of assessment
algorithms/rubrics/rules
forsituations that
features of
can be varied to shift difficulty or
evaluating
Work
tasks
focus. Products to
Potential
workevidence
What students
actually
interpret
from
thesay, do, or
products
to produce evidence.
student’s make,
performance.
Potential
observations
Aspects of work products we might
identify and evaluate, as evidence
about students’ KSAs.
Potential
rubrics
Ways of evaluating work products to
produce values of observations.
Current Catalog of Design Patterns
ECD/PADI related projects have produced over 100
Design Patterns
Domains include science inquiry, science content,
mathematics, economics, model-based reasoning
Span grades 3-16+
Organized around themes, models, and processes,
not surface features or formats of tasks
Support the design of scenario-based, multiple choice, and
performance tasks
Current Catalog of Design Patterns
Education
Standards
Unifying
Themes/
Inquiry
Big Ideas
within
Disciplines
Learning
Progressions
Language
Proficiency
Total
Science
17
57
4
2
0
80
Mathematics
30
2
3
0
0
35
Economics
0
0
3
0
0
3
Language Arts
30
0
0
0
1
31
Management/
Business
0
7
3
0
0
10
Second
Language
Acquisition
0
0
0
0
3
3
Grand Total
77
66
13
2
4
162
Subject Areas
For more information…

PADI: Principled Assessment Design for
Inquiry




http://padi.sri.com
Links to NSF & IES follow-on projects
Lots of Tech Reports, interactive online examples
Bob Mislevy home page



http://www.education.umd.edu/EDMS/mislevy/
Links to papers on ECD
Cisco applications
Now for the Good Stuff …
 Examples of design patterns with content
 Different projects
 Different grain sizes
 Different users
 How they are being used to tackle
pervasive challenges of large-scale
assessment.
 How they evolved to suit needs of users
 Same essential structure, but
 Representations, language, emphases, and
affordances tuned to users and needs