Transcript The Portal Assessment Design System

Implications of Evidence-Centered Design
for Educational Testing:
Lessons from the PADI Project
Robert J. Mislevy
University of Maryland
Geneva D. Haertel
SRI International
April 12, 2007
Presented at Invited Symposium K3, “Assessment Engineering: An Emerging
Discipline” at the annual meeting of the National Council on Measurement in
Education, Chicago, IL, April 10-12, 2007
PADI is supported by the Interagency Educational Research Initiative (IERI) under grants REC0089122 (PADI Planning Grant)and REC-0129331 (PADI Implementation Grant).Any opinions,
findings, and conclusions or recommendations expressed in this material are those of the authors and
do not necessarily reflect the views of the National Science Foundation.
July 6, 2005
University of Maryland
Slide 1
Outline

Remarks on assessment engineering

Evidence-centered assessment design
 Assessment arguments
 Layers

Implications
 Explicitness, representations, reusability, generativity

PADI

Too many notes?
July 6, 2005
University of Maryland
Slide 2
Assessment Engineering

Is assessment design art or science?

Assessment engineering submits there are
recurring structures and relationships we can
exploit
– in terms of language, data structures,
representational forms, and processes –
to enhance efficiency and validity in
assessment/task design.
July 6, 2005
University of Maryland
Slide 3
Assessment Engineering

Idea not new or exclusive; some examples:
 Guttman, Suppes, Hively et al., Obsbourne, Bormuth ~ 60s.
 Roid & Haladyna (1982): Technology for Test-Item Writing
 Embretson (1985): Test design: Developments in psychology
and psychometrics.
 Embretson, Wilson, Baker, Luecht, Gorin, Tatsuoka, Bejar,
Shavelson, Irvine & Kyllonen…

Evidence-centered design (ECD)
 Mislevy, Steinberg, & Almond (2003)
 PADI project
July 6, 2005
University of Maryland
Slide 4
Assessment Arguments
What complex of knowledge, skills, or other
attributes should be assessed?
What behaviors or performances should
reveal those constructs?
What tasks or situations should elicit those
behaviors?
(Messick, 1994)
July 6, 2005
University of Maryland
Slide 5
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

Design structures: Student, evidence, and
task models
How do we choose and present tasks,
and gather and analyze responses?
How do students and tasks
actually interact?
How do we report examinee
performance?
Layers in the assessment enterprise
From Mislevy & Riconscente, in press
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Design structures: Student, evidence, and
task models
How do we choose and present tasks,
and gather and analyze responses?
Cog Psych,
expertise studies,
domain research
How do students and tasks
actually interact?
How do we report examinee
performance?
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Specific implementations.
Surface elements.
Domain Modeling
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
How do we represent key aspects of the domain in
terms of assessment argument.
Design structures: Student, evidence, and
task models
How do we choose and present tasks,
and gather and analyze responses?
How do students and tasks
actually interact?
How do we report examinee
performance?
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument.
Design structures: Student, evidence, and
task models
Conceptual Assessment
Framework
How do we choose and present tasks,
and gather and analyze responses?
Assessment
Implementation
Assessment Delivery

Assessment Argument
From Mislevy & Riconscente, in press
How do students and tasks
actually interact?
How do we report examinee
performance?
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument.
Conceptual Assessment
Framework
Design structures: Student, evidence, and
task models
How do we choose and
tasks, & purpose
Explicit connection
topresent
domain
and gather and analyze responses?
 Generative structures for recurring kinds
of proficiencies (e.g., inquiry cycles,
Assessment Delivery
How do students and tasks
troubleshooting)
across projects
actually interact?
How do we report examinee
 PADI Design
patterns
performance?

Assessment
Implementation

From Mislevy & Riconscente, in press
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery
Design structures: Student, evidence, and
task models
How do we choose and present tasks,
and gather and analyze responses?
How do students and tasks
actually interact?
How do we report examinee
performance?
Generative Design Schemas

From Mislevy & Riconscente, in press
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument.
Conceptual Assessment
Framework
Assessment
Implementation
Design structures: Student, evidence, and
task models
How do we choose and present tasks,
and gather and analyze responses?
Explicit connection of argument to machinery
 Generative structures for recurring task
Assessment Delivery
do students
and tasks
situations (e.g.,How
item
shells)
actually interact?
How do we report examinee
 Re-usable / interoperable
data structures
performance?
 PADI templates


From Mislevy & Riconscente, in press
PADI: Principled Assessment
Design for Inquiry
Supported by NSF, via IERI program
 Focus on scaling up ECD ideas specifically
focusing on science inquiry tasks
 Partners:

» SRI International (Geneva, co-PI)
» University of Maryland (Mislevy, co-PI)
» University of California at Berkeley (BEAR –
Wilson – & FOSS –Long )
» University of Michigan (BioKIDS – Songer)
July 6, 2005
University of Maryland
Slide 13
PADI

Conceptual frameworks & representations
» Domain Modeling: Design patterns for inquiry
» CAF: Task-design (templates)

Object model for task design
» Data structure

Software tools to aid the design process
» Design system
» Wizards
July 6, 2005
University of Maryland
Slide 14
PADI

Libraries of exemplars …
» Design patterns for assessing inquiry
» Task templates and pointers to tasks
“Scoring engine” (MRCMLM)
 Worked-through applications

» FOSS, BioKids; GLOBE, FCI, Mystery
Powders, Mystery Boxes

Evaluation studies – FOSS & BioKids
July 6, 2005
University of Maryland
Slide 15
PADI
For more information:
padi.sri.com/
Includes lots of tech reports.
July 6, 2005
University of Maryland
Slide 16
Too many notes?


As Emperor Joseph II said to Mozart.
Power in general/abstract representations,
» Reveals fundamental similaries under
assessments that look different on the surface.
» Right perspective for design of systems, but not
necessarily for work within systems.


Some people need to work from first
principles, but…
Scaling up for wide range of users requires
exemplars, support tools, tuned interfaces.
July 6, 2005
University of Maryland
Slide 17