Transcript STANDARDS-BASED ACCOUNTABILITY SYSTEMS: DESIGN …

Some Implications of Expertise
Research for Educational
Assessment
Robert J. Mislevy
University of Maryland
National Center for Research on Evaluation,
Standards, and Student Testing (CRESST)
2005 CRESST Conference
UCLA, Los Angeles, CA
September 8-9, 2005
The Central Idea
 Expertise research seeks to understand
in detail the knowledge, representations,
development, and contexts of expertise.
 Results can ground assessment
arguments & task design.
 Must recast in terms of purpose,
perspective, principles, and structures of
assessment.
 Especially valuable with complex
proficiencies, situations, or
performances.
Outline
 An example: Architectural design
 Implications of expertise research
 Assessment arguments
 Generative schemas for task design
 Conclusion
The Architectural Registration
Examination
 Architectural design; CAD-like
environment.
(ETS: Katz, Bejar, Braun, Hone,
Brittingham, Bennett, et al.)
 To replace 10-hour hand-drawn
design problem
 Reflects changing of profession to
CAD
 Premium on thinking, not drawing
Cognitive task analysis
Planning the firestation site:
A “block diagram” design
problem
An Example of a Task Prompt for the ARE
An Illustrative Base Diagram for an ARE Task
A Sample Solution to an ARE Task
Some Results
Differences between novices and
experts
• Success rate: 98% vs. 88%
• Planning time & execution time
• Patterns of revision involving
rework
Assessment Arguments
What complex of knowledge, skills, or
other attributes should be assessed?
What behaviors or performances
should reveal those constructs?
What tasks or situations should elicit
those behaviors?
(Messick, 1994)
Implications for task design (1)
Experts perceive/understand/act via
fundamental principles rather than
surface features
• Chi, Feltovich, & Glaser on physics
In architecture, Constraints
•
•
•
•
Number
Variation in importance/difficulty
Degree of conflict
Implicit constraints
Implications for task design (2)
Importance of interaction with
situation

Intelligence built into situations, tools,
processes

“Attunement to affordances”

Cycles of …
 Approach
to solution, as in ARE
 Hypothesize,
test, revise, as in
inquiry & troubleshooting
Implications for task design (3)
External knowledge representations
Maps, insurance forms, symbol systems, diagrams,
Punnett squares, wiring charts, blueprints
Roles in practice



Gather/create/share/hold/transform knowledge
Embody generative principles of domain
Central to practice in the domain
Roles in assessment

Environment / stimuli / work of tasks

Design KRs to create / implement assessment
CTA for Assessment
CTA can help ground assessment, re
validity argument and
principled/explicated task construction.

Must aim to specify elements of Student-,
Evidence-, and Task-models

in light of the intended assessment’s
purpose,

and constraints of the assessment-building
and assessment-delivery contexts.
Design patterns
Look across domains to find recurring
difficulties, blockages, overloads
(Salthouse, 1991)
Classes of expertise / observables /
situations that arise in many domains.
Can create “Design patterns” in, e.g.,
• Design under constraints (engineering)
• Problem-solving in finite domains
(troubleshooting)
• Model-based reasoning (science)
Advances in assessment design
A cognitive design system approach to
generating valid tests (Embretson,1998)
Model-basel assessment (Baker, 1997)
Constructing measures (Wilson, 2004)
Understanding by design (Wiggins, 1998)
On the structure of educational
assessments (Mislevy, Steinberg, &
Almond, 2003)
Conclusion
• Insights from expertise research
can improve the practice of
assessment.
• Suitable conceptual frameworks,
tools, and exemplars are now
appearing.
• Design & delivery frameworks are
key to making technology-based /
complex assessment practical.