Transcript Some Terminology and Concepts for Simulation-Based Assessment

Some Terminology and Concepts for
Simulation-Based Assessment
Robert J. Mislevy
University of Maryland
In collaboration with
John T. Behrens & Dennis C. Frezzo
Cisco Systems, Inc.
Presented at the Lidquist Center at the University of Iowa,
Iowa City, IA, September 19, 2007
Sept 19, 2007
U Iowa
Slide 1
Examples
Computer-based, interactive, simulation-based.
Info-processing and sociocultural roots.

PacketTracer (Cisco Systems)
Computer network design & troubleshooting

The Architectural Registration Examination
Architectural design; CAD-like environment

Hydrive
Troubleshooting the F15 hydraulics system


Nat’l Board of Medical Examiner’s Primum® Computer Case
Simulations (CCS)
DISC simulator
Simulations for problem-solving in dental hygiene
Sept 19, 2007
U Iowa
Slide 2
Two Not-Entirely-Satisfactory Ways of Assessing
Competence in Computer Networking
The Cisco Networking Academy supports 60,000+
academies in schools. CNA supplies curriculum &
assessment resources online.
 Standardized multiple-choice exams
» Okay on content knowledge, but ...
» nothing about interacting with functioning networks or
troubleshooting in inquiry cycles.

High fidelity local hands-on exams
» Authentic, high face validity.
» Huge local variation, unreliable, expensive
Sept 19, 2007
U Iowa
Slide 3
PacketTracer

Goal is improving instruction by fostering
communication and enabling agents

The patterns of activity and structure in our
software embody both cognitive theory and
theory of the domain structure.

Tasks are tasks whether you are teaching with
them or testing with them.
Sept 19, 2007
U Iowa
Slide 4
PacketTracer




Create and edit networks
Multiple representations in the logical layer
Inspection tool links to a deeper physical world
Simulation mode
» Detailed visualization and data presentation

Standard support for world authoring
» Library of elements
» Simulation of relevant deep structure
» Copy, paste, save, edit, annotate, lock
Sept 19, 2007
U Iowa
Skip
Screens
Slide 5
Instructors and students can author
their own activities
Explanation
Experimentation
Two ways to design simulation-based assessment
system when you’re looking at thousands of people,
with hundreds of tasks, at high cost and with high
stakes...
The WRONG way
Ask ‘how do you score it?’ after you’ve built the
simulator and scripted a handful of idiosyncratic
scenarios.
Sept 19, 2007
U Iowa
Slide 9
Two ways to design simulation-based assessment
system when you’re looking at thousands of people,
with hundreds of tasks, at high cost and with high
stakes...
A RIGHT way
Design the simulator and the scenarios around what
you want to make inferences about, what you need to
see to ground them, and the structure of the
interrelationships.
Sept 19, 2007
U Iowa
Slide 10
Evidence-Centered Assessment Design
Formal, multiple-layered framework from
Messick’s (1994) guiding questions:



What complex of knowledge, skills, or other
attributes should be assessed?
What behaviors or performances should reveal
those constructs?
What tasks or situations should elicit those
behaviors?
Sept 19, 2007
U Iowa
Slide 11
Evidence-Centered Assessment Design




Principled framework for designing, producing, and
delivering assessments
Process model, object model, design tools
Explicates the connections among assessment
designs, inferences regarding students, and the
processes needed to create and deliver these
assessments.
Particularly useful for new / complex assessments.
Sept 19, 2007
U Iowa
Slide 12
Evidence-Centered Assessment Design

PADI: Principled Assessment Design for Inquiry
»
»

NSF project, collaboration with SRI et al.
http://padi.sri.com
Bob Mislevy home page
»
»
»
http://www.education.umd.edu/EDMS/mislevy/
Links to papers on ECD
Cisco NetPASS
Sept 19, 2007
U Iowa
Slide 13
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Layers in the assessment enterprise
From Mislevy & Riconscente, in press
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Sept 19, 2007
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
U Iowa
Slide 19
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms
of assessment
argument. Conceptualization.
Evidence
Model(s)
Task Model(s)
Student Model
Scoring
Measurement
Model evidence, and
Conceptual Assessment ModelDesign structures: Student,
task models. Generativity.
Framework






Assessment
Implementation


Assessment Delivery

From Mislevy & Riconscente, in press
Sept 19, 2007
X1
X1
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
X2
X2
details, statistical models. Reusability.
1.
3.
5.
7.
xxxxxxx x
xxxxxxx x
xxxxxxx x
xxxxxxx x
2.
4.
6.
8.
xxxxxxxx
xxxxxxxx
xxxxxxxx
xxxxxxxx
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
U Iowa
Slide 20
Packet Tracer: Task Models & Design Wizard
-
-
-
Sept 19, 2007
U Iowa
Inline editing of
feedback for
each assessable
item.
Activity and
observable level
feedback
possible
Scoring based on
network
configuration,
functionality,
and/or solution
process
Slide 21
Packet Tracer:
Variable Manager - Variable Assignment
-
Sept 19, 2007
U Iowa
When the activity
starts, instead of
using the initial
network as the
starting values,
the activity will
configure the
network with the
contents of the
variables as
entered by the
student.
Slide 22
Sample evidence rule from HYDRIVE
IF an active path which includes the failure has not been
created and the student creates an active path which does
not include the failure and the edges removed from the
problem area are of one power class,
THEN the student strategy is splitting the power path
ELSE the student strategy is not splitting the power path.
Note…
•Sequential dependence
•Contextualized definition
•Multiple ways to succeed & fail
•Higher-level definition of what we’re looking for
Sept 19, 2007
U Iowa
Slide 23
NetPass: An Evidence-Model Bayes-Net
Fragment for Troubleshooting Tasks
DK and ImplementE
Efficiency of ProcedureE(OB)
Implement/Configure(SM)
Correctness of ProcedureE(OB)
Networking Disciplinary Knowledge(SM)
Network Modeling(SM)
Sept 19, 2007
Implement ContextE
Correctness of OutcomeE(OB)
DK and Network ModelingE
Correctness of Outcome2E(OB)
U Iowa
Slide 24
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Sept 19, 2007
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
U Iowa
Slide 25
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Sept 19, 2007
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
U Iowa
Slide 26
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Sept 19, 2007
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
U Iowa
Slide 27
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Sept 19, 2007
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
U Iowa
Slide 28
Domain Analysis
What is important about this domain?
What work and situations are central in this domain?
What KRs are central to this domain?
Domain Modeling
How do we represent key aspects of the domain in
terms of assessment argument. Conceptualization.
Conceptual Assessment
Framework
Assessment
Implementation
Assessment Delivery

From Mislevy & Riconscente, in press
Sept 19, 2007
Design structures: Student, evidence, and
task models. Generativity.
Manufacturing “nuts & bolts”:
authoring tasks, automated scoring
details, statistical models. Reusability.
Students interact with tasks,
performances evaluated,
feedback created. Fourprocess delivery architecture.
U Iowa
Slide 29
Conclusion
Insights from cognitive research can improve the
practice of assessment.
Doing so requires …
» Shift in perspective and
» deeper understanding of assessment design.
“Too many notes” ? (Emperor Joseph II on Mozart)
Suitable conceptual frameworks, tools, and exemplars
are now appearing.
» Generativity, re-usability, and inter-operability are keys.
Sept 19, 2007
U Iowa
Slide 30