Transcript Slide 1
Using the PADI Design System
to Examine the Features of a
NAEP Performance
Assessment
Kathleen C. Haynie
Kathleen Haynie Consulting
Andrea A. Lash, Geneva D. Haertel,
Edys S. Quellmalz, Angela Haydel DeBarger
SRI International
AERA April 2005
Background
Use of 21st century collaborative workplace
tools is on the rise
Technologically rich “dispersed” collaboration
requires (Olson & Olson, 2000):
Common ground
Coupling of work
Technological readiness
Collaborative readiness
2
PADI Project
Seeks to improve assessment of inquiry
in science learning
Networked collaboration
Development and use of Web-based tool
3
Central Process
The central process for project strands working
with the PADI Design System can be stated as
follows:
The interaction between assessment materials (e.g.,
assessment tasks) and the Evidence-Centered Design
framework (Mislevy, Steinberg, & Almond, 2002) is
mediated by various tools, representational forms, and
representations utilized by collaborative workgroups.
4
PADI Assessment Design
5
NAEP Performance Assessment
Goals:
To reverse engineer a performance task from a
large-scale, national assessment
To use the PADI System as an analytical tool for
understanding task features
In reverse engineering and analyzing a
performance assessment task via PADI, what
types of knowledge were created?
6
Reverse Engineering & Analysis Process
1. Selection of a Set of Items (July – August)
2. Exploration of the Floating Pencil Task (August
– December)
3. Development of a Task Specification
(December – March)
7
The Student Model
We selected a set of items for analysis
NSES inquiry standards, PADI Design Patterns
Defined our Student Model
Exploration of multiple student models
Choice of NAEP framework
Refined the Student Model
8
The Evidence Model
We explored evaluation & measurement
issues via the PADI representational forms
Evaluative Sub-Model based on NAEP rubric
Multiple levels of conditional dependencies
MRCML psychometric model
Clarified our Measurement Model
Based closely on NAEP practices
Elements of PADI
9
The Task Model
We explored attributes of the Floating Pencil
task
Content-lean, inquiry-constrained
Use of lab materials; High verbal demand
Constructed Task Model
Sought to define a ‘Family of Tasks’
10
So What?
Process: Floating Pencil team engaged in
collaboration using technological workplace tools
Results:
Analysis of the Floating Pencil task
A ‘trace’ on PADI: Floating Pencil Task Specification
The impact of our work on PADI Design System
Next Steps:
Task family and TMVs
Empirical data analysis
Technical report on Floating Pencil work
Consideration of alternative student models
11