Transcript Slide 1

Using the PADI Design System
to Examine the Features of a
NAEP Performance
Assessment
Kathleen C. Haynie
Kathleen Haynie Consulting
Andrea A. Lash, Geneva D. Haertel,
Edys S. Quellmalz, Angela Haydel DeBarger
SRI International
AERA April 2005
Background
 Use of 21st century collaborative workplace
tools is on the rise
 Technologically rich “dispersed” collaboration
requires (Olson & Olson, 2000):




Common ground
Coupling of work
Technological readiness
Collaborative readiness
2
PADI Project
 Seeks to improve assessment of inquiry
in science learning
 Networked collaboration
 Development and use of Web-based tool
3
Central Process
The central process for project strands working
with the PADI Design System can be stated as
follows:
The interaction between assessment materials (e.g.,
assessment tasks) and the Evidence-Centered Design
framework (Mislevy, Steinberg, & Almond, 2002) is
mediated by various tools, representational forms, and
representations utilized by collaborative workgroups.
4
PADI Assessment Design
5
NAEP Performance Assessment
 Goals:
 To reverse engineer a performance task from a
large-scale, national assessment
 To use the PADI System as an analytical tool for
understanding task features
 In reverse engineering and analyzing a
performance assessment task via PADI, what
types of knowledge were created?
6
Reverse Engineering & Analysis Process
1. Selection of a Set of Items (July – August)
2. Exploration of the Floating Pencil Task (August
– December)
3. Development of a Task Specification
(December – March)
7
The Student Model
 We selected a set of items for analysis
 NSES inquiry standards, PADI Design Patterns
 Defined our Student Model
 Exploration of multiple student models
 Choice of NAEP framework
 Refined the Student Model
8
The Evidence Model
 We explored evaluation & measurement
issues via the PADI representational forms
 Evaluative Sub-Model based on NAEP rubric
 Multiple levels of conditional dependencies
 MRCML psychometric model
 Clarified our Measurement Model
 Based closely on NAEP practices
 Elements of PADI
9
The Task Model
 We explored attributes of the Floating Pencil
task
 Content-lean, inquiry-constrained
 Use of lab materials; High verbal demand
 Constructed Task Model
 Sought to define a ‘Family of Tasks’
10
So What?
 Process: Floating Pencil team engaged in
collaboration using technological workplace tools
 Results:
 Analysis of the Floating Pencil task
 A ‘trace’ on PADI: Floating Pencil Task Specification
 The impact of our work on PADI Design System
 Next Steps:




Task family and TMVs
Empirical data analysis
Technical report on Floating Pencil work
Consideration of alternative student models
11