Transcript Document

Surveys of Enacted Curriculum
Using Curricular Measures
for Description & Analysis
SEC Collaborative Meeting
Tampa, FL
February 21, 2005
John L. Smithson, Director, Measures of Enacted Curriculum
Wisconsin Center for Education Research, University of Wisconsin-Madison
[email protected]
The SEC Data - Sets
Distinctions
On-Line
•Data collection/processing/reporting
•Descriptive Data
•Limited Reporting Options
•Easy Access / Indiv. Results
Off-line
•Analytic analyses
•Unlimited reporting options
•Requires data manipulation
Conducting Inquiry Using SEC Data
Forms of Inquiry
Collaborative
or
Evaluative
Teacher Enrichment
School Improvement
Professional Lrng. Comm.
Program Evaluation
Indicator Reporting
Program Management
Performance Modeling
A short history of SEC research
• Reform-Up-Close
(Porter, Kirst, Osthoff, Smithson, Schneider, 1993)
Validation of teacher self-report survey data.
• Upgrading Mathematics (Gamoran, Porter, Smithson, White, 1997)
First content analysis of assessment using content language.
Predictive validity of alignment index comparing instruction & assessments
• Data on Enacted Curriculum (Blank, Porter, Smithson, 2004)
Use of SEC data to facilitate school improvement efforts
First content analysis of state standards
• MSP-PD Study (In progress: Blank, Smithson, Porter, Garet, Birman)
Use of SEC data for program evaluation
Alignment Relationships in Standards-based Reform
Alignment Relationships in Standards-based Reform
Intersection of what is taught with what is tested.
Assessment
Taught,
tested, and in
the standards
Instruction
Standards
Intersection of what is taught with what is in standards.
Intersection of what
is taught with what
is in the standards
A Quantitative Approach to Alignment
Porter-Smithson/SEC Alignment Process
Content analyses of curriculum documents and reports of practice
by content experts using two-dimensional content language.
Multiple raters (w/ content & assessment expertise) using
independent ratings in combination with team discussions.
Content Description [Topic(s) by Cognitive Demand(s)]
Yields Alignment Index based on:
Alignment Analyses
The analytic power of quantitative alignment measures
• Prediction
• Program Evaluation*
• Planning
• Performance Modeling
* Not teacher evaluation
Alignment Analyses
Students perform best when tested on material for which they
have been provided the opportunity to learn.
Correlation of Alignment Index to Achievement Gains
Alignment Index
Class Gains
Student Gains
0.451
0.259
From: Gamoran, et.al. (1997). Uprgading High School Mathematics Instruction. EEPA v19n4pp325-338.
Explaining variation in student learning gains
Learning Gains by Course Type
From:
Upgrading High School Mathematics Instruction ,
(Gamoran, Porter, Smithson, & White, 1997),
EEPAv19n4
12
11.5
11
10.5
Learning Gains Controlling for Content
10
12
9.5
11.5
Time 0
Time 1
Time 2
11
Regents
Algebra
Stretch Regents
Math A/B/UCSMP
Gen. Mth. / Pre-alg.
10.5
10
9.5
Time 0
Time 1
Time 2
Alignment Analyses for School Improvement
Using alignment as an outcome measure
Alignment Index:
Instruction to Standards
Mathematics
Across 4 Districts
Counts
Treatment
99
Control
124
Leaders
16
(Measuring change in alignment over time)
Alignment Analyses for Planning Purposes
Mapping Curriculum Materials
SEC - Online
(www.seconline.org)
Administrative Functions:
Administration Set-up
Review Registrants, Completion Rates
Administrator Report Generator
SEC Reports
Dynamic Sample
Selection
Dynamic Data
Disaggregation
Floating Bars
Sample and
Disaggregate
Counts
Comparison
Charts
Individual
Results
SEC Reports: Instructional Content
Coarse Grain Maps
Content Areas
by
Cognitive Demand
SEC Reports: Instructional Content
Fine Grain Maps
Topics
by
Cognitive Demand
Alignment as a Systemic Tool
Fine Grain
Calculating Alignment
1-((|x-y|(1-n))/2)
X
1
2
3
Y
1
2
3
A
0.5
0.1
0.1
A
0.3
0.2
0
B
0
0.2
0
B
0.1
0.1
0.1
C
0.1
0
0
C
0.2
0
0
z
A
B
C
1
2
3
0.3
0.1
0
0
0.1
0
0.1
0
0
=
Alignment
0.6