Transcript Longitudinal Study to Measure Effects of MSP Professional

Professional Development
Activity Log: A New Approach
to Design, Measurement, Data
Collection, and Analysis
AERA Annual Meeting
San Diego
April 13, 2004
Longitudinal Study to Measure
Effects of MSP Professional
Development on Improving
Math and Science Instruction
Math and Science Partnership
A collaborative study conducted by:
Council of Chief State School Officers (CCSSO)
American Institutes for Research (AIR)
Wisconsin Center for Educational Research (WCER)
Authors
Kwang Suk Yoon, AIR
 Reuben Jacobson, AIR
 Mike Garet, AIR
 Bea Birman, AIR
 Meredith Ludwig, AIR

Research Questions

To what extent is the quality of the professional
development supported by MSP activities consistent
with research-based definitions of quality (e.g., content
focus, active learning, coherence, collective
participation, and sustained efforts) (Garet et al., 2001)?

What effects do teachers' professional development
experiences have on instructional practices and content
taught in math and science classes? Are high-quality
professional development activities more likely than
lower-quality activities to increase the alignment of
instructional content with state standards and
assessments?
Logic Model
Implementation of
Professional
Development
Content Focus
Collective Participation;
Active Learning;
Coherence;
Sustained Effort
Teacher Characteristics:
Background Variables,
Prior PD Experiences
Target Class Students:
Diversity
School Culture:
Trust
PDAL
Year 0
Pre-PD:
Post-PD:
Alignment of
Instruction with Content
Standards;
Instructional Practice
Alignment of
Instruction with Content
Standards;
Instructional Practice
Survey of Enacted
Curriculum wave 1
Survey of Enacted
Curriculum wave 2
Year 1
Year 2
Year 3
Participants

Four MSP projects were selected for the
study. In each project, we are collecting data
with teachers mostly in middle schools or
middle grades about their professional
development in mathematics and science
education.

N=472 teachers
Survey of Enacted Curriculum
(SEC)

Instructional practice (e.g., instructional time
in target class)
 Content coverage and alignment:
– Instructional time on topics and subtopics
– Expectation for students (e.g., memorize facts,
perform procedure, or solve non-routine problems)


Past experiences in professional development
Teacher characteristics (e.g., gender, teaching
experience)
Why PDAL?





Gathers accurate, time-sensitive information;
Minimizes recall problem with retrospective
reports
Collects disaggregate information about specific
PD activities – Reduces bias introduced by gross
data aggregation
Generates context sensitive questions
Is able to tailor technical assistance to teachers
based on their response patterns
Allows teachers to review their own logs –
Teachers can reflect on their own PD experiences
Professional Development Activity Log
(PDAL)

Teachers create an ongoing monthly log of any
professional learning activity in which they
participate
 Longitudinal data collected over 15 months
 Web-based, self-administered log
 Aligned with SEC (e.g., content coverage)
 Inclusive approach to professional development
– Includes MSP-sponsored and non-MSP-sponsored
activities
– Documents one-time and recurring activities
– Captures both formal and informal activities
PDAL Entries








Name of activity
Number of hours spent on each activity and its duration
Whether the activity is a one-time or continuous event (e.g.,
recurring over a number of months)
Type of activity (e.g., workshop, summer institute, study group)
Purpose of activity (e.g., strengthening subject matter knowledge)
PD quality features (e.g., active learning, coherence, collective
participation)
Content focus (e.g., algebraic concepts: absolute values, use of
variables, etc.)
Instructional practice – instructional topics covered in each activity
(e.g., use of calculators, computers, or other educational
technology)
Analysis of PDAL Data

Implementation analysis
– Patterns of responses to monthly logs
– Response rates; sample attrition; extent of missing data

Descriptive analysis
– Patterns of teachers’ PD experiences
– Correlates of high-quality PD activities
– Latent classes of teachers based on their PD
experiences

Impact analysis
– Assess the impact of PD on math & science instruction
Table 1: PDAL Data Structure:
Disaggregated log data: Teacher by
activity by time
Teacher
Mike
Julie
Reuben
Kwang
Bea
Activity
Jul-03
AA
B
C
AA
D
EE
EE
F
G
1
Aug-03
Sep-03
1
Oct-03
Nov-03
1
Dec-03
Jan-04
1
1
1
1
1
Feb-04
1
1
1
1
1
1
1
1
1
1
1
1
Mar-04
1
# of logs
1
5
1
1
6
2
3
1
1
Table 2: PDAL Data Structure:
Activity-level data aggregated across
teachers
Teacher
Activity
AA
B
C
D
EE
F
G
# of teachers
Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04 Mar-04 # of logs
2
0
0
0
1
0
0
3
0
0
0
1
2
0
0
3
0
1
0
1
2
1
0
5
0
1
0
0
0
0
0
1
0
0
0
1
0
0
1
2
0
1
1
1
0
0
0
3
0
1
0
1
0
0
0
2
0
0
0
1
0
0
0
1
0
1
0
0
0
0
0
1
2
5
1
6
5
1
1
21
Implementation Analysis

# of Activities (over 9 months)
– Range: 1 - 21
– Mean: 3.1

# of Logs (over 9 months)
– Range: 1 - 31
– Mean: 4.9

Sessions
– Per session: mean 12 min.
– Per teacher over 9 months: mean of 50 min.
83% finished in one session; 13% finished in
two sessions.
Implementation Analysis

Timing of log entries:
– Weekends are least popular. Tues. - Thurs. are
most popular
– Spike at the beginning of each month

User understanding of the instrument
– 95% of logs created were fully completed

Communication:
– Info Packets
– Phone calls, emails, letter reminders
– Helpline
Table 3: Duration and Contact Hours of
PD Activities
Variable Source
Duration
N
Mean
SD
Min
Max
1121
1.84
1.30
0
4
229
1.87
1.00
0
4
1121
15.72
20.98
1
160
229
18.61
20.91
1
150
229
76.97
95.17
1
900
1
PDAL: Logs
PDAL: Teachers
Contact Hours
PDAL: Logs
PDAL: Teachers
Total Contact Hours
PDAL: Teachers
Note: 1 How many days: 0=less than a day, 1=one day, 2=2-4 days, 3=a week, 4=entire month.
Table 4: PD Quality Construct
Active Learning
Source
PDAL: Logs
PDAL: Teachers
SEC1: Teachers
N
Mean
SD
Min
Max
1107
1.08
0.76
0
3
225
382
1.15
1.26
0.63
0.67
0
0
3
3
Sample Items:
Observe demonstrations of teaching techniques?
Review student work or score assessments?
Receive coaching or mentoring in the classroom?
Give a lecture or presentation to colleagues?
On the scale of 0=never, 1=rarely, 2=sometimes, 3=often
Table 5: PD Quality Construct
Coherence
Source
PDAL: Logs
PDAL: Teachers
SEC1: Teachers
N
Mean
SD
Min
Max
1067
2.30
0.66
0
3
224
359
2.28
1.85
0.49
0.71
0
0
3
3
Sample Items:
Designed to support the school-wide improvement plan adopted by your school?
Consistent with your department or grade level plan to improve teaching?
Consistent with your own goals for your professional development?
Based explicitly on what you had learned in earlier professional development activities?
On the scale of 0=never, 1=rarely, 2=sometimes, 3=often
Table 6: PD Quality Construct
Collective Participation
Source
N
Mean
SD
Min
Max
1105
0.64
0.74
0
2
PDAL: Teachers
225
0.72
0.61
0
2
PDAL: Teachers(Agg)
225
382
1.21
1.18
0.77
0.86
0
0
2
2
PDAL: Logs
SEC1: Teachers
Items:
I participated with most or all of the teachers from my school.
I participated with most or all of the teachers from my department or grade level.
On the scale of 0 - 2: Sum of the two collective participation items
Conclusion:
Revisiting the Logic Model
Implementation of
Professional
Development
Content Focus
Collective Participation;
Active Learning;
Coherence;
Sustained Effort
Teacher Characteristics:
Background Variables,
Prior PD Experiences
Target Class Students:
Diversity
School Culture:
Trust
PDAL
Year 0
Pre-PD:
Post-PD:
Alignment of
Instruction with Content
Standards;
Instructional Practice
Alignment of
Instruction with Content
Standards;
Instructional Practice
Survey of Enacted
Curriculum wave 1
Survey of Enacted
Curriculum wave 2
Year 1
Year 2
Year 3
Contact Information
Kwang Suk Yoon
(202) 403-5358
[email protected]
Visit us in the future
www.pdal.net
Reuben Jacobson
(202) 403-6925
[email protected]