Transcript Longitudinal Study to Measure Effects of MSP Professional

Symposium - Evaluating the Quality of Professional Development:
Implications for Districts and States
Professional Development Activity
Log: Comparing Teacher Log and
Survey Approaches to Evaluating
Professional Development
Kwang Suk Yoon
Reuben Jacobson
American Institutes for Research
AERA Annual Meeting
Montreal, April 11, 2004
Overview




Comparisons between teacher logs
with survey
Teachers’ experience with logs
Lessons learned
Next steps
Professional Development Activity Log
(PDAL)



The PDAL is a web-based, self-administered,
longitudinal data collection tool for teachers
to record their professional development
experiences in detail with the assistance of a
series of structured prompts
Teachers log on to their password-protected
web account and fill out their PDAL at regular
intervals
Visit www.PDAL.net for more information
PDAL Entries








Name of activity
Number of hours spent on each activity and its duration
Whether the activity is a one-time or continuous event (e.g.,
recurring over a number of months)
Type of activity (e.g., workshop, summer institute, study group)
Purpose of activity (e.g., strengthening subject matter knowledge)
PD quality features (e.g., active learning, coherence, collective
participation)
Content focus (e.g., algebraic concepts: absolute values, use of
variables, etc.)
Instructional practice – instructional topics covered in each activity
(e.g., use of calculators, computers, or other educational
technology)
Why PDAL?





Collects disaggregate information about specific PD
activities – Increases the level of specificity of PD data
and reduces bias introduced by gross data
aggregation
Gathers accurate, reliable, and time-sensitive
information – Minimizes recall problem with
retrospective reports
Tailors technical assistance to teachers based on their
response patterns
Allows teachers to review their own logs – Teachers
can reflect on their own PD experiences
Generates context sensitive questions
“Black Box” of Survey Data
Hypothetical Data
Teacher
Time
Activity
Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04
Total
Contact
Hours
Mean
Active
Learning
A,B,C
32
30
12
-
21
-
14
-
109
0.8
Ms. Lopez
A,D
32
32
-
-
10
10
12
10
106
1.3
Mrs. Kelly
E
-
20
20
-
-
-
-
-
40
0.5
E,F
18
18
44
18
-
-
-
-
98
0.8
Mrs. Smith
G
-
-
-
-
8
-
-
-
8
1.0
Aggregate
0
82
100
76
18
39
10
26
10
361
0.9
Mr. Anderson
Mr. Lee
Fine-grain log-level data on contact hours :
Disaggregated by teacher by activity by time
Hypothetical Data
Teacher
Time
Activity
Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04
Total
Contact
Hours
Mean
Active
Learning
Mean
Contact
Hours
A
32
30
-
-
-
-
-
-
62
2
31.0
B
-
-
12
-
13
-
14
-
39
3
13.0
C
-
-
-
-
8
-
-
-
8
1
8.0
A
32
32
-
-
-
-
-
-
64
2
32.0
D
-
-
-
-
10
10
12
10
42
4
10.5
Mrs. Kelly
E
-
20
20
-
-
-
-
-
40
2
20.0
Mr. Lee
E
18
18
16
-
-
-
-
-
52
3
17.3
F
-
-
28
18
-
-
-
-
46
2
23.0
G
-
-
-
-
8
-
-
-
8
1
8.0
Total Contact Hours
82
100
76
18
39
10
26
10
361
20
18.1
No of Logs
3
4
4
1
4
1
2
1
27.3
25.0
19.0
18.0
9.8
10.0
13.0
10.0
Mr. Anderson
Ms. Lopez
Mrs. Smith
Mean Contact Hours
Validation of Teacher Logs as an
Alternative Data Collection Method

Comparing PDAL and exit survey
results
– Measurement properties
– Correlations
– Mean levels

Assessing the relative efficacy of
teacher logs and survey
Method

Instruments
– PDAL
• Conduced over 15 months
– Exit Survey
• Follow-up questions about PD activities over the same
period of 15 months
• Questions about teachers’ experiences with PDAL

Sample
– 4 Math-Science Partnership Program projects
– Sample
• 476 math and science teachers mostly in middle or high
schools
– Participants
• 326 teachers completed at least for a month in PDAL
• 165 teachers participated in the Exit Survey
Measurement Properties of PDAL
# of
items
# of
factors
Active learning
8
2
0.86
0.76
Coherence
Collective participation
5
2
1
1
0.83
0.62
0.85
0.76
Variable
Reliability (a coef.)
PDAL Survey
Agreement between Log & Survey Methods:
Correlations between methods
Correlation
between methods
p
Active learning
Coherence
0.44
0.42
***
***
Collective participation
Contact hours in WIC
0.56
0.55
***
***
Frequency of PDAL use
0.79
***
Variable
Note: Comparisons are restricted to teachers who participated in PDAL for 12 months or longer.
WIC denotes workshop, summer institute, and college course.
Agreement between Log & Survey Methods:
Mean Levels
Log
Mean
Survey
Mean
Survey
effect
1.04
2.21
1.42
2.11
Collective participation
Contact hours in WIC
0.65
68.60
Frequency of PDAL use
4.82
Variable
Active learning
Coherence
(s.e.)
t
d.f.
p
0.38
-0.11
0.06 6.05
0.06 -1.75
96
94
***
ns
0.93
68.29
0.28
-0.30
0.07 3.80
5.26 -0.06
96
89
***
ns
4.57
-0.25
0.07 -3.54
111
***
Note: Comparisons are restricted to teachers who participated in PDAL for 12 months or longer.
Teachers’ Experience with or Opinions
about PDAL:
Results from the PDAL Exit Survey
Variable
N
Mean
SD
Min
Max
On the scale of 1=strongly disagree, 2=disagree, 3=agree, and 4=strongly agree.
Usability of the PDAL
Support for PDAL Use
Technical Difficulties
Technology Obstacles
Lack of Interest in Surveys
Benefits of the PDAL
148
148
159
161
161
148
2.78
3.00
2.03
1.58
2.23
2.60
0.50
0.64
0.67
0.55
0.67
0.60
1.3
1.0
1.0
1.0
1.0
1.0
4.0
4.0
4.0
3.0
4.0
4.0
161
1.75
0.43
1.0
2.0
On the scale of 1=yes and 2=no.
Prefer a paper version?
Unique Benefits of PDAL

Rich, in-depth data with a high level of
specificity
– Differences in features between PD sponsored by
MSP vs. other PD
– Topic intensity (i.e., amount of contact hours per
topic)

Time-dependent measures

Can be used for on-going formative evaluation
to continuously improve PD
– Percent of months with PD
– Average contact hours by month
– Average span of activity
– Episode-specific comments and feedback
Average Contact Hours by Month
40
35
30
Hours
25
20
15
10
5
0
Jul- Aug- Sep- Oct- Nov- Dec- Jan- Feb- Mar- Apr- May- Jun- Jul- Aug03 03 03 03 03 03 04 04 04 04 04 04 04 04
Month
Lessons Learned & Implications

Complementary uses of logs & surveys for
different purposes
– Globally estimating the mean level of PD activities
– Investigating the variability of specific PD elements
and relating it to other outcomes
– Improving PD design

Cost and benefits of different data collection
methods
 Improving survey method: Increase the level of
specificity
Next Steps

Final phase of data analysis
– Using both PDAL and survey data to assess
change in teaching practice and assess their
relative predictive validity



PDAL users focus group (May 2005)
PDAL usability study
Need for follow-up studies
– Such as a new CCSSO-AIR study on improving
evaluation of professional development in math and
science at state and local levels
Contact Information
Kwang Suk Yoon
(202) 403-5358
[email protected]
Reuben Jacobson
(202) 403-6925
[email protected]
Visit us
www.PDAL.net
www.air.org