Transcript Predictors of Success: High Improvement Schools versus Low

Key SINet Impacts:
Immediate Impacts
Predictors of Success
2-Year Sustained Impacts
7-Year Longevity
Prepared by: Steven H. Shaha, PhD, DBA
1
Disclosures & Bio
Prof. Steven H. Shaha, PhD, DBA
Professor, Center for Public Policy & Administration
Independent Evaluator
100+ peer-reviewed publications, 200+ peer-reviewed presentations, 2 books
Advisory and consulting work for 11 governments on four continents
Educational work in 16 States
Professor or Lectured or at 14 universities and graduate schools
e.g. UCLA, Harvard, Cambridge (UK), King’s College, London University,
South Manchester, Yale, Princeton, Columbia, Cornell, Zayed Univ (UAE)
Advisory and consulting to 50+ non-education-focused organizations, for example:
Disney, Ritz-Carlton, Coca-Cola, New Line Cinema, IBM, AT&T,
Time Warner, Johnson & Johnson, Marriott, 3 Depts. of Defense,
and Pharma: Sanofi, Aventis, Novartis
Education:
PhD, Research Methods & Applied Statistics
DBA, Business Administration (PhD)
MA, MEd, BS
Executive Summary
• Next Study – Aligned Seminars
• Next Study – Aligned
Implementation
• Next Study – Aligned Observation
4. Advantages with Longevity into
6th and 7th Years
3. Advantages Sustained into
Second Year of PD 360
2. Prdictors of Success
High vs. Lower Engagement
1. Advantage with PD 360
versus none, Year-over-Year
4
Immediate Impacts
Intent: Define year-to-year changes associated with PD 360
Participation
– Systematic sample of 169 elementary schools, in 73 districts, in 19 States
– Schools eligible for inclusion in the sample as participating Schools met the
following criteria:
• More than 10 teachers total
• 80% or more of teachers viewed materials
• Minimum average of 90.0 minutes of viewing per teacher for the school
– Districts included were only those for which eligible schools were included
• Normalized for socio-economic and demographic factors
Data
– Unit of measure was percent Proficient or Advanced as metric
– Student performance data were captured from publically available, Internetaccessed sources
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation
and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
Immediate Impacts
MATH:
11.1 net PD 360 gain in Pct Adv+Prof (p<0.001)
Vs. 2.6 net gain for Districts
Reading:
10.3 net PD 360 gain in Pct Adv+Prof (p<0.001)
Vs. 1.5 net gain for Districts
That equals:
18.9% improved year-over-year
Vs. 4.2% improvement for Districts
That’s 4.5 times MORE Improvement
than their Districts
That equals:
18.1% improved year-over-year
Vs. 2.5% improvement
That’s 7.4 times MORE improvement
than their Districts
Comparative Change in
Reading Performance
Comparative Change in
Math Performance
72.0
70.0
68.0
66.0
64.0
62.0
60.0
58.0
56.0
54.0
52.0
69.5
65.5
62.8
Schools
Districts
58.4
Year 1
Year 2
68.0
66.0
64.0
62.0
60.0
58.0
56.0
54.0
52.0
50.0
67.2
61.2
62.7
Schools
56.9
Year 1
Districts
Year 2
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation
and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
Predictors of Success
Closed the Gap:
Surpassed the Gap:
267% advantage in gains for higher
engagement schools (p<.001)
Nearly 4 times the impact
3,520% advantage in gains for higher
engagement schools (p<.001)
36 times greater impact
18.0% gain for higher engagement schools
4.9% gain for lower engagement schools
(p<.001)
(p<.01)
18.9% gain for higher engagement schools (p<.001)
0.5% gain for lower engagement schools (p=ns)
Shaha SH, Ellsworth H (2014). Predictors of Success for Professional Development: Linking Student Achievement to School and Educator Successes through OnDemand, Online Professional Learning. Journal of Instructional Psychology (JIP), Vol. 40, No. 1
7
Predictors of Success
Higher
Engagement
Schools
Lower
Engagement
Schools
Difference
p-va l ue
Percent
Difference
Magnitude
Educator Utilization
Segments Viewed (s chool cumul a tive)
2298.0
528.9
1769.1
0.000
334.5%
Programs Viewed
588.7
223.5
365.3
0.000
163.5%
Minutes Viewed
359.9
180.6
179.3
0.000
99.3%
Registered Users
87.7%
49.0%
38.7%
0.000
78.9%
Registered Users Viewing at 90 min
79.1%
48.7%
30.4%
0.000
62.4%
Forums Viewed
138.7
87.0
51.7
0.000
59.4%
Total Users Licensed
93.9%
61.4%
32.5%
0.000
52.9%
Follow-up Questions Answered
359.6
167.0
192.6
0.000
115.3%
Reflection Questions Answered
588.3
340.4
247.9
0.000
72.8%
46.3
29.3
17.0
0.000
58.0%
3120.7
2149.0
971.7
0.000
45.2%
Files Downloaded
45.2
35.4
9.8
0.011
27.7%
Forums Posted
28.5
23.4
5.1
0.031
21.8%
43.0%
36.5%
6.5%
0.045
17.8%
65.2
29.2
36.0
0.000
123.2%
3120.7
2149.0
971.7
0.000
45.2%
4.3
2.6
2.0
1.8
1.6
1.6
1.5
Times
2.2
1.7
1.6
1.5
1.3
1.2
1.2
Times
2.2
1.5
Times
Times
Times
Times
Times
Times
Times
Educator Engagement
Files Uploaded
Focus Objectives Set Up
Pct of Users in Communities
Times
Times
Times
Times
Times
Times
Leadership, Implementation, Accountability
Observations Performed
Focus Objectives Set Up (school cumulative)
Times
8
Shaha SH, Ellsworth H (2014). Predictors of Success for Professional Development: Linking Student Achievement to School and Educator Successes through OnDemand, Online Professional Learning. Journal of Instructional Psychology (JIP), Vol. 40, No. 1
2-Year Sustained Impacts
Intent: Quantify sustained impacts of PD360
Participation
–
–
–
–
Second Year follow-on study
Systematic sample of 169 elementary schools, in 73 districts, in 19 States
Same eligibility criteria for School versus District inclusion
Districts included were only those for which eligible schools were included
• Normalized for socio-economic and demographic factors
Data
– Unit of measure was percent Proficient or Advanced as metric
– Student performance data were captured from publically available, Internetaccessed sources
Glassett, K. & Shaha, S. (2014). Lessons learned from a two year implementation project: Sustaining Student Gains with Online On-demand Professional Development.
In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2014 (pp. 660-667). Chesapeake, VA: AACE.
Glassett K, Shaha S & Ellsworth H. (2014). Are Student Gains Sustainable when using Online On-demand Professional Development? Association of Teacher Educators 2014
Summer Conference. Niagara Falls, NY: The Association of Teacher Educators. August 2014.
2-Year Sustained Impacts
MATH vs. baseline:
16.4 net PD 360 gain in Pct Adv+Prof (p<0.001)
13.5 more net gain than respective Districts
Reading vs. baseline:
17.2 net PD 360 gain in Pct Adv+Prof (p<0.001)
13.5 more net respective Districts (again)
That equals:
28.1% improved from baseline (p<0.001)
That’s 4.6 times MORE improvement
than their Districts
That equals:
30.2% improved from baseline (p<0.001)
That’s 5.0 times MORE improvement
than their Districts
Glassett, K. & Shaha, S. (2014). Lessons learned from a two year implementation project: Sustaining Student Gains with Online On-demand Professional Development.
In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2014 (pp. 660-667). Chesapeake, VA: AACE.
Glassett K, Shaha S & Ellsworth H. (2014). Are Student Gains Sustainable when using Online On-demand Professional Development? Association of Teacher Educators 2014
Summer Conference. Niagara Falls, NY: The Association of Teacher Educators. August 2014.
7-Year Longevity
Intent: What is the long-term impact of PD 360 participation
Design
– Quasi-experimental, longitudinal study
– 25 states and 78 school districts
– Each district is included only in the longevity duration class to which it
belongs
Shaha SH, Glassett K, Ellsworth H (2015). Long-term Impact of On-demand Professional Development on Student Performance: A Longitudinal Multi-State Study.
Journal of International Education Research. Vol 11(1).
Glassett, K. & Shaha, S. (2014). Long-term Impact of On-demand PD on Student Performance: A Longitudinal Multi-State Study. In Proceedings of World Conference on
E-Learning in Corporate, Government, Healthcare, and Higher Education 2014 (pp. 668-671). Chesapeake, VA: AACE.
Shaha, S., Glassett, K., & Ellsworth, H. (2014). Long-term Impact of On-demand PD on Student Performance: A Longitudinal Multi-State Study. 2014 International
Education Conference. San Francisco, CA: The Clute Institute. August 2014.
7-Year Longevity
Longitudinal Effect on Change in Student Performance
Versus Pre-PD Baselines
14.0
Net Change in Pct Adv+Prof
12.0
10.0
8.0
2-3 years
6.0
4-5 years
6-7 years
4.0
2.0
0.0
Change in Reading Perf
Reading gains from Baseline:
2-3 years: 5.58 net gain in pct Adv+Prof
4-5 years: 7.34 net gain in pct Adv+Prof (p=0.042)
6-7 years: 12.77 net gain in pct Adv+Prof (p=0.037)
Change in Math Perf
MATH gains from Baseline:
2-3 years: 2.90 net gain in pct Adv+Prof
4-5 years: 6.92 net gain in pct Adv+Prof (p=0.031)
6-7 years: 12.15 net gain in pct Adv+Prof (p=0.009)
Shaha SH, Glassett K, Ellsworth H (2015). Long-term Impact of On-demand Professional Development on Student Performance: A Longitudinal Multi-State Study.
Journal of International Education Research. Vol 11(1).
Executive Summary
PD360 Schools versus their Districts, Year-over-Year
–
–
Math:
•
•
•
11.1 Net gain in Math vs. 2.6 for District peers
That’s 18.9% better than Pre vs. 4.2% better for District peers
That’s 4.5 times more improvement than their peer Districts
Reading:
•
•
•
10.3 Net gain in Math vs. 1.5 for District peers
That’s 18.1% better than Pre vs. 2.5% better for District peers
That’s 7.4 times more improvement than their peer Districts
Predictors of Success
–
–
Math :
•
•
•
13.16 Net gain in for High Observation schools versus 2.22 net gain for Lower Observation Schools
That’s 24.9% improvement with High Observation & PD360 baseline versus 3.9% improvement for Lower Obs w/PD360
That’s 6.4 times more improvement for Higher Observation versus Lower
Reading:
•
•
•
12.47 Net gain in for High Observation schools versus 2.21 net gain for Lower Observation Schools
That’s 24.6% improvement with High Observation & PD360 baseline versus 3.5% improvement for Lower Obs w/PD360
That’s 7.0 times more improvement for Higher Observation versus Lower
2-Year Sustained Advantage versus their Districts
–
–
Math :
•
•
16.4 Net gain in Math from baseline – that’s 13.5 more net gain than District peers
That’s 28.1% better than baseline, which is 4.6 times more improvement than their peer Districts
Reading:
•
•
17.2 Net gain in Math from baseline – that’s 13.5 more net gain than District peers (matches Math)
That’s 30.2% better than baseline, which is 5.0 times more improvement than their peer Districts
7-Year Longevity: into 6th and 7th Years
–
–
Math net gains:
•
•
•
5.58 for educational organizations with 2-3 years of PD
7.34 for orgs with 4-5 PD years
12.77 for orgs with 6-7 PD years
Reading net gains:
•
•
•
2.90 for educational organizations with 2-3 years of PD
6.92 for orgs with 4-5 PD years
12.15 for orgs with 6-7 PD years
13
Executive Summary
Next Studies
• Aligned Observation
• Aligned Seminars
• Aligned Implementation
4. Advantages with Longevity into
6th and 7th Years
3. Advantages Sustained into
Second Year of PD 360
2. Prdictors of Success
High vs. Lower Engagement
1. Advantage with PD 360
versus none, Year-over-Year
14
Soon to be Published
Observation Integrated
– Math :
• 13.16 Net gain in for High Observation schools
– versus 2.22 net gain for Lower Observation Schools
• 24.9% improvement with High Observation & PD360 baseline
– versus 3.9% improvement for Lower Obs w/PD360
• That’s 6.4 times more improvement for Higher Observation versus Lower
– Reading:
• 12.47 Net gain in for High Observation schools
– versus 2.21 net gain for Lower Observation Schools
• 24.6% improvement with High Observation & PD360 baseline
– versus 3.5% improvement for Lower Obs w/PD360
• That’s 7.0 times more improvement for Higher Observation versus Lower
15
Key Impacts:
Immediate Impacts
2-Year Sustained Impacts
7-Year Longevity
Predictors of Success
Prepared by: Steven H. Shaha, PhD, DBA
16