Transcript Evaluating the Effectiveness of

Online Professional Development for Transition:
Evaluating the Effectiveness
Mary E. Morningstar, PhD – [email protected]
Dana L. Lattin, MSEd – [email protected]
Amy Gaumer Erickson, PhD – [email protected]
Ryan Kellems, MEd – [email protected]
October 18, 2007
DCDT International Conference – Orlando, FL
First, a little background….
2004
1980’s
- 1960’s
Present
1990’s
2000-2004
1970’s
Late
Federal
Kansas
Systems
Change
National
study
of
high
school
Interprofessional/Transition
KU
TransCert:
Online
Transition

grant
for
secondary
Department
of
Special
Education
Project
- Statewide
Transition
programs,
1985-88
Personnel
Preparation,
2000-03
Graduate
Certificate
Program
teacher
education
personnel
commitment to secondary special
Training
1993-1998
2004-2008
 preparation
1985
Department
ofdevelopment
Defense
model,
1972-76
education
and
career

National
Transition
Outreach
Dependents’
Kansas
Transition
Network
Schools
Summer –of
Evaluating
the
Effectiveness
starts
in
late
1960s
Project
(National
Transition
contract
with
KSDE
1998-2000
Institute
on
career/
vocational
 Federal
RETOOL
project
for
Online
Professional
Coalition),
2001-2004
 higher
KU Secondary/Transition
Summer
transition
workshops
education
on late
Development
forfaculty
Transition
Personnel
Preparation
1980s
career
education
for students
2004-2007
 National Online Specialization in
Program
1994-1999
with
Secondary
Special
Education/
disabilities,
1975-79
Transition
Project,
2001-2004
KU Secondary
Transition
 Transition
KU Secondary/Transition
personnel preparation
Leadership
Program
2005-2009
Outreach
project
1997-2000.
grant,
1989-91
Late
1960’s
1970’s
1980’s
1990’s
2000-2004
2004-?
Noninstructor-led Online
Professional Development
Instructor-led Online
Professional Development
Graduate
Coursework
KU TransCert
Online
Program
KU-SET
Doctoral
Program
Professional
Development
•
•
•
•
Online Modules
KU Summer
Institute
SEAs
Online training
Face to face
State Needs Assess
COPs
Continuing Ed. Online Classes
• Overview of Transition
•
•
•
•
Transition Assessment
Interagency Collaboration
Employment
Student & Family Involvement
Online Resources
Searchable Databases
Models of Success
Non-Instructor Led Online
Professional Development
Online Training
• User friendly and
interactive information
Online Resources
• Making sense of
complex information
Searchable Databases
• Applying information
Models of Success
• Connect & network
Online Training Modules
Currently Available
• Working with Families
• Transition Assessment:
The Big Picture
• Best Practices in
Transition and IDEA 2004
• Cultural Diversity and
Transition
• Students with ED/BD and
Employment
In Design Phase
• Aligning Transition IEPs
with Standards
• Self-Determination
• SBR & Transition
• Students with Significant
Disabilities & PBS
Online Module Framework
o Learning Objectives
o Case-Based Examples
& Activities
o Conceptual
Models/Unifying
Frameworks
o Important Point
o Now It’s Your Turn
o More Information & My
Library
o Interactivity: Click &
Compare; True/False;
Games; Audio/video
o Session Summary
User
Portfolio
General
Content
oPre/Post Test
oLearning activities
oReflective questions
•Impact on teaching
•Barriers to implementation
•Reflection of general
learning content
Learning
Objectives
Case-based Examples
&
Activities
Conceptual Models &
Unifying Frameworks
Important Point
Now It’s Your Turn
More Information &
My Library
True/False or
Myth/Reality
Click & Compare
Games
Audio/Video
Session Summary
Demographic Data of All Users
• 2405 practitioners
– 68% educators (sped, gen ed, transition, related services,
administrators)
~ 50% serving secondary-aged students
– 18% College/university students
• Types of disability groups served
– 30% LD; 18% multiple groups; 5% MR; 4% ED/BD; 2% autism;
5% SMD; 25% Programmatic
– 57% regular schools; 7% special schools; 6.5% community
agencies
• Level of Education
– 44% Masters/specialist; 44% bachelors; 4% doctorates; 3%
associates
March, 2007
Educators
• Certification Status
– 43% fully certified for current teaching assignment;
3% emergency certif.; 7% certif. in field other than
teaching; 8% provisional; 17% NA
• Years Teaching
– 36% 10+; 8% 7-9yrs.; 15% 4-6yrs; 20% 1-3yrs;
9% student teaching; 11% not teaching
• Transition Training & Information Sources
– 46% inservice training; 26% transition course; 6%
2+ courses; 9% infused content; 17%
conferences; 17% newsletters/online; 43% on-thejob training
March, 2007
Satisfaction with Training Modules
March, 2007
Families
Best
Practices
Assessment
Kept attention
86% SA or A
80%
89%
Helped dev. Understand of
content
95% SA or A
93%
92%
Contributed to Learning:
- Case-based activities
- Interactive examples
- Important points
- text and written information
93% SA or A
87% SA or A
95% SA or A
94%
NA
NA
91%
91%
NA
NA
91%
Training was of high quality
81% SA or A
91%
89%
Important resource for the
future
92% SA or A
78%
87%
Effective way to offer inservice
training
85% SA or A
84%
91%
Qualitative Data Results: Online Training Modules
PRO’s
• Ease of use
–
–
–
–
Graphics
Navigation
Flexibility
Portfolio
• Information was current,
useful & to the point
• Examples and
interactive case studies
•
•
•
•
•
•
CON’s
Pre/Post Test
Limited information
Portfolio
Printer-friendly
version
More reflective
practice and
application
Links not working
Evaluation of Online Modules: Knowledge Gain
Repeated measures design - Paired Sample t-test
Working with Families during Transition
N= 282; Post-test (M=11.51, SD 2.77) significantly
greater than pre-test (M=13.90, SD 2.86); t(164) = 10.85 p<.00, d=.84
Best Practices in Transition Planning
N= 282; Post-test (M=11.51, SD 2.77) significantly greater than pretest (M=13.90, SD 2.86); t(164) = -10.85 p<.00, d=.54
Transition Assessment
N= 282; Post-test (M=11.51, SD 2.77) significantly greater than pretest (M=13.90, SD 2.86); t(124) = -14.318 p<.00, d=1.28
Random Sample-Control Group Study:
2 Online Modules
Kim (2006). Kim & Morningstar (2006).
• Working with Families & Working with CLD
Families and Youth Modules
• Control group (43; no intervention) &
Intervention group (43; A-B; B-A)
• Content Knowledge test (pre/post)
• Competencies (level of preparation) +
Attitudes (level of importance) Survey (.79.94 reliability)
Results: Knowledge & Competency Gains
• 28/42 of control group completed pre & post tests
• 25/42 of intervention group completed study
• Comparison of groups
– Chi Square tests = no significant difference
among 2 groups for years teaching, types of
students, training, certification, sources of training
in transition
– Content Knowledge: Experimental group
showed significant changes in knowledge &
control group showed no change
– Perceived competencies: WWF: significant
difference; WCLD: control & experimental groups
showed no change
– Perceived importance (Attitudes): no change
pre/post both; WWF’s between groups ANCOVA
on posttest was significant for control
Online Professional Development Evaluation
Recruitment &
Sampling
Control Group
Training + CoP
Training Only
15-20 practitioners
No training
No online CoP
15-20 practitioners
2 training modules
15-20 practitioners
2 training modules
Online CoP
Comparisons
• Knowledge
 Competencies & skills
 Frequency of implementation
Access to ALL
online training
upon completion
of the study
•



Mixed Measures
Pre-post test
Competencies surveys
Performance-based content analysis
Qualitative data
Access to ALL
online training
modules upon
completion of the
study
Recruitment of Participants
• Initial recruitment in 3 states (CO, ID, AZ)
• Contacted SDE staff at other states to find other
participants
• NEW box on TC website – available to anyone
• Online agreement to participate
• Blocked from all training modules until research
started
• Needed 150 participants with min. 90 completing
research – recruited 156
Procedures
• Communication with participants was through email
• Participants in all groups (C, T, TCoP) were asked to
begin research at the same time.
• Participants received email reminders and prompts to
complete the research and were given approximate
times for how long it would take to complete each step.
• All participants received incentives for completion of
research (honorarium or graduate credit) and for
completing various steps of the research within specific
timelines (jump drives, bonuses, transition books)
• Phone support was provided to those participants
needing intensive technical support using the website.
Participant Demographics
94 Completed Research (C=35; T=30; TCoP=29)
• 81 (86%) Educators
8 (9%) School Related Services Providers
• 5 (5%) Other
– Community Service providers, Family members, Administrators
• 88 (94%) Serve secondary aged students
• Types of disability groups served
– Across Multiple Disabilities (55%); High Incidence (33%)
Low Incidence (7%); None (4%)
• Level of Education
– 64% Masters/specialist; 30% bachelors; 4% doctorate; 1% associates;
1% HS Diploma
Educators
• Educators Only - Certification Status
– 75% fully certified for current teaching assignment; 6%
provisional/emergency certif.; 3% certif. in field other than
teaching; 16% NA
• Educators Only - Years Teaching
– 64% 10+; 6% 7-9yrs.; 16% 4-6yrs; 13% 1-3yrs; 1% not
teaching
• Educators Only - Transition Training &
Information Sources
– 75% inservice training; 72% conferences; 64%
newsletters/articles/books; 62% on-the-job training; 48%
online; 20% professional associations; 9% 1+ transition
course; 13% infused content
Measures
•
•
•
•
•
STTS (pre/post)
Best Practices pre/post tests
Transition Assessment pre/post tests
Satisfaction Surveys for each module
Performance-based Assessment of 3 specific
activities within each module (total of 6 activities
across both modules)
DOMAINS
Instructional Planning
Curriculum and Instructio
Transition Planning
Assessment
Collaboration
Additional Competencie
RELIABILITY TESTS:
557 teachers nationally
Alpha scores:
.96 Preparation subscal
.94 Frequency subscale
Control Group
1
Pre STTS
Best Practices
Pre test
2
Transition Assessment
Pre test
Best Practices
Post test
Transition Assessment
Post test
Post STTS
Training Group
Pre STTS
Best Practices
1. Pre test
2. Complete Module
and all Activities
3. Post test
4. Satisfaction Survey
Transition Assessment
1. Pre test
2. Complete Module and
all Activities
3. Post test
4. Satisfaction Survey
Post STTS
Training + CoP Group
Pre STTS
Best Practices
1. Pre test
2. Complete
Module and all
Activities
3. Post test
4. Satisfaction
Survey
Participate in
Community
of Practice
Transition Assessment
1. Pre test
2. Complete Module and
all Activities
3. Post test
4. Satisfaction Survey
Post STTS
Performance-based Assessment
• Developed, validated, user-tested, modified
rubrics for 6 activities
• 3 reviewers met to reach consensus on scoring
methods
• Compared scores on sample
• Scored all to determine Inter-rater Reliability
– Range 75%-90%
– Average 86%
• Mean Scores
• Comparisons between T & T+CoP groups
Results: Knowledge Gained
• Measure: pre/post tests for modules
– Best Practice Module
• Statistically significant gain in knowledge for both training
groups; high effect size
• Statistically significant difference in knowledge on post-tests
between control and training groups, accounting for 35% of
the variance in scores
– Assessment Module
• Statistically significant gain in knowledge for both training
groups; high effect size
• Statistically significant difference in knowledge on post-tests
between control and training groups, accounting for 8% of
the variance in scores
Results: Satisfaction with Training
Kept attention
91.2% SA or A
94.8%
Helped dev. Understand of
content
96.4% SA or A
94.8%
Contributed to Learning:
- Case-based activities
- Interactive examples
- Practice activities
- text and written information
93.1% SA or A
86.2% SA or A
93.1% SA or A
91.3% SA or A
100%
91.4%
94.8%
96.6%
Training was of high quality
94.7% SA or A
96.5%
Relevant to my job
98.2% SA or A
98.3%
Important resource for the
future
96.5% SA or A
96.6%
Effective way to offer
inservice training
94.8% SA or A
96.5%
Results: Perceived Skill & Frequency
• Measure: STTS pre/post tests
– Statistically significant increase in perceived
skills for training groups
– CoP group showed more realistic perceptions
of their skills and frequency on the post-test
Current & Future Evaluation Efforts:
Personnel Preparation
Increased
Transition
Knowledge
Improved
Transition
Planning
Participant
satisfaction
• Competency Surveys
• PBA
• QI Program Evaluation
• Goal Attainment Scaling
• Focus groups
• Content Analysis of
Discussions
Increased
communication
& linkages
Attention to
teaching transition
content