Transcript How to implement EBPs

Piloting Evaluation Systems:
What Implementation
Research Reveals for
Effective Design
CCSSO
National
Summit on
Educator
Effectiveness
Colleen Mileham
Oregon Department of Education
Barbara Sims
University of North Carolina at Chapel Hill
Seattle WA
May 2012
Welcome and Introductions



Deb Hansen, Senior Policy Analyst, West
Wind Education Policy Inc., and Consultant,
CCSSO/SCEE
Colleen Mileham, Assistant
Superintendent, Oregon Department of
Education
Barbara Sims, Associate Director, National
Implementation Research Network
SISEP 2012
Objectives


Describe the purposeful application of
implementation science to scaling effective
innovations in education
Explore the use of Active Implementation
Frameworks for designing and testing Educator
Evaluation Systems
 Implementation Teams
 Usability Testing
 Transformation Zones
Flexibility Review Guidance

3B: Is the SEA’s process for ensuring that each LEA
develops, adopts, pilots, and implements, with the
involvement of teachers and principals, evaluation and
support systems consistent with the SEA’s adopted
guidelines likely to lead to high-quality local teacher
and principal evaluation and support systems?

SISEP 2012
Is the SEA’s plan likely to be successful in ensuring
that LEAs meet the timeline requirements by either (1)
piloting evaluation and support systems no later than
the 20132014 school year and implementing
evaluation and support systems consistent with the
requirements described above no later than the
20142015 school year;
Flexibility Guidance Review


Is the SEA plan for providing adequate
guidance and other technical assistance to
LEAs in developing and implementing teacher
and principal evaluation and support systems
likely to lead to successful implementation?
Is the pilot broad enough to gain sufficient
feedback from a variety of types of educators,
schools, and classrooms to inform full
implementation of the LEA’s evaluation and
support systems?
SISEP 2012
Check-In

How many of your states are planning a pilot of
your evaluation system this fall—or will be
having districts planning their own pilots?
SISEP 2012
Developing the Capacity
to Implement Well
“A serious deficiency is the
lack of expertise to
implement best practices and
innovations effectively and
efficiently to improve student
outcomes.”
Rhim, Kowal, Hassel, & Hassel (2007)
The Challenge

Science to Service Gap
 What is known is not what is adopted to help
students

Implementation Gap
 What is adopted is not used with fidelity and good
outcomes
 What is used with fidelity is not sustained for a
useful period of time
 What is used with fidelity is not used on a scale
sufficient to broadly impact student outcomes
SISEP 2012
Implementation Science
“In theory there is no difference
between theory and practice;
in practice there is.”
variously attributed to
Jan La Van De Snepscheut
or Albert Einstein
or Yogi Berra
SISEP 2012
Implementation Science

Implementation science is the scientific study
of variables and conditions that impact changes
at practice, organization, and systems levels;
changes that are required to promote the
systematic uptake, sustainability and effective
use of evidence-based programs and practices
in typical service and social settings.
~Blase and Fixsen, 2010
National Implementation Research Network
Formula for Success
Effective Intervention practices
X
Effective Implementation practices
=
Effective Outcomes
Shifting Accountability
Student
SISEP 2012
Practitioner
System
Making It Happen

Letting it happen
 Recipients are accountable

Helping it happen
 Recipients are accountable

Making it happen
 Purposeful use of implementation practice and
science
 Implementation system is accountable
Based on Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou, 2004
SISEP 2012
Active Implementation
Frameworks
Implementation Teams
Implementation Stages
Implementation Drivers
Improvement Cycles
IMPLEMENTATION TEAMS
Organized, expert assistance to develop
and sustain an accountable structure
“Who” Makes It Happen?
Organized, “Expert” Assistance:


Purveyors
Implementation Teams
Implementation Teams



Provide accountable structure to move intervention through
stages of implementation
Scope of the initiative determines the development of linked
Implementation Teams and communication protocols
Focus is on
 Ongoing “buy-in” and readiness
 Installing and sustaining the Implementation Drivers
 Fidelity & Outcomes
 Systems Alignment and Stage-based work
 Problem-solving and sustainability
Implementation Team
IMPLEMENTATION
INTERVENTION
Impl. Team
Effective
NO Impl. Team
80%, 3 Yrs
14%, 17 Yrs
Making it Happen
Letting it Happen
Helping it Happen
Fixsen, Blase,
Timbers, & Wolf, 2001
Balas & Boren, 2000
Green & Seifert, 2005
Linked Team Structures
State-based
Implementation
Team
Regionally-based
Implementation
Team
District-based
Implementation
Team
School-based
Implementation
Team
“We tend to focus on snapshots of isolated
parts of the system and wonder why our
deepest problems never seem to get solved.
(Senge, 1990)
Reflection
Implementation Teams
•
Supporting
New Ways of Work
•
•
•
In your experience, who
supports new processes?
How is the transition made
from external expertise to
building internal capacity?
What role will your SEA have
in the implementation of your
evaluation systems this fall?
Could you create a State
Implementation Team?
Would you be able to support
implementation teams at
several levels of the system?
Effective Implementation
Implementation Team members
make effective use of:
 Implementation Stages
 Implementation Drivers
 Improvement Cycles
STAGES OF IMPLEMENTATION
Purposeful matching of critical
implementation activities to the stage of the
process
SISEP 2012
Implementation Stages
2 - 4 Years
Exploration
Installation
Assess needs
Examine
Acquire resources
Prepare
innovations
Examine
Implementation
Assess fit
organization
Prepare
implementation
Prepare staff
Initial
Implementation
Implementation
drivers
Manage change
Data systems
Improvement
cycles
Full Implementation
Implementation
drivers
Implementation
outcomes
Innovation outcomes
Standard practice
What to Choose:
Careful Assessment and Selection






What is our theory of change?
What would be a good fit for this state?
What is the strength of the evidence?
What is the effect size?
Do we have what it takes to fully and
effectively implement?
How will we ensure sustainability?
Assessing Readiness:
EBPs and Implementation
Need in Education Setting
Socially Significant Issues
Parent & Community Perceptions
of Need
Data indicating Need
Need
Fit with current -
Capacity
Staff meet minimum qualifications
Able to sustain Imp Drivers
• Financially
• Structurally
Buy-in process operationalized
• Practitioners
• Families
• Agency and Departments
•Initiatives
• Educational Priorities
• Organizational structures
• Community Values
Fit
Capacity to Implement
Readiness
Qualified purveyor
Expert or TA available
Mature sites to observe
# of replications
How well is it operationalized?
Are Imp Drivers operationalized?
Resource
Availability
Intervention Readiness
for Replication
EBP:
5 Point Rating Scale: High = 5; Medium =
3; Low = 1. Midpoints can be used and
scored as a 2 or 4.
High
Medium
Low
Need
Evidence
Fit
Resources Availability
Resources
Staffing
Training
Data Systems
Coaching & Supervision
Administrative & system
Supports needed
Time
Evidence
Outcomes – Is it worth it?
Fidelity data
Cost – effectiveness data
Number of studies
Population similarities
Diverse cultural groups
Efficacy or Effectiveness
Evidence
© National Implementation Research
Network 2009
Readiness for Replication
Capacity to Implement
Total Score:
Adapted from work by Laurel J. Kiser, Michelle Zabel,
Albert A. Zachik, and Joan Smith at the University of Maryland
IMPLEMENTATION DRIVERS
Common features of successful
supports to help make full and
effective uses of a wide variety of
innovations
Why:
What:
Positive Outcomes for Students
Effective Educational Practices
Staff capacity to
support
children/families
with the selected
practices
Professional
How:
Development/
Professional
Learning
Institutional capacity to
support teachers & staff
in implementing
practices with fidelity
Core
Implementation
Components
Leadership
Capacity to provide direction
and vision
© Fixsen & Blase, 2008
The Infrastructure for Change
Implementation Drivers

Help to develop, improve, and sustain educators’
competence and confidence to implement effective
educational practices and supports.

Help ensure sustainability and improvement at the
organization and systems level

Help guide leaders to use the right leadership
strategies for the situation
Benefits of Driver-Based Action
Planning




Infrastructure needed becomes visible to all
Strengths and progress get celebrated
Next right steps are planned and results
measured
Resources can be aligned and re-purposed to
improve implementation
IMPROVEMENT CYCLES
Changing on purpose to support the new
way of work
SISEP 2012
Types of Improvement Cycles
Plan-Do-Study-Act Cycles
 Rapid cycle problem solving (Shewhart;
Deming)
 Usability testing (Neilson; Rubin)
 Practice-policy communication loops
Rapid Cycle Problem Solving
SISEP 2012
Usability Testing
SISEP 2012
Usability vs. Pilot Testing
Usability
 Clear description of the
program



SISEP 2012
Pilot
 Clear description of the
program
Trial and learning
approach

Small number of
participants with multiple
iterations

Rapid problem solving
applied to program and
system
Trial and learning
approach
Sufficient number of
participants and
sufficient time to realize
potential results
Transformation Zone

A “vertical slice” of the service system
(from the classroom to the Capitol)
 The “slice” is small enough to be
manageable
 The “slice” is large enough to include all
aspects of the system
 The “slice” is large enough to “disturb the
system” – a “ghost” system won’t work.
SISEP 2012
Implementation
Team
Teachers
Innovations
Students
Practice- Policy
Communication Cycle
System
Change
State
Management
Team
Policy Supports
Effective Practice
SISEP System Change Support
System Alignment
Reflection
Supporting
New Ways of Work
Building Feedback Loops
Into Evaluation System
“Pilots”
•
•
•
Which districts will pilot your
system? Which pilots will
your state (implementation)
team be involved in?
What will be the role of your
implementation team?
What might the work look
like? How often would you
meet? What would you be
meeting about?
Oregon
Scaling–Up
Oregon
State Leadership
New Way of Work
May 2012
Oregon
Department
Of
Education
IMPLEMENTATION SCIENCE
LEADERSHIP ROLE
New Way of Work
Communication
Organization of Work
School Districts
Stakeholders
Higher Education
Initiatives
Innovations
State
Management
Team
Oregon
Department
Of
Education
LEADERSHIP FUNCTIONS
Implementation Science – What Oregon is Learning




Massive undertaking
Mechanics of operation aligned to best practices
Need relevant data to support plan
Must invest resources (time, funds) to achieve
high quality implementation statewide

Practice to scale – requires understanding
drivers

Pilot – start with end in mind
Oregon
Department
Of
Education
LEADERSHIP FUNCTIONS
Practice Informs Policy

Statewide feedback brings practice to reality

What works – what supports implementation

What does not work – what hinders
implementation
Oregon
Department
Of
Education
LEADERSHIP FUNCTIONS
Structure of Oregon’s Feedback Loop

Implementation Zones

Regional Implementation Team

State Management Team
Oregon
Department
Of
Education
LEADERSHIP OUTCOMES

Focused message

Framework not cookie cutter

Data based decisions

Scale-worthy initiatives

Statewide communication/network
Policy
Practice
Feedback Loops
Policy
Policy Enabled Practices
(PEP)
Feedback
Practice Informed Policy
(PIP)
Study - Act
Expert Implementation Support
Policy (Plan)
Structure
Procedure
Practice (Do)
FORM SUPPORTS FUNCTION
Practice
Reflection
Supporting
New Ways of Work
Building Feedback Loops
Into Evaluation System
“Pilots”
•
•
•
How can we develop direct
communication between
levels in the pilot?
How can we communicate
between the pilot(s) and the
rest of the state/districts/
schools?
How can we develop direct
communication between
levels when we go to scale?
GENERAL DISCUSSION
& QUESTIONS
Stay
Connected!
www.scalingup.org
@SISEPcenter
SISEP
For more on Implementation Science
http://nirn.fpg.unc.edu
www.implementationconference.org
Implementation Science
Implementation
Research:
A Synthesis of
the Literature
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005).
Implementation Research: A Synthesis of the Literature. Tampa, FL: University of
South Florida, Louis de la Parte Florida Mental Health Institute, The National
Implementation Research Network (FMHI Publication #231).
HTTP://NIRN.FPG.UNC.EDU