Transcript Implementation Research - Welcome to Gibson & Associates

Factors Contributing to Successful
Implementation of EBPs
 The success of the intervention strategies
themselves and
 The success of the implementation processes
Too often, evaluations focus more on changes in
client outcomes without due consideration of
fidelity to the intervention model or the
effectiveness of the implementation process itself.
2
Implementation Defined
 Implementation is a specified set of
activities designed to put into practice an
activity or program of known dimensions.
 The National Implementation Research
Network at the University of Southern
Florida conducted an exhaustive review
of research on implementation and is the
primary source for this presentation.
3
A Conceptual Framework for Understanding
Implementation of Evidence-Based Practices
The essential implementation outcomes
are:
 Changes in adult professional behavior
 Changes in organizational structures and
cultures
 Changes in relationships to consumers,
stakeholders, and system partners
4
Implementation Framework
5
Stages of Implementation
 Exploration & Adoption
 Program Installation
 Initial Implementation
 Full Operation
 Innovation
 Sustainability
6
Stages of Implementation
Exploration & Adoption
 Identify the need for an intervention
considering existing conditions
 Assess the fit between the intervention and
program and community needs
 Prepare organization, staff, and resources for
mobilizing information and support.
This was essentially accomplished in the work of the Work Groups.
7
Program Installation
Program installation begins after an
adoption has occurred and involves:
 Ensuring availability of funding
 Human resource strategies (recruitment, hiring,
training, etc.)
 Policy development to support change (e.g. referral
mechanisms, reporting frameworks and outcome
expectations)
 Facility requirements
8
Initial Implementation
 Initial implementation begins once
conditions identified in the installation
phase have been sufficiently completed.
 Initial implementation must NOT be
confused with full operation, as during
initial implementation, many factors
contributing to fidelity to the model may
not be fully or even partially in place.
9
Barriers to Full Operation
Organizational Barriers
 Personnel rules
 Social stressors
 Union stewards
 Anxious administrators
 Political pressures
 Interpersonal rivalries
 Staff turnover
10
Barriers to Full Operation
Human Barriers
 Fear of change
 Inertia
 Investment in the status quo
 Inherently difficult work of doing
something differently
11
Full Operation
Full Operation ONLY occurs when:
 New learning becomes integrated into practitioner,
organizational and community practices, policies, and
procedures.
 Full case loads
 Full staffing
 Fidelity to the model being implemented
Only once fidelity measures are above criterion levels most of the
time, can the effectiveness of an evidence-based practice be
evaluated. This is a very important point.
12
Dangers of Premature Evaluation
“
Outcome evaluations should not be
attempted until well after quality and
participation have been maximized and
documented in a process evaluation.
Although outcome data can determine
the effectiveness of a program, process
data determine if a program exists at all.
Gilliam, Ripple, Zigler & Leiter
(2000).
”
13
Innovation or Drift
When evidence-based practice meets new
local conditions. Two results can occur:
 Innovation: When desirable changes in the
standard model are identified.
 Drift: Undesirable changes in the standard
model.
It is critical to implement the model with fidelity before
considering innovation and to make changes only after
due consideration of client benefit.
14
Innovation or Drift Illustrated
15
Sustainability
Throughout the two to four-year implementation
process, the forces that led to and supported reform
will change:
 Staff and funding changes
 New social problems emerge
 Partnerships and political alliances change
 Champions and advocates move on to other
causes
Throughout the implementation process, it is essential
that leadership maintain focus on sustaining the core
components of reform.
16
Factors Contributing to Effective
Implementation of EBP

Objective decision-making strategies that involved staff, good
information about the reform and organizational leadership
support during the exploration stage;

Evidence of a ‘learning culture’ within the organization
implementing the reform;

A system in place for monitoring implementation;

Access to technical assistance throughout implementation;

The perceived ability of the organization to manage risks;

Belief in the validity of the reform; and most of all….

Adherence to the core components of BOTH the EBP and
effective implementation (discussion follows).
17
Core Intervention Components
An essential first step to implementing EBP or system
reform is to identify what are the core components of the
intervention itself. Leadership must:
 Carefully research alternatives
 Consider well-evaluated experiential learning from a number
of replications and
 Achieve a clear understanding of what of the model must be
maintained to achieve fidelity and effectiveness at the
consumer level.
In other words, before considering the core components of
any intervention, leadership must be clear about what is to
be implemented.
18
Core Components for Implementation
The following Implementation Drivers influence staff
behavior and organizational culture:
 Practitioner Selection
 Preservice and inservice training
 A consultant coach
 Staff & program evaluation
 Facilitative administration and
 System Interventions
This is illustrated in the graphic that follows.
19
Implementation Drivers
20
Implementation Drivers
These drivers are integrated in their
influence, in that strengths in one area
can compensate for weaknesses in other
areas.
The relative effectiveness of the
implementation drivers is as important as
the demonstrated effectiveness of the
EBP itself.
21
Practitioner Selection
How and who are selected to carry out the
EBP
 Who is qualified to carry out the practices and
programs?
 What are the methods for recruiting & selecting
practitioners?
 How important are individuals skills,
experience and personal characteristics to
effective implementation of EBP?
22
Research on Practitioner Selection
There has been relatively little research to isolate the factors
involved in selection of practitioners for EBP. Factors
identified in effective practitioner selection for a national
MST program included:



Responses to behavioral vignettes;
Responses to role playing situations related to the therapeutic;
environment for which they were being considered; and
Responses to mini-training that requires behavior change.
Responses are rated with a rubric along several dimensions
including: collaborative and strength focused, efforts to overcome
barriers, ability to use behavioral language, uses of logical thinking
and openness to feedback.
23
Research on Practitioner Selection
Morris & Stuart, 2002 are attempting to identify the generic skills
needed by frontline practitioners in a transformed behavioral
health field. Among those qualities considered:







Assessment skills;
Family and support system involvement;
Social and cultural engagement skills;
Treatment skills;
Methods to optimize recovery and empowerment;
Consumer relationship skills; and
Community resource and coordination skills.
It is interesting to note here and throughout this research,
cultural competence was never discussed as an issue or
factor.
24
Research on Staff Selection
A Housing & Urban Development study by
Wanberg & Banas (2000) examined
practitioner characteristics in the context of
organizational change and found that personal
resilience and self-efficacy were associated
with greater acceptance of change in the work
place.
This study would seem particularly important in
SMCMH’s system transformation.
25
Research on Staff Selection
From the research we might conclude that as
part of the implementation planning process,
SMCMH leadership should:
 Construct a rubric of ideal staff characteristics
particularly as relates to operating in a transformed
organization
 Design an interview process that includes vignettes
and role plays that require candidates to demonstrate
those characteristics
26
Preservice and Inservice Training
Training is an efficient way to provide:
 Background knowledge, background information,
theory, philosophy, and values;
 Demonstration of new skills (through video or role
play);
 Opportunities to practice new skills through role plays
and behavioral rehearsals; and
 Feedback in a safe training environment.
27
Preservice and inservice training
Role playing & behavior rehearsals are
critical for practicing new skills in training.
 Role playing asks you to pretend you are someone
else and try this… which builds empathy
 Behavioral rehearsals asks you to be in your
practitioner role and you are asked to confront a
specific situation and perform your practitioner role in
reaction to that situation ….which serves as direct
preparation for the real thing.
28
Recommendations for Training

Model or demonstrate new skills using role play, behavior
rehearsals and video tape.

Emphasize practice and use feedback on practice to teach the
finer points of mapping.

Use practice sessions to help trainees integrate thinking and
doing.

Provide guidance with respect to the boundaries of using the
technique, describing when it may be useful and when it may not
be useful.

Provide guidance on the flexible use of the core components.

Encourage peer and administrative support.
29
Limitations of Training
Numerous evaluations on the impact of training in
health and human services have found that training
alone rarely impacts practice.

The “train-and-hope” approach (Stokes & Baer, 1977) to
implementation does not appear to work.

Kelly et al (2000) in a study of HIV service organizations reported
the largest increase in adoptions of HIV service guidelines
occurred when consultation was added to training.

A meta-analysis (Davis, 1995) found similar results in medicine.
Davis concluded that “formal CME conferences and activities
without enabling or practice reinforcing strategies, had little
impact.” (page 700)
30
Limitations of Training: Spray & Pray
While training may introduce knowledge,
philosophy, and new approaches, there is no
research study that has demonstrated
significant changes in practice resulting from
an intervention that provided only training.
The limitations of training are directly related to
the way in which adult learning occurs.
31
Stages of Adult Learning
Generally adult learning progresses in stages:
 Orientation & new learning
 Mechanical use
 Routine use
 Refinement
 Integration
 Innovation
32
Challenges of Learning New Behaviors
Based upon decades of research, Joyce
& Showers (2002) concluded that
 The newly-learned behavior is crude (i.e. mechanical)
compared to performance by a master practitioner.
 Newly-learned behavior is fragile and needs to be
supported in the face of reactions from consumers and
others in the service setting.
 Newly-learned behavior is incomplete and will need to
be shaped to be most functional in a service setting.
33
The Importance of Coaching
While most skills can be introduced in training,
newly-learned behaviors are only really
learned on the job with help of a consultant or
coach precisely because:
 The challenges of adopting new behaviors can’t be
replicated in a training.
 Single-point-in-time training needs to be reinforced
continuously in the work setting for behavior
changes to be sustained.
34
Consultant or Coach Support
 Implementation of EBP requires changes in behavior at
the practitioner, supervisory and administrative levels.
 Training and coaching are the primary strategies in
“
which behavior change is brought about.
In addition to training, substantial hands-on coaching
and practice may be necessary before a counselor
feels comfortable with a new strategy.
~ Dansereau & Dess (2002)
”
35
Components of Effective Coaching
Coaching must be work-based, opportunistic,
readily available, and reflective (Spouse 2001).
Spouse described four roles for the coach:
 Supervision
 Teaching while engaged in practice activities
 Assessment and feedback
 Provision of emotional support.
36
Factors that Impact Coaching
 Amount of time devoted to coaching (MST coaching
occurs once or twice a week for 90 minutes)
(Schoenwald et al., 2000)
 Includes direct observation of provision of direct
services (Smart et al., 1979)
 Utilizes coaches who are expert in the content,
techniques, and rationales of the program (Denton,
Vaughn & Fletcher 2003)
 Coaching relationships established during the
training experience (Smart et al., 1979)
37
Other Factors Contributing to Effective
Coaching
Walker, Koroloff & Schutte (2002) identified four
additional factors that accounted for 62% of the
variance in the perceived impact of supervision
and coaching on practice. The Supervisor
(coach):
 Taught new skills;
 Strengthened confidence;
 Offered safety in sessions;
 Devoted time to discipline-specific skills
38
Personal Qualities of a Coach
 Encouraging & enthusiastic
 Supportive
 Committed
 Sensitive
 Flexible
 Respectful & diplomatic
 Willing to share information, credit and recognition.
39
Barriers to Effective Coaching









Inadequate time allotted to coaching & scheduling conflicts
Role confusion due to the dual role of supervisor and coach
Feelings of inadequacy on the part of coaches (hence the
importance of selecting skilled coaches)
Poor match between coach and practitioner
Labor relationships that don’t support observation and feedback
Resistance in the organizational culture
Absence of strong leadership & commitment to implementation
Focus on paperwork compliance as opposed to changes in
practitioner behaviors
Cost
40
Research on the Impact of Coaching
The study below is based on an meta-analysis of hundreds
of studies in education. Note the dramatic differences in
impact upon practice.
41
Staff Evaluation
Evaluation provides critical feedback to
practitioners, trainers and managers
related to:
 Fidelity to the model being implemented
 Effectiveness of training and coaching strategies
 Impact of intervention upon clients
 Progress of implementation itself
42
Components of Evaluation
Staff & program evaluation and fidelity seem to
consist of some combination of measures of:
 Context—prerequisites that must be in place to
operate (staffing, qualifications, ratios)
 Compliance—extent to which the practitioner uses the
core intervention components as prescribed by the
EBP and avoids practices proscribed by the EBP
 Competence– the level of skill shown by the
practitioner in using the core intervention components
43
Effective Staff Evaluation Systems
Huber et al., (2003) described a highly effective
hospital management system for staff recruitment
and evaluation that included:






Ongoing training and education focusing on specific skills;
Cross-training on related roles, and in-services
Monthly dinners for discussion;
Performance evaluations based on direct observation to assess
practice knowledge, communication skills, and use of time;
Prompt verbal feedback followed by a write up with
recommendations; and
Quality improvement systems to keep the system on track.
44
Effective Staff Evaluation
In other words, in a highly effective
system, staff evaluation is part of a
sequence of supports designed to have
good people well prepared to do an
effective job.
45
Staff Evaluation & Fidelity to a Model
One of the most critical purposes of staff
evaluation in the context of implementing
EBP is to test practitioner fidelity to the
EBP core components.
46
Challenges to Measuring Fidelity
McGrew et al., (1994) noted that development of
fidelity measures is hampered by three factors:
 Most treatment models are not well defined
conceptually making it difficult to identify core
intervention components;
 When intervention components are identified, they are
not operationally defined with agreed-upon criteria; and
 Only a few models have been around long enough to
study planned and unplanned variations.
47
Staff Evaluation & Fidelity to a Model
Fortunately the SMCMH transformation relies upon
implementation of EBPs with effective fidelity
instruments already in place.

Multisystemic Treatment (MST) utilizes the Therapist Adherence
Measure, a 27-item measure used in phone interviews with
parents.

The Wraparound Fidelity Index (WFI) consists of asking
facilitators, parents and youth to rate 11 dimensions of services.

Assertive Community Treatment (ACT) utilizes a 73-item tool
with 17-item subset used to construct a fidelity index with three
subscales: staffing, organization, and service.
48
Fidelity to System Transformation
A major challenge to SMCMH is to go
beyond program level (MST,
Wraparound, ACT) fidelity and to
construct a fidelity index that reflects the
qualities of a transformed system as
defined and described in our plan.
49
Theory of Change & Fidelity Models
Possible resources to explore for developing a
fidelity model for the transformation…
 Hodges, Hernandez, Nesman & Lipien (2002)
demonstrated how a theory of change exercise can
help programs clarify their strategies to develop
fidelity measures.
 Shern, Trochim & LaComb (1995) used concept
mapping to develop fidelity measures for a mental
health program.
50
Program Evaluation
Program Evaluation assesses key
elements of the overall performance of
the organization in facilitating the
implementation of the program or
practice.
51
Facilitative Administration
Facilitative administration
 Provides leadership;
 Makes use of data inputs to inform decision
making;
 Supports overall processes; and
 Keeps staff organized and focused on the
desired clinical outcomes.
52
Systems Interventions
Systems interventions are strategies to
work with external systems to ensure the
availability of the financial, organizational,
and human resources required to support
the work of the practitioners.
53
Relationship between Core Components &
External Factors
54
Relationship between Core
Components & External Factors
55
Key Elements to Organizational Change
Implementation of EBP almost always requires organizational
change. Research points to 8 key elements to organizational
change:

Commitment of Leadership to the Implementation Process (Hunter,
Hunter & Rogers, 1993). Research indicates that leadership takes many
forms, including to:
 Initiate and shepherd the organization through the complex
change process
 Set explicit goals, communicate them clearly throughout the
organization, resolve conflicts with other goals, and reinforce
persistence
 Help create the details of activities, processes, and tasks in order
to operationalize implementation policies
 Inspire, guide, and provide direction
 Recruit, select, train, locate, advance, promote, or dismiss
employees to further the aims of implementation policies
56
Other Key Elements to Organizational
Change
 Involvement of stakeholders in planning and selection
of programs to implement; (Bierman et al., 2002)
 Creation of an implementation task force made of
consumers, stakeholders (including unions), etc.
(Joyce & Showers, 2002)
 Suggestions for “unfreezing” current organizational
practices (Cheung & Cheng, 1997)
 Resources for extra costs, effort, equipment, manuals,
materials recruiting, access to expertise, re-training for
new organizational roles; (Phillips, et al., 1978)
57
Other Key Elements to Organizational
Change
 Alignment of organizational structures to integrate staff
selection, training, performance evaluation, and ongoing training (Blasé et al., 1984)
 Alignment of organizational structures to achieve
horizontal and vertical integration (Unger et al., 2000)
 Commitment of on-going resources and support for
providing time and scheduling for coaching,
participatory planning, exercise of leadership, evolution
of teamwork and for generating and using local data
(Park & Han, 2002)
58
Considerations for Effective
Implementation
 At a system level and with staff and
consumer input, develop a rubric that
depicts what a ‘transformed system’
would look like at the system and
program levels.
59
Considerations for Effective
Implementation
 Construct site-based highly structured
exercises for sites working with their
consumers to identify how focusing on
this rubric would require changes in daily
operations.
 How would you change staff meetings,
staff evaluation, hours of operation,
charts, reports, referrals, office
operations, schedules, the waiting room?
60
Considerations for Effective
Implementation
 Develop program level rubric that
describe the extent to which a program is
achieving fidelity to operating in
conformance with the new transformed
system.
 Build in personnel and program
evaluation components that incorporate
how staff performance and program
structures are supporting the
61
transformation.
Considerations for Effective
Implementation
 At a structural level, consider a wide
variety strategies that reinforce fidelity to
transformation and the various program
EBPs.
 Examine what specific practitioner and
program changes are important and build
a system of reinforcements that extend
the impact of training.
62
Considerations for Effective
Implementation
 Be mindful that changing behaviors is
challenging and threatening.
 Training alone will not generate change in
practitioner behavior or program priorities.
 Since implementation drivers are integrative
and compensatory, everything you do
administratively to facilitate and support
changes, will reinforce the other drivers and
compensate for where drivers are not as
strong.
63
Considerations for Effective
Implementation
 In so many ways, implementation is
about the small things you do:





Small changes in intake forms, assessment tools, client
engagement protocols, and client charts, changes that subtly
cause practitioners to operate slightly differently
Notes at the top of agenda and specific standing agenda items
that focus on transformation issues and allow staff dialogue and
input into implementation
E-reminders to managers to observe practitioner groups
Changes in personnel review tools to reinforce transformation
Use of funds to foster individual staff research and presentations
to teams about issues related to implementing transformation
strategies.
64
Considerations for Effective
Implementation
In closing, Effective implementation of
transformation is about:
 Selection of the most effective, high-leverage
programs;
 Providing training and work-based support to reinforce
the values and practices inherent in a transformed
system; and
 Building structures, procedures and policies that
reinforce the priorities of the transformation…
And the devil is in the details.
Implementation is All.
65