Lisa Saldana - Institute of Medicine
Download
Report
Transcript Lisa Saldana - Institute of Medicine
Toward Efficient and
Sustainable Delivery of
Interventions: The Stages of
Implementation Completion
Lisa Saldana, PhD
Oregon Social Learning Center
Institute of Medicine:
Innovations in Design and Utilization of
Measurement Systems to Promote Children’s
Cognitive, Affective, and Behavioral Health
November 6, 2014
FUNDING
PI: Saldana
NIMH R01 MH097748
NIMH R01 MH097748-S1
NIDA K23DA021603
PI: Chamberlain
NIMH R01MH076158
NIDA R01MH076158-05S1
NIDA P50 DA035763-01
SIC/COINS TEAM
Patricia Chamberlain, PhD
John Landsverk, PhD
Jason Chapman, PhD
M ark Campbell, M S
Holle Schaper, M S
Courtenay Padgett, M S
Vanessa Ewen
Katie Lewis
Co-I
Co-I
Co- I A nalyst
Research Economist
Statistician Coordinator
Project Coordinator
Data M anager
Editorial A ssistant
Off-Site I nvestigative Team:
Sonja Schoenwald, PhD Co-I
David Bradford, PhD Co -I
Larry Palinkas, PhD Co -I
A dvisory Board:
Greg A arons, Sarah Horowitz, Lonnie Snowden, Lynne M arsenich,
IMPLEMENTATION MEASUREMENT
Growing Body of Measures Targeting
Key Aspects of Implementation
Organizational Culture and Climate
e.g. Glisson, Aarons, Steckler
Organizational Readiness
e.g., Simpson, Weiner, Lehman, Helfrich
Leadership
e.g., Aarons, Kivipõld
Attitudes Toward EBPs
e.g., Aarons, Melnyk, Upton
Research Evidence Use
e.g., Palinkas, Stomski
MEASUREMENT GAP
Measure of Implementation Process
Rate of Implementation
Implementation Activities
Patterns of Implementation Behavior
Measure of Implementation Outcomes
Implementation Milestones
Penetration
CHALLENGES IN MEASURING
IMPLEMENTATION
Implementation of EBP entails extensive
planning, training, and quality assurance
Involves a complex set of interactions
between developers, system leaders,
front line staff, and consumers
Recursive process of well defined stages
or steps that are not necessarily linear
IMPLEMENTATION PROCESS HAS BEEN
ASSUMED TO BE EBP SPECIFIC
Little is known about which methods
and interactions are most important
for successful implementation
Little is known about how and if the
process influences successful
outcomes
THE STAGES OF IMPLEMENTATION
COMPLETION (SIC)
Developed as part of an
implementation trial focused on
scale-up of MTFC to fill the gap in
the lack of measures available
Developed out of necessity while
also addressing the challenges of
measuring implementation process
Chamberl ain , P., Brown, C.H., & Sal dana, L. (2011). Observ ational
measure of impl ementati on progres s : The Stages of I mpl ementati on
Compl eti on (SI C). I mplementation Science, 6, 116.
OBSERVATIONAL OPPORTUNIT Y: CAL-OH
STUDY
(PI: CHAMBERLAIN)
53 Sites observed from Engagement - Competency
Span 3 Phases of Implementation
Pre-Implementation, Implementation, Sustainment
Developed a Measurement Tool to measure
Rate of Implementation (Duration)
Thoroughness of Implementation (Proportion)
Date Driven
Stages of Implementation Completion
8 Stages from Engagement to Competency
Activities within Stages
SIC BASICS: THE SCIENCE OF
MEASURE DEVELOPMENT
Developed through an iterative process
Designed to target the general process and steps
of implementation
Involves assessment of implementation behavior
of different levels of agents
Initially designed to assess implementation
activities specific to MTFC
STAGES OF IMPLEMENTATION COMPLETION
(SIC)
Pre
Imp
8 Stages:
1. Engagement
2. Consideration of Feasibility
3. Readiness Planning
Involvement:
System Leader
System Leader, Agency
System Leader, Agency
4. Staff Hired and Trained
Agency, Practitioner
5. Adherence Monitoring
Practitioner, Client
Established
6. Services and Consultation
Practitioner, Client
7. Ongoing Services,
Practitioner, Client
Consultation, Fidelity, Feedback
Sus 8. Competency (certification ) System Leader, Agency,
Practitioner, Client
WITHIN STAGE ACTIVITIES
1.
Engagement
D a t e a g r e e d t o c o n si d e r i m p l e m e nt a ti o n
2 . C o n si d e r a ti o n o f F e a si bi li t y
D a t e o f s t a k e h ol d e r m e e t i ng # 1
3 . R e a d i ne ss P l a nni ng
D a t e o f c o s t c a l c ul a t o r /f u ndi ng p l a n r e v i e w
4. Staff Hired and Trained
D a t e o f i ni ti a l s u p e rv i so r t r a i ni ng
5 . A d h e r e nce M o ni t o ri ng E s t a b l is he d
D a t e f i d e l i t y t e c h n ol o gy s e t - u p
6 . S e r v i c e s a n d C o n s ul t a ti on B e g i n
D a t e o f f i r st c l i e nt s e r v e d
7 . O n g o i ng S e r v i c e s, C o n s ul t a ti on , F i de l i t y , F e e d b a c k
D a t e o f I m p l e m e nt a ti o n R e v i e w # 1
8 . C o m p e t e nc y (c e r ti f i ca ti o n )
D a t e o f f i r st c e r ti f i ca ti o n a p p l i ca ti on s u b m i t t e d
YIELDS THREE SCORES
Duration
Proportion
Stage Score
SCORING AND PSYCHOMETRICS
Challenges related to challenges of measuring
implementation
Recursive Nature Means Scoring is not Linear
Not Possible to total Duration ACROSS Stages
Rasch Based Modeling Helps Account for
Challenges
MTFC-SIC: PSYCHOMETRICS
Demonstrated Reliability Using Rasch Modeling Across All 8 Stages
A ctivity Rel iability (Proportion) = .77
Site Reliability (Proportion) = .92
A ctivity Rel iability (Duration) = .91
Site Reliability (Duration) = . 67
Demonstrated Face Validity
3 distinct cl usters of sites based on Pre -Implementation Behavior
Cl uster 1: High Proportion -Relatively Fast (23 Sites)
Cl uster 2: Low Proportion -Relatively Slow (22 Sites)
Cl uster 3: Non -Completers (8 Sites)
Demonstrated Predictive Validity
Sites that both took l onger to complete each stage and
completed fewer activities had significantly lower hazard of
successful program start -up during the study period
HR = 0.090, p < 0.001
(Cox Proportional Hazard Survival M odel)
UTILITY: REAL-WORLD SITES
75 most real-world teams
Sites were successfully clustered
Failed Sites spent significantly longer in pre implementation than successful sites
Sites that took longer to complete Stages 1 -3
significantly lower hazard of successful program
start-up
HR = 26.50, p < 0.002
(Cox Proportional Hazard Survival Model)
MTFC-SIC UTILIT Y:
REGARDLESS OF IMPLEMENTATION STRATEGY
Reliably distinguish good from poor
performers
Reliability distinguishes between
implementation strategies
Meaningful prediction of implementation
milestones
Pre-implementation SIC behavior predicts
successful program start-up
Pre-implementation SIC behavior predicts
discontinuing program
Pre-implementation and implementation
behavior combined predict development of
Competency (Stage 8)
17
ADAPTATION
Can the SIC be adapted to other treatments and
service sectors?
Will similar utility be found?
Is there a universality in implementation?
ADDITIONAL EBPS
SCHOOLS:
C a m p - C o p e - A l o t : C o m p ute r A s s i s te d Ve r s i o n o f C o p i n g C a t
C B I T S ( t r a um a i n t h e s c h o o ls )
S o u rc e o f S t r e n g t h ( s u i c id e p r ev e n t i o n )
C H I L D W E L FA R E :
KEEP
Safe Care
PTC
L i n ke d E B P s
M T F C ( O r i g in a l )
JUVENILE JUSTICE
MST
S U S TA N C E A B U S E / J J
MDFT
BSFT
Additional Services
HIV Prevention- Mujer Segura
Adult Services-Collaborative
Care
Integration of Services—
Housing First
Medical Interventions—
perinatal care
M O R E S I M I L A RI T I E S T H A N D I F F E R E N C E S I N I M P L E M E N TAT I ON AC T I V I T I E S
19
ADAPTATION PROCESS
Iterative Process
Collaborative with Developers and End Users
Retrospective Data Collection
Does the adaptation match reality?
Modification
Prospective Data Collection
Assessment of Utilization
INFORMING IMPLEMENTATION STRATEGIES
Modification to Existing Practice
UNI-SIC PSYCHOMETRICS
Evidence of Dimensionality
Item Bi-factor suggests distinct pre -implementation,
implementation, and sustainability phases
Site Distinction
Can distinguish 3 “types” of sites
Items Demonstrate Order Ef fects
Easier items early on, harder items in later stages
Reliability– Taking into account EBP, site, and phase
Indication of high reliability (0.91)
Noisiness
Out of 45 items, 4 appear to be noisy.
These are with purpose
WHERE DO WE GO FROM HERE?
Standardization of Measuring Implementation
Process and Milestones
Detection of Sites that are “at risk” for
Implementation Failure and targeted intervention
Evaluation of Implementation Strategies (e.g., CDT vs
IND)
COINS
THANK YOU
Lisa Saldana
[email protected]