Transcript Relationship of Fidelity to Wraparound and Outcomes for Youth and Families 6th Annual CANS Conference Betty Walton, Vicki S.

Relationship of Fidelity to Wraparound
and Outcomes
for Youth and Families
6th Annual CANS Conference
Betty Walton, Vicki S. Effland & Eric Bruns
San Francisco, CA
April 20, 2010
Objectives
 Effective implementation
of research based
practices
 Relationship of fidelity
to Outcomes
 Case Study:
Implementation of
Wraparound in Indiana
 Next Steps
What is Wraparound?
 Wraparound is a family-driven, youth guided, team-based
process for planning and implementing services and
supports.
 Through the wraparound process, teams create plans that
are geared toward meeting the unique and holistic needs of
children and youth with complex needs and their families.
 The wraparound team members (e.g., the identified youth,
his or her parents/caregivers, other family members and
community members, mental health professionals,
educators, and others) meet regularly to implement and
monitor the plan to ensure its success.
For which children and youth is the
wraparound process intended?
 Youth with needs that span home, school, and
community
 Youth with needs in multiple life domains
 (e.g., school, employment, residential stability,
safety, family relationships, basic needs)
 Youth for whom there are many adults involved and
they need to work together well for him or her to
succeed
For which youth in a system of care?
More
complex
needs
Most Intensive
intervention
level
Targeted
Intervention
Level
Prevention and
Universal Health
Promotion
Level
Less complex
needs
2% Full Wraparound
Process
3%
15%
80%
Targeted and
Individualized
Services
What is wraparound intended to achieve?
 Basing plans on strengths, needs, and culture leads to more complete




engagement of families
High-quality teamwork and flexible funds leads to better plans,
better fit between family needs and supports, and greater
integration of effort by helpers
 = Greater relevance, less dropout, better follow-through
As family works with a team to solve its own problems, develops
family members’ skills and self-efficacy
Process focuses on developing supportive relationships
Focus on setting goals and measuring outcomes leads to more
frequent problem-solving, more effective plans, greater
success
What is the research base?
Nine Published Controlled Studies of Wraparound
Study
Target population
Control Group Design
N
1. Hyde et al. (1996)*
Mental health
Non-equivalent comparison
69
2. Clark et al. (1998)*
Child welfare
Randomized control
132
3. Evans et al. (1998)*
Mental health
Randomized control
42
4. Bickman et al. (2003)*
Mental health
Non-equivalent comparison
111
5. Carney et al. (2003)*
Juvenile justice
Randomized control
141
6. Pullman et al. (2006)*
Juvenile justice
Historical comparison
204
7. Rast et al. (2007)*
Child welfare
Matched comparison
67
8. Rauso et al. (2009)
Child welfare
Matched comparison
210
9. Mears et al. (2009)
MH/Child
welfare
Matched comparison
121
*Included in 2009 meta-analysis (Suter & Bruns, 2009)
Research to Date on Wraparound
 There have been 9 controlled studies of wraparound
published in peer review journals
 Results consistently indicate superior outcomes for
wraparound compared to “services as usual”*
 Moderate (ES = .50) effects for living situation outcomes
 Small – medium (ES = .25 - .40) effects for behavioral,
functional, and community outcomes
 These ESs are similar to studies of evidence based therapies
and interventions (e.g., MST) as implemented in real world
conditions against alternative treatment conditions
*Suter, J.C. & Bruns, E.J. (2009). Effectiveness of the Wraparound Process for Children
with Emotional and Behavioral Disorders: A Meta-Analysis. Clinical Child and Family
Psychology Review, 12, 336-351
We Have Issues.
 17 year gap
between proving
that an approach
works and
implementation in
practice
 Effective
implementation
involves multiple
factors (Fixsen,
Naoom, Blasé,
Friedman &
Wallace, 2005)
Implementation with High Fidelity
Requires…
State
Support
County Context
and Readiness
Program
Evaluation
Organizational
Supports
Staff
Selection
Training
Performance
Management
Supervision
and
Coaching
National Implementation Research Network (NIRN)
Fidelity is critical to outcomes
F Higher levels of fidelity to organizational level assessment for
ACT was associated with greater reductions in days spent in
psychiatric hospitals (McGrew, Bond, Dietzen & Salyers, 1994)
F Improved youth delinquency outcomes for higher fidelity
Teaching Family model (Kirigin et. al. 1982)
F Improved youth delinquency outcomes for higher fidelity MST
(Henggler, Melton, Browndino, Scherer & Hanley, 1997)
F Better overall outcomes for youth receiving model adherent
FFT (Alexander, Pugh, Parsons and Sexton, 2000)
F Better outcomes for school-wide behavioral
management when implemented with fidelity (Felner et. al.
2001)
Monitoring fidelity of
implementation of child and
family teams
 Have facilitators and team members fill
out activity checklists
 Look at plans of care and meeting notes
 Sit in on and observe team meetings
 Ask the people who know– parents,
youth, facilitators, program heads
Wraparound Fidelity
Assessment System
TOM – Team
Observation
Measure
WFI-4 –
Wraparound
Fidelity Index
WFAS
CSWI – Community
Supports for
Wraparound Index
DRM - Document
Review Measure
The Wraparound Fidelity Index,
version 4
 Assesses implementation of the wraparound process
through brief interviews with multiple respondents
 Caregivers
 Youths
 Wraparound Facilitators
 Team Members
 Found to possess good psychometric characteristics
 Test-retest reliability
 Inter-rater agreement
 Internal consistency
 Used in research on wraparound
 Even more widely as a quality assurance mechanism by
wrap programs
Wraparound Fidelity Index, v.4
 Items on the principles and core activities, organized by the 4 phases of
wraparound
 Engagement: Did you select the people who would be on your youth
and family team?
 Principle = Team based
 Planning: Does the plan include strategies for helping your child get
involved with activities in the community?
 Principle = Community based
 Implementation: Does the team evaluate progress toward the goals of
the plan at every team meeting?
 Principle = Outcome based
 Transition: Will some members of your team be there to support you
when formal wraparound is complete?
 Principle = Persistence
Findings from fidelity research
 Fidelity-outcomes associations are tenuous and inconsistent
at the family/youth level
 At the site/program level, there is a discernable pattern of
WFI Fidelity scores across studies
 Wraparound vs. non-wraparound programs
 Wraparound programs with different levels of system
support
 Wraparound programs that achieve better outcomes
 Beginning to be able to interpret the “meaning” of WFI
scores
 < 65% = Not wraparound
 65 – 75% = Low average / Borderline
 75% - 85% = High average / Acceptable
 > 85% = “High fidelity”
Wraparound
in Indiana
 Began in mid- 1990s….
 3 CMHI grants
 State legislation to
replicate
 2 year development
grants
 Pilot 1915c Medicaid
Waiver
Lessons Learned
Not Enough
 Sources inspiring,




but …
Legislation and
state policy
Targeted funding
and contracts
Training
Persuasion
More Effective
 Collaborating with




stakeholders (youth &
families, providers, systems)
Gauging readiness & adjusting
tactics
Ongoing training, coaching
and support
Develop adequate
infrastructure
Monitor progress and use
feedback
CA-PRTF Grant
Section 6063 of the Deficit Reduction Act of 2005 ( PL 109-171) authorized up to
$217 million for a demonstration program for grantee states to use Medicaid
funding for home and community-based services as an alternative to
psychiatric residential treatment facilities (CA-PRTF).
Research Questions. Does the provision of Medicaid-funded home and
community based services to youth under this demonstration:
 Result in the maintenance or improvement in child’s functional status?
 On average, cost no more than anticipated aggregate PRTF expenditures in
the absence of the demonstration?
 National & State Evaluation: IMPAQ & Westat, Project Director, Oswaldo
Urdapilleta; Principal Investigator (PI), Garrett Morgan
 Wraparound
 9 States
Evaluation Minimum Data Set
 Demographic Information
 Functioning (CANS, CBCL or CAFAS)
 Fidelity (WFI-4)
 Services
 Child Welfare and Juvenile Justice Involvement
 Youth Services Survey (YSS & YSS-F)
 Costs
+ Strengths Based Site Assessment (Effland, 2010)
Indiana Youth Receiving Grant Services
with Outcome & Fidelity Measures
Jan 2008 - March 31, 2010 – 660 participants
372 youth had at least one CANS reassessment
& a WFI survey
Average age = 12.95 years old
71.3% Male
79.2% White
15.7% African American
4.4 % Hispanic
Measuring Improvement
Improvement in any Domain
 Behavioral Health
 Risks
 Functioning
 Strengths
 Caregiver
Reliable Change Index
Calculated for Each CANS
Domain, p < .05.
22
Expected Reliable Change
 60-80% of youth are expected to improve in at least
one of the dimensions
 20-40% of youth will improve in a specific
dimension
Improvement for Indiana Youth Ending an Episode
of Public Mental Health Services
Jan – March 2010, n = 2218
60
One Domain
55.55
50
Behavioral
Health
40
32.64
30
20
25.20
Risks
29.35
20.2
20.15
Functioning
Strengths
10
Care Giver
0
Percentage Improvement
CA-PRTF Grant
Monitoring Improvement (CANS), n = 484
80
71.2
In 1 Domain
70
60
Behavioral Health
50.7
50
40
43.4
44
Risks
37
25.5
30
Functioning
Strengths
20
Care Giver
10
0
Percentage Improvement
Variance in Improvement in 1 domain by Provider
Provider
N
Mean
SD
1
41
.6341
.48765
2
32
.6563
.48856
3
7
1.0
.00000
4
36
.8333
.37796
5
27
.7407
.44658
7
11
.9091
.301151
9
40
.7500
.43853
10
30
.6667
.47946
11
17
.9412
.24254
12
48
.6458
348332
15
7
1.0
.00000
16
11
.6364
.50452
17
7
1.0
.00000
18
16
.6250
.5000
20
10
.50
.52705
22
6
1.0
.00000
24
10
.50
.52705
Total
372
.7124
.45327
Variance in Improvement in Each CANS
Domain by Provider
Provider
N
BH
Risk
Functioning
1
42
.3571
.4048
.3333
.1190
.1463
2
32
.3438
.5313
.4063
.2500
.3125
3
7
.5714
.5714
1.0000
.5714
.0000
4
36
.5278
.5278
.5278
.5556
.2500
5
27
.4444
.5926
.4074
.2963
.1481
6
4
.2500
.000
.2500
.5000
.0000
7
11
.4545
.6364
.6364
.4545
.3636
8
2
.5000
.5000
.5000
.5000
.5000
9
40
.5250
.5250
.6000
.5500
.4000
10
30
.3333
.5000
.3000
.2333
.1677
11
17
.5294
.8235
.6471
.6471
.4706
12
48
.4583
.4792
.3642
.3542
.2292
Continued on next slide………
Strengths
Caregiver
Variance in Improvement in Each
CANS Domain by Provider (continued)
Provider
N
BH
Risks
Functioning
Strengths
Caregiver
13
3
.3333
.3333
.0000
.3333
.3333
14
2
.0000
.0000
.0000
.0000
.0000
15
7
.7143
.8571
.8571
.4286
.7143
16
11
.2727
.3636
.3636
.5455
.2727
17
7
.8571
.8571
.8571
.7143
1.0000
18
16
.2500
.2500
.2500
.1875
.0625
19
2
1.0000
1.0000
1.0000
1.0000
.0000
20
10
.3000
.3000
.2000
.2000
.3000
21
2
.0000
.0000
.0000
.5000
.0000
22
6
.667
.8333
.6667
.5000
.0000
23
1
.0000
.0000
.0000
.0000
.0000
24
10
.4000
.4000
.2000
.2000
.1000
Total
373
.4343
.5067
.4397
.3700
.2544
CA-PRTF Grant Fidelity, n = 481 teams
WFI Element
Nat’l
Average*
Indiana
December 2009
Indiana
March 2010
Outcomes Based
67%
68%
70%
Persistent
82%
84%
84%
Strengths Based
83%
88%
89%
Individualized
69%
68%
70%
Culturally Competent
91%
93%
95%
Community Based
71%
69%
68%
Collaborative
85%
90%
91%
Natural Supports
64%
60%
57%
Team Based
72%
83%
85%
Family Voice
Overall WFI
83%
77%
92%
79%
93%
80%
*( Buchard& Bruns, 2007)
Wraparound Fidelity
(n = 481 teams, 3/2010)
Using WFI Scores for Quality
Improvement
Indiana CA-PRTF: WFI Averages for XXXXXX County
January 2008-March 2010
Indiana CA-PRTF Medicaid Demonstration Grant
 Statewide Profile of
Wraparound Fidelity Index (WFI-4)
WFI Elements, Phases
and Total Fidelity
 Personalized Reports
for each Access Site
(County) with 5 or
more completed WFI
surveys
 Distributed at
Quarterly SOC
Meetings
n = 34 child and family teams
Overall Wraparound Fidelity = .84
Comparison of Respondent’s Perspectives
WFI Element
Caregiver
n = 179
Youth
n = 51
Facilitator
n = 457
Mean
Family Voice
.9248
.8388
.9410
.93
Team Based
.8247
.6790
.8623
.84
Natural Supports
.6371
.8611
.5703
.59
Collaborative
.9000
.8190
.9103
.90
Community Based
.7021
.6509
.7055
.69
Culturally Competent
.9421
.8989
.9507
.94
Individualized
.6707
.6447
.6970
.69
Strengths Based
.8772
.8375
.8993
.88
Persistent
.8069
.8431
.8698
.85
Outcomes Based
.6736
.5602
.7113
.69
TOTALWFI
.7953
.7603
.8115
.80
Variance in Caregiver’s WFI by Provider
Provider
N
Mean
SD
1
24
.811709
.1360389
2
15
.783264
. 1118299
4
18
.795267
. 1070815
5
11
.887348
.0761959
9
14
.767857
.2092316
10
16
.809809
.0897326
11
10
.837917
.0781106
12
25
.814852
.1035446
16
6
.7559028
.0943331
18
6
.793819
.1034879
20
6
.703781
.2380446
24
7
.810714
.1328737
Total
179
.794845
.1312387
Strengths Based Site Assessment
(Effland, 2010)
 Completed by TA Center
Coaches
 Rating Guide
Community characteristics at
each stage of implementation
1. Collaboration & partnership
2. Capacity building & staffing
3. Acquiring services & Supports
4. Accountability
5. Family Support
Overall Stage of
Implementation =
mean of ratings across
the 5 community
characteristics
Stage of Implementation* Results
 Stage 1 (Exploration & Adoption)
 Stage 2 (Program Installation)
 Stage 3 (Initial Implementation)
 Stage 4 (Full Operation)
20.0%
34.5%
32.7%
12.7%
*National Implementation Research Network (NIRN)
(Fixsen, et. al, 2005)
Wraparound Fidelity by Stage of
Implementation
 WFI interviews for at least one youth in 33 of 51
communities
 263 interviews in the 33 communities
 Average of 8 interviews per community
Wraparound Fidelity by
Stage of Implementation
Relationship of Fidelity and Outcomes
(Effland, McIntyre & Walton, 2010)
Level of
# of Youth %
Improvement
Wraparound with at
Nov 2009
Fidelity
least one
WFI &
CANS
outcomes*
# of Youth %
with at least Improvement
1 WFI &
March 2010
CANS
outcomes
High
28
82.1
123
78
Adequate
41
65.9
153
66.7
Borderline
13
69.2
66
72.7
9
55.6
33
63.3
Not
Wraparound
Logistic Regression Predicting
Improvement in 1 CANS Domain
__________________________________________________________
Predictor
ß
SE
Odds ratio
__________________________________________________________
Outcomes Based(WFI)
2.058
.782
7.833**
Risks_Baseline(CANS)
.145
.046
1.156**
Functioning_Baseline
.098
.043
1.103*
Strengths_Baseline
.076
.030
1.079*
__________________________________________________________
*p< .05, **p< .01
Note: Total Fidelity Element Scores used in model.
Logistic Regression Predicting
Improvement in Behavioral Health
_____________________________________________________
Predictor
ß
SE
Odds ratio
Age
Psychosis_Baseline
Anxiety_Baseline
Conduct_Baseline
Trauma_Baseline
Substance Use-Baseline
Team Based (WFI)
Natural Supports (WFI)
Collaborative (WFI)
Persistent (WFI)
Outcomes Based (WFI)
*p<.05, **p< .01, ***p<.001
-.116
.295
.589
.646
.387
.674
-2.484
1.690
-3.217
-1.843
2.299
.049
.148
.155
.163
.129
.213
1.029
.591
1.327
.830
.944
.890
1.343*
1.803***
1.908***
1.472**
1.963**
.083*
5.421**
.040*
.158*
9.965**
Logistic Regression Predicting
Improvement in Risk Domain
_____________________________________________________
Predictor
ß
SE
Odds ratio
______________________________________________________
RiskDomain_Baseline
.150
.048
1.162**
Functioning_Baseline
.087
.044
1.091*
StrengthDomain_Baseline .085
.032
1.089**
Outcomes Based (WFI)
1.887
.787
6.600*
______________________________________________________
*p<.05, **p< .01, ***p< .001
Total Fidelity Scores used in model.
Logistic Regression Predicting
Improvement in Functioning
_____________________________________________________
Predictor
ß
SE
Odds ratio
_____________________________________________________
Race(White)
.610
.357
1.840
Race (Black)
21.947
19472.230
3.400
Race(Native American) -.016
.698
.984
FunctioningDomain_B .339
.048
1.404***
FamilyVoice (WFI)
-2.582
1.298
.076*
Natural Supports (WFI) 1.523
.563
4.588**
Outcomes Based (WFI) 2.815
.955
16.700**
Age
-.119
.046
.888**
*p< .05, **p < .01, p<.001
Logistic Regression Predicting
Improvement in Strengths
_____________________________________________________
Predictor
ß
SE
Odds ratio
_____________________________________________________
StrengthsDomain_B
.315
.040
1.371***
Total WFI
4.379
1.500
79.734**
Age
-.122
.047
.885**
_____________________________________________________
*p<.05, **p<.01, ***p<.001
.
Logistic Regression Predicting
Improvement in Caregiver Strengths & Needs
_____________________________________________________
Predictor
ß
SE
Odds ratio
_____________________________________________________
Age_Baseline
-.100
.044
.905*
Anxiety_Baseline
.126
.152
1.201
CGDomain_Baseline .126
.029
1.134**
______________________________________________________
*p< .05, **p<.01, ***p < .001
Conclusions
 Evidence of association between level of system of
care development, wraparound fidelity and
outcomes
 Considering elements of wraparound, domains
and key item CANS scores enrich understanding
 Outcomes for “No Wraparound” similar to usual
public services
 Emerging evidence of the importance of outcomes
based practice & overall fidelity link to building
strengths
 Further research indicated
Next Steps
 Update analysis every 6 months to monitor trends,




& continue using WFI information to improve
services
Compare Group(s) of Youth with Similar Needs who
received different services
Consider refining Fidelity categories
Explore what factors predict high fidelity and
positive outcomes
Explore who benefits from intensive community
based services under what circumstances
For more information, contact:
 Betty A. Walton: [email protected]
 http://dmha.fssa.in.gov/darmha
 Vicki S. Effland: [email protected]
 Eric J. Bruns: [email protected]
 www.wrapinfo.org; www.nwi.pdx.edu