Transcript Wrap Fidelity and Outcomes

Improving Practice to Wraparound
Through Use of Fidelity Measures
Enrica Bertoldo, Quality Support Manager
Eleanor Castillo, Director of Outcomes & Quality Assurance
Mary Ann Wong, Research Specialist
Veronica Padilla, Manager of Outcomes & Evaluations
26th
Annual California Mental Health Advocates for Children & Youth
Conference, Asilomar, California
May 4, 2006
EMQ Children & Family
Services
Our Mission
To work with children and their families
to transform their lives, build emotional,
social and familial well-being, and to
transform the systems that serve them.
Our Philosophy





Family Voice
 Team Based
Collaborative/Integrative
 Community-Based
Culturally Competent  Individualized
Strength-Based
 Natural Supports
Persistence
 Outcomes-Based
Purpose
Given the complexity of Wraparound, existing
fidelity measures are utilized to improve
practice.
This workshop will share successes and
challenges in the implementation of fidelity
measures (WFI-3.0 & WOF-2.0) that affect
practice and supervision to Wraparound.
A pilot in utilizing a Wraparound supervisory
adherence measure (W-SAM) will also be
discussed.
What is Wraparound?
Team-based, individualized service
planning process
Principles of community-based, strengthbased, family-centered, and culturally
competent
Wraparound is described as a promising
practice (Burns, Hoagwood, & Maultsby,
1998)
11 Core Elements of Wraparound
1. Voice and Choice
2. Youth/Family Team
3. Community-Based
Services/Supports
4. Cultural
Competence
5. Individualized
Services/Supports
6. Strengths-Based
Services/Supports
7. Natural Supports
8. Continuation of
Services/Supports
9. Collaboration
10. Flexible Resources/
Funding
11. Outcome-Based
Services/Supports
Essential Elements of Wraparound
Element 1: Voice and Choice
Families must be full and active partners at every level of the wraparound
process. If the team cannot reach consensus, the final decision should be
up to the caregiver.
Element 2: Youth and Family Team
Wraparound is a team-driven process involving caregivers, youth, natural
supports, and community services working together to develop,
implement, and evaluate the individualized plan.
Element 3: Community-based Services and Supports
Services and supports that the youth and family receive should be based
in their community. The family should not have to leave their community if
more restrictive services are necessary.
Element 4: Cultural Competence
The team should not only be respectful of the family’s beliefs and
traditions, but also actively seek to under-stand the family’s unique
perspectives and convey them to others.
Element 5: Individualized Services
This means that services and supports are tailored to the unique
situation, strengths, and needs of teach individual, and may involve
existing categorical services and informal sup-ports; modifying existing
services and supports; and or creating new services and supports.
Further, the team should create a specific plan to meet the family’s goals
and crisis/ safety plan to manage potential emergencies.
Element 6: Strengths-Based Services
The focus of the team should be on what is working and going well for
the family. While goals may be drawn up based on the family’s needs, the
plan should capitalize on the family’s positive abilities and characteristics.
Element 7: Natural Supports
Services and supports should reflect a balance of formal and informal
community and family supports rather than a reliance on formal
professional services.
Element 8: Continuation of Care
Services and supports must be provided unconditionally. In a crisis,
services and supports should be added rather than placing the youth with
a new provider.
Element 9: Collaboration
The team should coordinate services and supports so they seem
seamless to the family rather than disjointed.
Element 10: Flexible Funding and Resources
Successful wraparound teams are creative in their approach to service
delivery and have access to flexible funds and resources to implement
their ideas.
Element 11: Outcome-Based Services
Specific, measurable out-comes should be monitored to assess the youth
and family’s progress toward goals.
The description of each element is taken directly from Wraparound Fidelity Index 3.0 (2002),
produced by the Wraparound Evaluation and Research Team at the University of Vermont.
What is Fidelity to Wraparound?
Treatment Fidelity = The degree to which a
program is implemented as intended (Rast &
Bruns, 2003; Moncher & Prinz, 1991)
Adherence to the 11 Core Wraparound
Elements (e.g., WFI-3.0; Suter et al., 2002)
Adherence to the Child and Family Team
Process (e.g., WOF-2.0; Epstein et al., 2002)
Why Measure Fidelity?
Measuring fidelity is essential to families,
providers, policy makers and researchers
1. Without measuring fidelity, how do you ensure the
Wraparound process is occurring?
2. Without outcomes, the Wraparound process is just
one more fad
3. High fidelity has been associated with positive
outcomes in some studies
4. It improves quality assurance
5. It helps agencies secure more funding by proving
outcomes
6. It can even help create legislation on how families
and kids receive services
More Why’s…..
All nationally recognized Wraparound programs
have extensive fidelity measurements in place.
Why?
Without fidelity measurement standards firmly in
place, Wraparound programs and team
members have difficulty knowing what they are
doing well and what they need to improve in
order to achieve the ideal as presented in the
Wraparound vision.
Fidelity Implementation Measures
Wraparound Fidelity Index-3.0 (WFI-3.0)
Measures fidelity to Wraparound principles through
structured interviews
Administered to Youth, Caregiver, & Resource
Facilitator
Higher Score = Greater Fidelity
Wraparound Observation Form-2.0 (WOF-2.0)
Measures fidelity to Wraparound through
observation of the Child and Family Team meetings
Higher Score = Greater Fidelity
Fidelity to Wraparound
EMQ Sacramento has collected the
Wraparound Fidelity Index (WFI) since
2001
Data from WFI-3.0:
January 2003 – September 2005
A Comparison of EMQ Data &
National Data
WFI Total Scores by Respondent:
EMQ & National Data
79
81
Facilitator
67
Caregiver
74
EMQ
National
67
Youth
74
73
77
Total
0
20
40
60
80
100
*National WFI-3 Dataset came from the Wraparound Evaluation and Research Team presentation for the 18th Annual
Research Conference on Systems of Care and Children's Mental Health in Tampa on 3/7/05 by Eric J. Bruns, Ph.D. for "Is it
Wraparound Yet?" - Bootstrapping wraparound fidelity standards using the WFI .
WFI Elements: EMQ & National Data
83
87
Voice/Choice
56
Youth/Family Team
66
62
Community
73
83
88
Cultural Competence
EMQ
National
86
83
Individualized
80
81
Strengths
0
20
40
60
80
100
*National WFI-3 Dataset came from the Wraparound Evaluation and Research Team presentation for the Technical Assistance Partnership
Webinar on 4/5/04 by Eric J. Bruns, Ph.D. for "Ensuring High-Quality Wraparound"
WFI Elements: EMQ & National Data
Natural
Supports
47
63
76
Continuation
85
EMQ
National
76
Collaboration
68
Flex Funds
66
68
OutcomeBased
85
77
0
20
40
60
80
100
*National WFI-3 Dataset came from the Wraparound Evaluation and Research Team presentation for the Technical Assistance Partnership
Webinar on 4/5/04 by Eric J. Bruns, Ph.D. for "Ensuring High-Quality Wraparound"
The Relationship Between
Fidelity & Outcomes
The Relationship Between
Fidelity & Outcomes
Previous research has found an association
between greater Wraparound fidelity and
better child and family outcomes (Bruns,
2004), but the relationship has not been clearly
understood
EMQ (2005) completed a study to further
understand the relationship between fidelity to
the 11 core elements of Wraparound and
treatment outcomes
Demographics
Average Age at Admission: 14 years
63% Male; 37% Female
62% Caucasian; 26% African-American;
8% Latino; 2% Asian/Pacific Islander;
1% Native-American; 1% Other
Average Length of Stay: 15 months
Fidelity to Wraparound
Implementation Measure:
Wraparound Fidelity Index – 3.0 (WFI-3.0)
Collection since 2003
Sample in study included the following:
146 WFI’s collected from Youth
124 WFI’s collected from Caregivers
183 WFI’s collected from Resource
Facilitators
Outcome Measures at Discharge
Child and Adolescent Functional Assessment Scale
(CAFAS; Hodges, 2000)
Measures youth’s level of functioning; completed by Resource Facilitator
Lower Score = Greater Functioning
Exit Total CAFAS Score
Child Behavior Checklist (CBCL; Achenbach, 2001)
Measures youth’s level of behavior problems; administered to Caregiver
Lower Score = Lower Behavior Problems
Exit Total, Externalizing, & Internalizing CBCL Scores
Living Arrangement at Discharge: Community or
Facility
Correlations Among WFI Scores and Outcomes at Discharge
CBCL
Total
CBCL
Externalizing
CBCL
Internalizing
CAFAS
Total
Living in
Community
1. Voice and Choice
.12
.07
.12
.15*
.47
2. Youth and Family Team
-.04
-.04
.01
.07
.10
3. Community-Based
Services
-.19*
-.25**
-.40**
.21**
4. Cultural Competence
.06
.01
.02
.10
.15
5. Individualized Services
.09
.06
.01
.11
.05
6. Strengths-Based
-.10
-.09
-.14
.08
.05
7. Natural Supports
-.19*
-.12
-.13
-.07
.13
8. Continuation of Services
-.06
-.02
-.12
.12
-.02
9. Collaboration
-.04
-.03
-.17*
.13
-.11
10. Flexible Resources/Funds
-.06
.04
-.14
.05
.20**
11. Outcome-Based Services
-.16
-.04
-.18*
.14*
.21**
12. RF Total Fidelity
-.09
-.20
-.08
-.15
.16
13. Caregiver Total Fidelity
-.04
-.02
-.12
.04
-.15
14. Youth Total Fidelity
-.03
-.01
-.05
.02
-.05
15. WFI Total Fidelity
-.10
-.09
-.15
.05
.04
Note: *p<.05; **p<.01
-.21*
Significant Results
Community-Based Services
CBCL Total
-.19*
CBCL Externalizing
-.25**
CBCL Internalizing
-.21*
CAFAS Total
-.40**
Living in Community
.21**
Note: *p<.05; **p<.01
Greater community-based services is related
to lower impairment on the CBCL and CAFAS
at discharge
Greater community-based services is related
to community living arrangement at discharge
Results
Greater fidelity to the provision of Community
Services/Supports in Wraparound is related to
positive outcomes
No other elements had as significant correlations
to outcomes as Community Services/Supports
No significant relationship found between Total
Fidelity Scores and outcomes at discharge
Correlations were consistently low even when
significant
Implications
Implications for teams to support a
youth’s school attendance and
involvement in work, training, and other
community activities (e.g., church,
sports, art, etc.) as well as support
community-based living situations for
youth (as an alternative to residential or
institutional care) in order to improve
outcomes.
Challenges of the WFI
Direct feedback to teams was not
provided
The utility of the WFI in its relationship to
outcomes was limited based on EMQ’s
analysis of the data
Time intensive and limited resources
available to conduct 3 structured
interviews per youth
EMQ Practice Standards Initiative
Initiative to Improve Practice
Quality Improvement Project launched in
January 2004 to implement practice standards
and feedback tools across positions
 Family Facilitator
 Family Specialist
 Family Partner
Goal was to standardize job positions, provide
role clarity, and improve consistency in job
performance and supervision
Example of EMQ Feedback Tool:
Family Facilitator
Scoring:
Opportunity for Improvement (1)
Acceptable (2)
Exemplary (3)
Not Observed (*)
3 Dimensions:
A. Families receive services that are consistent with
wraparound values
B. Families receive services that are collaborative, integrated
and adhere to best practices
C. Children and Families served are safe and stable at home
and in the community
Facilitator Field Feedback Tool
A. Families receive services that are consistent with
wraparound values
1. Arrived on time and was prepared for the CFT
2. Started the CFT with “what’s working” to maintain strength based
focus and future orientation
3. Facilitated the CFT to validate and incorporate ideas, requests, and
concerns of all members
4. Evidence that needs and strategies were reviewed within the CFT to
assure the family voice and preference guide all activities
5. Evidence that plans were reviewed for progress on current
measurable, observable goals and updated as needed to address
new and changing needs
6. Evidence that the next CFT and staff visits were scheduled
according to family preference
Facilitator Field Feedback Tool
B. Families receive services that are collaborative, integrated and
adhere to best practices
1. For children open six months or longer, evidence that the CFT was comprised of
2.
3.
4.
5.
at least 50% non-paid informal members
Facilitated the CFT to allow pertinent agenda items to be covered according to
family preference and time availability
Demonstrated engagement and rapport with all team members including system
partners
Therapy, medications and adjunctive services were evaluated relative to plan
goals as needed
Specific tasks were assigned
C. Children and Families served are safe and stable at home and in
the community
1. Evidence that team has a plan B to back up plan A
2. If the CFT was for a child in out of home placement, this was addressed as a
priority with specific plan, tasks, and target date
3. Evidence that strategies and interventions developed emphasize the use of
typical resources in the child’s community
EMQ Feedback Tools
Data analysis showed that the tools were not
measuring what was intended; therefore, tools
were discarded
Led to an examination of the national
standards for Wraparound
EMQ Sacramento decided to collect data on
the Child and Family Team process using the
Wraparound Observation Form – Version 2.0
(WOF-2.0)
Wraparound Observation Form –
Second Version (WOF-2)
Measure to assess the implementation of the Wraparound
approach by direct observation of youth and family team
meetings
WOF has demonstrated good inter-rater reliability
(kappa=.886; Nordess & Epstein, 2003)
48 items that measures 8 characteristics:




Community-Based
Individualized
Family-Driven
Interagency Collaboration
 Unconditional Care
 Measurable Outcomes
 Management of Team Meeting
 Care Coordinator
Implementation of the WOF
WOF data collection began July 2005
Feedback provided in individual
supervision if completed by direct
supervisor
Verbal feedback provided to facilitators
after the child and family team meeting if
completed by neutral observer. Copy of
tools also provided to supervisors
Wraparound Observation Form –
Second Version (WOF-2)
Community-based resources (5 items)
1.
2.
3.
4.
5.
Information about resources / interventions in the area is offered to
the team.
Plan of care includes at least one public and/or private community
service/resource.
Plan of care includes at least one informal resource.
When residential placement is discussed, team chooses
community placements for child(ren) rather than out-of-community
placements, whenever possible.
Individuals (non-professionals) important to the family are present
at the meeting.
Individualized services for the family (9 items)
6.
7.
8.
9.
10.
11.
12.
13.
14.
If an initial plan of care meeting, the parent is asked what
treatments or interventions he/she felt worked/didn’t work prior to
WRAP.
Facilitator advocates for services and resources for the family
(e.g., identifies and argues for necessary services).
All services needed by family are included in plan (e.g., no
needed services were not offered).
Barriers to services or resources / interventions are identified and
solutions discussed.
The steps needed to implement the plan of care are clearly
specified by the team.
Strengths of family members are identified and discusses at the
meeting.
Plan of care that includes life domain(s) goals, objectives, and
resources/interventions is discussed (or written).
Plan of care goals, objectives, or interventions are based on
family/child strengths.
Safety plan/Crisis plan developed/reviewed.
Family-driven services (10 items)
15. Convenient arrangements for family’s presence at meeting are
made (e.g., location, time, transportation, day care
arrangements).
16. The parent/child is seated or invited to sit where he/she can be
included in the discussion.
17. Family members are treated in a courteous fashion at all times.
18. The family’s perspective is presented to professionals from other
agencies.
19. The family is asked what goals they would like to work on.
20. The parent is asked about the types of services or
resources/interventions he/she would prefer for his/her family.
21. Family members are involved in designing the plan of care.
22. In the plan of care, the family and team members are assigned (or
asked) tasks and responsibilities that promote the family’s
independence (e.g., accessing resources on own, budgeting,
maintaining housing).
23. The team plans to keep the family intact or to reunite the family.
24. Family Members voice agreement/disagreement with plan of
care.
Interagency collaboration (7 items)
25.
26.
27.
28.
29.
30.
31.
Staff from other agencies who care about or provide
resources/interventions to the family are present at the meeting.
Staff from other facilities or agencies (if present) have an
opportunity to provide input
Informal supports (if present) have an opportunity to provide input.
Problems that can develop in an interagency team (e.g., turf
problems, challenges to authority) are not evident or are resolved.
Staff from other agencies describe support resources
interventions available in the community.
Statement(s) made by a staff member or an informal support
indicate that contact/ communication with another team member
occurred between meetings.
Availability of alternative funding sources is discussed before
flexible funds are committed.
Unconditional Care (3 items)
32.
33.
34.
Termination of Wrap services is discussed because of the
multiplicity or severity of the child’s/family’s behaviors /problems.
Termination of other services (not Wrap) is discussed because of
the multiplicity or severity of the child’s/family’s behaviors
/problems.
For severe behavior challenges (e.g., gangs, drugs), discussion
focuses on safety plans/crisis plans (e.g., service and staff to be
provided) rather than termination.
Measurable Outcomes (3 items)
35. The plan of care goals are discussed in objective, measurable
terms.
36. The criteria for ending Wrap involvement are discussed.
37. Objective or verifiable information on child and parent functioning
is used as outcome data.
Management of team meeting (5 items)
38. Key participants are invited to the meeting (e.g., family members,
CPS worker, teacher, therapist, others identified by family).
39. Current information about the family (e.g., social history,
behavioral and emotional status) is gathered prior to the meeting
and shared at meeting (or beforehand).
40. All meeting participants introduce themselves (if applicable) or are
introduced.
41. The family is informed that they may be observed during the
meeting.
42. Plan of care is agreed on by all present at the meeting.
Facilitator (6 items)
43.
44.
45.
46.
47.
48.
Facilitator makes the agenda of meeting clear to participants.
Facilitator reviews goals, objectives, interventions and/or progress
of plan of care.
Facilitator directs (or reflects) team to discuss family/child
strengths.
Facilitator directs (or reflects) team to revise/update plan of care.
Facilitator summarizes content of the meeting at the conclusion of
the meeting.
Facilitator sets next meeting date/time.
EMQ WOF-2 Results:
Total Element Scores (N=30)
100
23
80
10
8
5
12
23
37
43
60
90
77
40
92
Undesired
95
88
77
63
57
Desired
20
lita
to
r
Fa
ci
en
t
an
ag
em
O
ut
co
m
es
M
Co
lla
bo
ra
Un
tio
co
n
nd
itio
na
lC
ar
e
en
-D
riv
ily
Fa
m
du
al
ize
d
In
di
vi
Co
m
m
un
ity
0
Results of WOF
EMQ Sacramento shows higher scores on
elements of Family-driven, Collaboration,
Unconditional Care & Management of CFT
Meetings
EMQ Sacramento continues to be challenged
by lower scores on Community-based
Resources, inclusive of natural supports
In contrast to the results of the WFI for
Outcomes (high scores), the WOF showed low
scores on this element
Slight increase in Total WFI scores since the
implementation of the WOF
Back to the WFI
Could the use of the WOF impact
WFI scores?
A look at WFI scores across time
WFI Total Scores Across Time
100
80
75
75
70
WOF Implementation
Begins 7/05
76
60
40
20
0
1/04-6/04
7/04-12/04
1/05-6/05
7/05-12/05
WFI Respondent Scores
Across Time
90
80
84
81 79 82
74
72
70
77
64
71 68 72 71
60
1/04-6/04
7/04-12/04
1/05-6/05
7/05-12/05
50
40
30
20
10
0
Facilitator
Caregiver
Youth
High Scores on Voice and Choice, Cultural Competence, Individualized Services, Strengths,
Outcome-Based Services
Low Scores on Natural Supports, Community-Based Services, and CFT Element
Challenges of the WOF
Limited resources in staff to become neutral
observers
Managers wanted to use the WOF as a
supervision tool
Willingness and comfort level of staff to
participate in direct observation
Staff perception of fear of families being
judged by observers
Sponsorship by management
Competing priorities in data collection
Next Steps
Supervisors will continue to utilize the
WOF to provide feedback
Operational realities limit the use of
neutral observers
EMQ will no longer collect the WFI due
to limitations in resources, the utility of
the tool for feedback, and its relationship
to outcomes
Continue to develop practice standards
References
Achenbach, T. M., & Rescorla, L.A. (2001). Manual for the ASEBA
school-age forms & profiles. Burlington, VT: University of
Vermont, Research Center for Children, Youth, & Families.
Epstein, M.H., Nordess, P.D., & Hertzog, M. (2002). Wraparound
Observation Form-Second Version.
Epstein, M.H., Nordess, P.D., et al. (2003). Assessing the
Wraparound Process During Family Planning Meetings.
Hodges, K. (2000). Child and Adolescent Functional Assessment
Scale (CAFAS). CAFAS Self-Training Manual and Blank Scoring
Forms (2003). Ann Arbor, MI
Nordness, P.D., & Epstein, M.H.(2003). Reliability of the
Wraparound Observation Form Second Version.
Suter, J., Force, M., Bruns, E., Leverentz-Brady, K., & Burchard, J.
(2002). Manual for training, administration, and scoring of the WFI
3.0. Burlington: University of Vermont.
Contact Information
Enrica Bertoldo
[email protected]
Quality Support Manager, Sacramento
Mary Ann Wong
[email protected]
Research Specialist, Sacramento
Wraparound Supervisor Adherence Measure (W-SAM):
A Pilot
Eleanor Castillo, Ph.D.
Veronica Padilla
Rationale
Supervision identified as a critical
aspect of Wraparound
1. No tool to date to measure this aspect of
Wraparound
Concept adapted from Multisystemic
Treatment’s (MST) Supervisor
Adherence Measure (SAM)
Rationale
WFI-3.0 measures adherence to
Wraparound principles from the
Youth, Caregiver, and Resource
Facilitator perspective
WOF-2.0 measures the Child and
Family Team aspect of Wraparound
Description
Facilitator rates the Clinical Program
Manager (CPM)
40 Items
Likert Scale (1 = Never; 5 = Almost
Always)
Administered quarterly
Highly reliable (alpha = .84-.98)
Process Implementation
Sponsorship from the Directors
Program quality improvement versus
“mandate”
Immediate feedback
Quality Management
Training
Evaluation
1. Overall Program Results
2. Individual CPM
Performance Appraisal
1. 360
2. Annual
Quality Management
3-Tiered: Agency-wide, Program, and
Individual
1. Training and Coaching
2. Evaluation
Performance Appraisal
1. Quarterly 360 feedback
2. Annual
Correlation between supervisor focus on
the Wraparound process and youth
outcomes
Quality Management
Trend analyses
1. Fidelity consistency over time
2. What factors affect fidelity
Learning curve with new manager or
facilitators
Next Steps
Continue pilot
Conduct further analyses on the
psychometric properties of the
scale
1. Factor analyses
2. Items reduction
Contact Information:
Eleanor Castillo, Ph.D.
Director of Outcomes & Quality Assurance
Email: [email protected]