Transcript Overview and Results of Pilot Testing Eric J. Bruns, April Sather, Kelly Hyde, Janet Walker & the Wraparound Evaluation and Research.

Overview and Results of Pilot Testing
Eric J. Bruns, April Sather, Kelly Hyde, Janet Walker & the Wraparound Evaluation and Research Team
Wraparound is a care coordination process that has evolved
over the past 20 years through efforts to help children with
complex emotional and behavioral needs function more
effectively in home, school, and community settings. As
wraparound has become more widely implemented, programs,
communities, and states have been increasingly interested in
measuring implementation fidelity (Bruns et al., 2004; Bruns et
al., in press; Walker & Bruns, 2006).
The Wraparound Fidelity Assessment System (WFAS) is a
multi-method approach to assessing the quality of
individualized care planning and management for children and
youth and their families. The instruments of the WFAS can be
used individually or in combination with one another as
research, evaluation, or quality assurance tools. WFAS
instruments include:
1) The Wraparound Fidelity Index, version 4 (WFI-4).
2) The Team Observation Measure (TOM)
3) The Document Review Measure (DRM)
4) The Community Supports for Wraparound Inventory (CSWI)
This poster presents a summary of psychometric data
collected to date for three of the four instruments that comprise
the WFAS.
Method and Sample
Data presented was compiled from pilot sites who requested
use of WFAS instruments. Local users were provided with the
instrument(s), a User’s Manual, and instructions for training
data collection staff to criteria. For the WFI-4, data collection
staff were provided with pre-recorded WFI-4 interviews and
instructed to score these practice interviews per instructions
and scoring rules presented in the scoring manual. For the
TOM, such training to criteria was completed via video
recorded team meetings provided on DVD. For the DRM,
training of reviewers to criteria was conducted using redacted
case file documents.
Thus far, WFI-4 data have been compiled for N=796 youth in
15 sites across 12 states nationally, including 642 completed
wraparound facilitator (WF) forms, 652 Caregiver (CG) forms,
308 Youth (Y) forms, and 260 Team Member (TM) forms. In
addition, TOM data on N=37 team meetings were collected
and compiled from 3 collaborating sites. CSWI data were
collected from N=259 across 7 sites. DRM data have been
received but remain in a preliminary phase of pilot testing.
Results presented in this poster are a summary of descriptive
and psychometric analyses that have been completed.
Youths in the WFI-4 sample were 14.4 years old on average
(SD = 3.5, range 1-22) and were 61% male. Fifty-eight percent
of caregivers were biological or adoptive parents, 14%
grandparents or other relatives, and 20% foster parents. Over
half (57%, n=362) of the youths currently were in or had
previously been in state custody. Youths had been enrolled in
wraparound 10.23 months on average (SD = 8.53, range 0-60
months).
WFI-4: Mean Total Scores
TOM: Distribution of Total Scores
100
90
80
70
60
50
Facilitator
Caregiver
Youth
Team
Member
Combined
7
6
5
4
3
2
1
0
10
90
Total Scores for Wrap vs. Non-Wrap Sites
80
70
60
50
Wrap sites (n=12)
Non-wrap sites (n=3)
83.8
77.1
73.6
78.4
74.4
62.1
61.9
70.1
Facilitator
Caregiver
Youth
Team Member
**
**
*
**
**p < .001
*p < .05
WFI-4: Internal Consistency (Cronbach’s alpha)
Scale (N items)
Facilitator Caregiver
Team
Member
Youth*
20
30
40
50
60
70
80
90
100
TOM: Reliability Results
•Cronbach’s alpha (20 items): .866
•Inter-rater reliability (indicators): 79%
•Inter-rater reliability (items): 72%
CSWI: Internal consistency (N=279)
Theme
alpha
Community partnership (6)
0.87
Phase 1: Engagement (6)
.557
.653
.529
.530
Collaborative action (6)
0.93
Phase 2: Planning (11)
.574
.664
.540
.677
Fiscal policies and sustainability (7)
0.94
Phase 3: Implementation (15)
.647
.819
.680
.743
Service array (7)
0.92
Phase 4: Transition (8)
.566
.736
.691
.668
Human resource development/support (8)
0.94
All WFI-4 items (40)
.807
.896
.852
.865
Accountability (7)
0.95
All CSWI items (40)
0.95
* Youth form has fewer items (N=32 total)
CSWI: Inter-rater reliability
WFI-4 Scale Scores: 10 Principles of Wraparound
83.6
P e r si st e nc e
72.3
C om m uni t y B a se d
86.8
C ol l a bor a t i v e
69.8
Out c om e s B a se d
67.1
85.2
S t r e ngt hs B a se d
Development of measures of the WFAS is supported by the National
Institute of Mental Health (R41MH077356). The authors would like
to thank all the communities participating in the pilot testing and
validation process for the WFI-4. Development of the WFI-4 was
supported through a task order from the Child, Adolescent and Family
Branch of the SAMHSA Center for Mental Health Services.
8
WFI-4 Discriminant validity
100
73.1
Te a m B a se d
71.3
I ndi v i dua l i z e d
84.3
Fa m i l y Voi c e a nd C hoi c e
92.5
C ul t ur a l c om pe t e nc e
0
10 20
30 40
50 60
70 80
Percent of possible total score
90 100
Summary and Conclusions
The results presented at this poster demonstrate some
strengths and weaknesses of the measures of the WFAS as
instruments for assessing implementation fidelity for the
wraparound process. For the WFI-4, strengths include its
overall internal consistency, adequate inter-rater reliability, and
ability to discriminate between wraparound and nonwraparound (or pre-wraparound) communities. Weaknesses
include moderate to low internal consistency scores for
subscales such as the 10 wraparound principle scores (likely
resulting from small n of items and heterogeneity of factors
being assessed). The WFI-4 also continues to show restriction
in its range of scores that is typical of self-report measures.
Mean = .63; SD = .18; Range = 22-93
N a t ur a l S uppor t s
Acknowledgments
(N=37)
9
Number of teams scored in this range
Introduction
Percent of total fidelity
www.rtc.pdx.edu/nwi
The Wraparound Fidelity Assessment System
Percent of total fidelity
National
Wraparound
Initiative
Site
n*
Intraclass correlation
1
16
.812
2
21
.781
3
14
.635
4
17
.713
5
30
.878
6
27
.893
7
17
.723
* Note: Limited to respondents with no missing data
Data from the TOM are preliminary but show promise: Scores
demonstrate good psychometrics and internal consistency, as
well as adequate inter-rater reliability. TOM data also have
been found to correlate significantly with WFI-4 scores (for the
CG and Y forms). More complete validation of the TOM will
require discriminant validity tests such as those completed for
the WFI-4, and additional inter-rater reliability tests.
Data from the CSWI indicate it performs well as a method for
assessing readiness or support for implementing the
wraparound process. An important next step will be to assess
the relationship between CSWI scores and fidelity as
assessed by the other measures of the WFAS. Finally, though
previous studies have found association between wraparound
fidelity instruments and outcomes (Bruns et al., in press), all
measures of the WFAS still need to be better studied with
respect to prediction of outcomes experienced by children,
youth, and families served via the wraparound process.
Supporting Technologies
Currently, we are working with Accountability Solutions
(www.accountabilitysolutions.com) to develop a webbased resource, the WFAS Online Data Entry and
Reporting System, that will allow users to enter their
data via a web portal that will compile data from the
WFAS instruments into one database, regardless of
how many people are entering data or where they are
located. This system will allow the user sites to create
a range of reports or export the data for their own
analyses. The system will be available in Spring 2008.
Literature cited
Bruns, E.J., Suter, J.C., Burchard, J.D., Leverentz-Brady, K. & Force, M. (2004).
Assessing fidelity to a community-based treatment for youth: the Wraparound
Fidelity Index. Journal of Emotional and Behavioral Disorders, 12, 69-79.
Bruns, E.J., Leverentz-Brady, K.M., & Suter, J.C. (in press). Is it wraparound yet?
Setting fidelity standards for the wraparound process. Journal of Behavioral
Health Services and Research.
Walker, J.S. & Bruns, E.J. (2006). Building on Practice-Based Evidence: Using
Expert Perspectives to Define the Wraparound Process. Psychiatric Services.
For more information
To learn more about the instruments of the WFAS, download a copy of this
poster, or inquire about becoming a collaborating site, please visit the Wraparound
Evaluation and Research Team at http://depts.washington.edu/ wrapeval, or
contact April Sather at [email protected] / 206-685-2310.
To learn more about the National
Wraparound Initiative, check out our
website at www.rtc.pdx.edu/nwi.