Transcript National Wraparound Initiative

Wraparound Fidelity Monitoring in
the Massachusetts CBHI
Summary of 2011 Results
August 12, 2011
Eric Bruns and April Sather
University of Washington
Wraparound Evaluation and Research Team
[email protected] / [email protected]
With thanks to Andrea Gewirtz and Melissa King
Purpose for this Meeting
1. Very quick review of wraparound
fidelity assessment and research
2. Review data from Massachusetts
fidelity monitoring
3. Discuss what has been found and what
it means
Wraparound Competence:
What do we want to measure?
Wraparound Principles
1. Family voice and choice
2. Team-based
3. Natural supports
4. Collaboration
5. Community-based
6. Culturally competent
7. Individualized
8. Strengths based
9. Persistence
Walker, Bruns, Adams, Miles, Osher et al., 2004
10.Outcome-based
Wraparound Competence
Implementing the practice model:
The Four Phases of Wraparound
Phase
1A
Engagement and Support
Phase
1B
Team Preparation
Phase
2
Initial Plan Development
Phase
3
Implementation
Phase
4
Transition
Time
Phase 1 A and B
Phase 1 :
Engagement and Team Preparation
•
Care Coordinator & Family Partner meets with the
family to discuss the wraparound process and listen to
the family’s story.
•
Discuss concerns, needs, hopes, dreams, and
strengths.
•
Listen to the family’s vision for the future.
•
Assess for safety and make a provisional safety plan if
needed
•
Identify people who care about the family as well as
people the family have found helpful for each family
member.
•
Reach agreement about who will come to a meeting to
develop a plan and where we should have that
meeting.
Phase 2
Phase 2: Initial Plan Development
• Conduct first Care Planning Team (CPT) meeting with people
who are providing services to the family as well as people who
are connected to the family in a supportive role.
• The team will:
– Review the family vision
– Develop a Mission Statement about what the team will
be working on together
– Review the family’s needs
– Come up with several different ways to meet those
needs that match up with the family’s strengths
• Different team members will take on different tasks that have
been agreed to.
Phase 3
Phase 3: Plan Implementation
• Based on the CPT meetings, the team has created a written
plan of care.
• Action steps have been created, team members are committed
to do the work, and our team comes together regularly.
• When the team meets, it:
– Reviews Accomplishments (what has been done and
what’s been going well);
– Assesses whether the plan has been working to achieve
the family’s goals;
– Adjusts things that aren’t working within the plan;
– Assigns new tasks to team members.
Phase 4
Phase 4: Transition
• There is a point when the team will no longer need to
meet regularly.
• Transition out of Wraparound may involve a final
meeting of the whole team, a small celebration, or
simply the family deciding they are ready to move on.
• The family we will get a record of what work was
completed as well as list of what was accomplished.
• The team will also make a plan for the future, including
who the family can call on if they need help or if they
need to re-convene their team.
• Sometimes transition steps include the family and their
supports practicing responses to crises or problems that
may arise
Wraparound Fidelity Index, v.4
• Items on the principles and core activities, organized
by the 4 phases of wraparound
– Engagement: Did you select the people who would
be on your youth and family team?
• Principle = Team based
– Planning: Does the plan include strategies for
helping your child get involved with activities in the
community?
• Principle = Community based
– Implementation: Does the team evaluate progress
toward the goals of the plan at every team meeting?
• Principle = Outcome based
– Transition: Will some members of your team be
there to support you when formal wraparound is
complete?
• Principle = Persistence
Team Observation Measure
• Consists of 20 items, with two items dedicated
to each of the 10 principles of wraparound.
• Each item consists of 3-5 indicators of highquality wraparound practice as expressed
during a care planning team meeting.
• Internal consistency very good
• Inter-rater reliability found to be adequate
(Average 79% agreement for all indicators)
• Correlates well with WFI scores
– In previous studies and evaluations…
Document Review Measure
• Mass version consists of 26 items
• Each wraparound principle linked to several items
• Scale = 0-3, with criteria for each point on the scale
• Source material = documentation (electronic or
paper) related to youth’s wraparound process
– Strengths, needs, culture discovery
documentation
– Wraparound plan of care
– Safety plan
– Transition plan
– Progress notes
Measures of the WFAS seem to tap
into the same constructs
Mean WFI-4 and TOM total scores by site in one state’s QI project
100%
90%
80%
70%
60%
50%
Site 1
Site 3
Site 5
Site 6
Site 7
Site 8
Site 9
Site 10
WFI Combined
84%
77%
79%
79%
69%
72%
72%
81%
Team Observation
93%
84%
83%
78%
67%
56%
63%
78%
WFAS Measures seem to be related to
what we would expect
• Families who experience better outcomes have
staff who score higher on fidelity tools (Bruns,
Rast et al., 2006; Effland, McIntyre, & Walton,
2010)
• Wraparound initiatives with positive fidelity
assessments demonstrate more positive
outcomes (Bruns, Leverentz-Brady, & Suter,
2008)
• Wraparound initiatives with better support to
implementation have higher fidelity scores
What does it take to get high
fidelity scores?
• Training and coaching: Found to be
associated with gains in fidelity and
higher fidelity
• Community, Fiscal, and Service system
support: Better developed supports =
fidelity scores
• Sticking with it: Fidelity scores tend to
increase over time and with persistence
of effort
– But… they can also backslide…
Average WFI Fidelity Score
Caregiver WFI Fidelity over time in NV
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
86%
64%
Pre Training
(2001)
73%
72%
Training Only
(2002)
Training and
After going to
Coaching (2004) scale (2007-09)
Bruns, Rast, Walker, Peterson, & Bosworth (2006).
American Journal of Community Psychology.
What can we do with the data?
• Measurement as communication
– What are we attempting to do for
families?
– How well are we stacking up against
benchmarks? What is holding us back?
– How well are we doing over time?
– What is associated with positive
outcomes in Massachusetts?
Measurement for quality
improvement
• Examine data at the program level to discuss
what is and is not happening consistently
• What can be reinforced in individual and group
supervision?
• What are the team, program, and resource
barriers?
• In what areas do we need additional training?
– At the community or system level by
collaborative groups, along with reports of
team, program, and resource barriers
reported up from CSAs or programs
What kind of data get used?
• What kind of data?
– Item scores – Ns who endorse each
response option (Yes, No, Somewhat)
– Members on teams
– Specific plan of care elements
– Qualitative feedback on open-ended
items
Part 2: Results from Massachusetts
Scores on the WFI, TOM, and
Mass-DRM
Total scores on the WFAS instruments
100
80
Percent of 60
total fidelity 40
20
0
Massachusetts 2010
Massachusetts 2011
National mean
WFI
TOM
DRM
78
77
74
83
85
77
61
74
-
Correlations between WFI, TOM, and
DRM across Massachusetts CSAs
2
• WFI – TOM = .121
0
• WFI – DRM = -.473 (p<.001)
1
0
• DRM – TOM = .012
2
• WFI – TOM = -.256
0
1
1
• WFI – DRM = .326 (p<.05)
• DRM – TOM =.168
WFI-DRM association at a site level
2010 –
DRM / WFI


80.00






70.00





DRM


 
60.00








50.00





40.00


72.00
76.00
WFI
80.00
84.00
-
2011
WFI-DRM association at a site level
2011 –
DRM/TOM
Wraparound Fidelity Index
Massachusetts Overall
WFI-4 Total Fidelity Scores
All CSAs
2010 N = 637 / 2011 N = 587
Total Phase
Principle
Mean
Overall
Eng
Pln
Imp
Tr
FVC
TB
NS
Col CB
CC
Ind SB
iv
Pe
r
OB
Mass
2010
78
86
82
79
64
88
84
53
89
74
95
70
83
83
63
Mass
2011
77
85
82
77
64
85
82
52
89
74
93
73
79
78
65
Nat
Mean
74
75
74
79
66
82
71
61
84
69
90
64
82
79
64
(CG only)
WFI Fidelity by Phase
All CSAs 2010 N=637 / 2011 N = 587
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
2010
2011
Nat Mean
Total
Eng
Plan
Impl
Trans
78%
77%
74%
86%
85%
75%
82%
82%
74%
79%
77%
79%
64%
64%
66%
WFI Fidelity by Principle
All CSAs
2010 N=637 / 2011 N = 587
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Total
FVC
TB
NS
COL
CB
CC
IND
SB
PER
OB
2010
78%
88%
84%
53%
89%
74%
95%
70%
83%
83%
63%
2011
77%
85%
82%
52%
89%
74%
93%
73%
79%
78%
65%
Nat Mean
74%
82%
71%
61%
84%
69%
90%
64%
82%
79%
64%
Item Scores: Engagement
All CSAs
ITEMS
2010
2011
CG1.1 - When you first met ICC, were you given time to talk
about strengths *and* Did this process help you appreciate?
1.78 1.74 1.65
CG1.2 - Before your 1st team meeting, did your ICC fully
explain the WA process and the choices you could make?
1.78 1.82 1.68
CG1.3 - At beginning of wrap process, did you have a
chance to tell WF what things have worked in the past?
1.83 1.78 1.75
CG1.4 - Did you select the people who would be on your WA
team?
1.41 1.34 0.86
CG1.5 - Is it difficult to get team members to attend team
meetings when they are needed?
1.66 1.65 1.57
CG1.6 - Before your 1st wrap team meeting, did you go
through a process of i.d.ing what leads to crises for child
and family?
1.81 1.90 1.52
NAT
MEAN
Item Scores: Planning
All CSAs
ITEMS
NAT
2010
2011
1.83
1.87
1.64
1.76
1.76
1.56
.99
1.10
0.61
CG2.4 - Are the supports and services in your WA plan connected
to the strengths and abilities of your child and family?
1.80
1.72
1.74
CG2.5 - Does the wrap plan include strategies for helping your
child get involved w/ activities in his/her community?
1.31
1.26
1.24
CG2.6 - Are there members of your wrap team who do not have
a role in implementing your plan?
1.73
1.73
1.67
CG2.1 - Did you and your team create a written plan that
describes how the team will meet your child's needs? *and* Do
you have a copy?
CG2.2 - Did the team develop any kind of written statement
about what it is working on with your child and family? *and*
Can you describe what your team mission says?
CG2.3 - Does your wrap plan include mostly professional
services?
MEANS
Item Scores: Planning
All CSAs
ITEMS
2010
2011
NAT
MEAN
CG2.7 - Does your team brainstorm many strategies
to address your family's needs before selecting one?
1.79
1.79 1.73
CG2.8 - Is there a crisis plan? *and* does this plan
specify how to prevent crisis?
1.48
1.57 1.43
CG2.9 - Do you feel confident that, in crisis your
team can keep your child in the community?
1.57
1.58 1.5
CG2.10 - Do you feel like other people on your team
have higher priority than you in designing your wrap
plan?
1.83
1.73 1.53
CG2.11 - During planning process, did team make
enough time to understand values? *and* Is your
wrap plan in tune w/ family's values?
1.89
1.84 1.73
Item Scores: Implementation
All CSAs
ITEMS
2010
2011
CG3.1 - Are important decisions made about your
child or family when you are not there?
1.90
1.86
1.64
CG3.2 - When your wrap team has a good idea for
support, can they find resources or make it happen?
1.58
1.54
1.7
CG3.3 - Does your wrap team get your child
involved w/ activities they like and do well?
1.05
0.95
1.2
CG3.4 - Does the team find ways to increase the
support you get from friends & family?
1.09
1.13
1.22
CG3.5 - Do the members of your team hold each
another responsible for doing their part?
1.73
1.73
1.7
.68
.66
0.95
1.75
1.70
1.74
CG3.6 - Is there a friend or advocate of your child or
family who actively participates in wrap team?
CG3.7 - Does your team come up w/ new ideas?
*and* Does your team come w/ ideas when
something's not working?
NAT
MEAN
Item Scores: Implementation
All CSAs
ITEMS
2010
2011
NAT
MEANS
CG3.8 - Are the services and supports in your wrap difficult for
you family to access?
1.61 1.66 1.54
CG3.9 - Does the team assign specific tasks to all team
members at end of mtng? *and* Does team review team
member's follow-through at next mtng?
1.67 1.63 1.59
CG3.10 - Do members of your team always use language you
can understand?
1.96 1.94 1.93
CG3.11 - Does your team create a positive atmosphere around
successes and accomplishments at each team meeting?
1.88 1.91 1.86
CG3.12 - Does your team go out of its way to make sure all
members present ideas and participate in decisions?
1.82 1.84 1.67
CG3.13 - Do you think your wrap process could be discontinued
before you're ready?
1.48 1.32 1.35
CG3.14 - Do all the members of your team demonstrate respect
for you and your family?
1.96 1.91 1.88
CG3.15 - Does your child have the opportunity to communicate
their own ideas when it comes to decisions?
1.49 1.39 1.71
Item Scores: Transition
All CSAs
ITEMS
2010
2011
CG4.1 - Has your team discussed a plan for how wrap will
end *and* Does your team have a plan for when?
.46
0.70
CG4.2 - Has the wrap process helped your child develop
friendships w/ other youth
.94
.92
1.2
CG4.3 - Has the wrap process helped your child to solve
their own problems?
1.09
1.02
1.3
CG4.4 - Has your team helped you and your child prepare
for major transitions?
1.50
1.64
1.35
CG4.5 - After formal wrap ends, do you think the process
will be able to be 're-started' if you need it?
1.82
1.80
1.61
CG4.6 - Has the wrap process helped your family to
develop or strengthen relationships that will support you
when WA is finished?
1.45
1.42
1.49
CG4.7 - Do you feel like you and your family will be able to
succeed on its own?
1.33
1.41
1.22
CG4.8 - Will some members of your team be there to
support you when formal wrap is finished?
1.60
1.40
1.65
NAT
MEAN
0.68
WFI – Summary
• Overall: Significantly higher than national mean
• Overall, no change from 2010
• Relative strengths:
– Engagement, Planning phases
– Family voice and choice, team based, collaborative,
Individualized
– Initial engagement, family selecting team, crisis planning,
vision/mission statements, balance of formal and informal
supports
• Relative weaknesses:
– Natural supports, involvement of friends/advocates,
involvement in the community
– Youth voice, youth friendships, youth involvement in
community
WFI – Summary of trends from
2010
• Areas of improvement from 2010:
– Crisis identification and plans
– Transition planning
• including prep for transitions from wrap and within wrap
• Areas of decline:
– Family voice and choice – selecting team, perception of
family influence
– Youth friends, youth voice, youth problem solving
– Perception that wrap could be discontinued before family
ready
Team Observation Measure
Mass CBHI overall
TOM Total Fidelity Scores
All CSAs
2010 N=285 / 2011 N=658
Mass CBHI
Mean 2010
Total
Principle
Overall
Mean
FVC
TB
NS
Col
CB
CC
Ind
SB
83
95
84
43
88
91
92
83
88
89
73
85
94
85 51 92 91 93 86 90
92
78
77
86
72
83
68
Per
OB
N = 285
Mass CBHI
Mean 2011
N = 658
National
Mean
58
81
84
82
72
79
TOM Fidelity by Principle
All CSAs
2010 N=285 / 2011 N=658
100
90
80
95 94
86 84 85
83 85
77
72
92 91 91 92 93
92
88
88 90 89
86
84
83
82 83
81
79
78
73
72
68
70
58
51
43
60
50
40
30
20
10
0
Total F V &
Score
C
TB
NS
Col
CB
CC
Indiv
SB
Per
OB
Mean 2010
83
95
84
43
88
91
92
83
88
89
73
National Mean
77
86
72
58
81
84
82
72
79
83
68
Mean 2011
85
94
85
51
92
91
93
86
90
92
78
TOM Item & Indicator Scores
All CSAs
TEAM BASED
2010
2011
Nat Mean
Item 1: Team Membership and Attendance
3.10
3.09
2.39
a. Parent/caregiver is a team member and present at the
meeting.
.99
.99
.93
b. Youth (over age 9) is a team member and present at the
meeting.
.61
.61
.75
c. Key school or other public stakeholder agency representatives
are present.
.61
.60
.38
Item 2: Effective Team Process
3.61
3.71
3.4
a. Team meeting attendees are oriented to the wraparound
process and understand the purpose of the meeting.
.88
.91
.86
b. The facilitator assists the team to review and prioritize family
and youth needs.
.95
.95
.89
c. Tasks and strategies are explicitly linked to goals.*
.91
.95
.85
d. Potential barriers to the nominated strategy or option are
discussed and problem-solved.
.86
.91
.84
TOM Item & Indicator Scores
All CSAs
COLLABORATIVE
2010
2011
Nat
Mean
Item 3: Facilitator Preparation
3.50
3.68
3.36
a. There is a clear agenda or outline for the meeting, which provides
an understanding of the overall purpose of the meeting and the
major sections of the meeting.
.86
.91
.84
b. The meeting follows an agenda or outline such that team
members know the purpose of their activities at a given time.
.84
.91
.84
c. The facilitator has prepared needed documents and materials
prior to the meeting.
.90
.96
.86
d. A plan for the next meeting is presented, including time & date.
.89
.91
.87
3.50
3.69
3.14
a. Team members demonstrate consistent willingness to
compromise or explore further options when there is disagreement.
.96
.97
.91
b. Team members reach shared agreement after having solicited
information from several members or having generated several
ideas.
.87
.95
.81
c. The plan of care is agreed upon by all present at the meeting.
.94
.98
.83
d. The facilitator summarizes the content of the meeting at the end
of the meeting, including next steps and responsibilities.
.76
.82
.87
Item 4: Effective decision making
TOM Item & Indicator Scores
All CSAs
INDIVIDUALIZED
2010
2011
Nat
Mean
Item 5: Creative Brainstorming and Options
3.11
3.16
2.46
a. The team considers several different strategies for meeting each
need and achieving each goal that is discussed.
.83
.87
.78
b. The team considers multiple options for tasks or action steps.
.81
.86
.73
c. The facilitator leads a robust brainstorming process to develop
multiple options to meet priority needs.
.67
.69
.40
3.53
3.70
3.3
a. Planning includes action steps or goals for other family members,
not just identified youth.
.85
.92
.79
b. Facilitator and team members draw from knowledge about the
community to generate strategies and action steps based on unique
community supports.
.85
.91
.78
c. Team facilitates the creation of individualized supports or services
to meet the unique needs of child and/or family.*
.89
.96
.83
d. Youth, caregiver, & family members give their opinions about
potential services, supports, or strategies; including describing what
has or has not worked in the past.
.93
.95
.91
Item 6: Individualized process
TOM Item & Indicator Scores
All CSAs
NATURAL SUPPORTS
2010 2011
Nat
Mean
1.54
1.61
2.38
a. Natural supports for the family are team members and are
present.
.27
.27
.26
b. Team provides multiple opportunities for natural supports to
participate in significant areas of discussion.
.75
.80
.66
c. Community team members and natural supports participate in
decision-making.
.72
.79
.66
d. Community team members and natural supports have a clear
role on the team.*
.72
.81
.61
1.94
2.47
2.37
a. Brainstorming of options and strategies include strategies to be
implemented by natural and community supports.
.70
.77
.66
b. The plan of care represents a balance between formal services
and informal supports.
.45
.58
.36
c. There are flexible resources available to the team to allow for
creative services, supports, and strategies.
.21
.58
.91
Item 7: Natural and community supports
Item 8: Natural support plans
TOM Item & Indicator Scores
All CSAs
UNCONDITIONAL/
PERSISTANCE
2010
2011
Nat
Mean
Item 9: Team mission and plans
3.44
3.61
3.23
a. The team discusses or has produced a mission/vision
statement.
.84
.90
.75
b. The team creates or references a plan that guides its work.
.91
.96
.87
c. The team has confirmed or is creating a crisis plan.
.78
.90
.77
d. The team plan contains specific goals that are linked to
strategies and action steps.
.92
.96
.87
3.66
3.72
3.42
a. The team explicitly assigns responsibility for action steps
that define who will do what, when, and how often.*
.87
.91
.81
b. There is a clear understanding of who is responsible for
action steps and follow up on strategies in the plan.
.92
.92
.86
c. Providers and agency representatives at the meeting
demonstrate that they are working for the family and not
there to represent a different agenda or set of interests.
.96
.98
.94
Item 10: Shared Responsibility
TOM Item & Indicator Scores
All CSAs
CULTURAL COMPETENCE
2010
2011
Nat
Mean
Item 11: Facilitation skills
3.55
3.62
2.97
a. Facilitator is able to impart understanding about what the
wraparound process is, how it will work for this family, and how
individual team members will participate.
.83
.88
.74
b. Facilitator reflects, summarizes, and makes process-oriented
comments.
.89
.92
.77
c. Facilitator is able to manage disagreement & conflict and elicit
underlying interests, needs, and motivations of team members.
.92
.92
.73
d. Talk is well distributed across team members and each team
member makes an extended or important contribution.
.93
.92
.87
3.76
3.86
3.56
a. The youth, caregiver, and family members are given time to talk
about the family’s values, beliefs, and traditions.
.87
.95
.86
b. The team demonstrates a clear and strong sense of respect for the
family’s values, beliefs, and traditions.
.95
.97
.88
c. Meetings and meeting materials are provided in the language the
family is most comfortable with.
.97
.98
.98
d. Members of the team use language the family can understand (i.e.
no professional jargon/acronyms)
.97
.98
.96
Item 12: Cultural and Linguistic Competence
TOM Item & Indicator Scores
All CSAs
OUTCOMES BASED
2010
2011
Nat
Mean
Item 13: Outcomes Based Process
2.88
3.06
2.83
a. The team uses objective measurement strategies.
.67
.76
.68
b. The team assesses goals/strategies using measures of progress.
.72
.77
.62
c. The team revises the plan if progress toward goals is not evident.
.84
.88
.87
2.99
3.15
2.62
a. The team conducts a systematic review of members’ progress on
assigned action steps.
.78
.84
.71
b. The facilitator checks in with the team members about their
comfort and satisfaction with the team process.
.74
.79
.73
c. Objective or verifiable data is used as evidence of success,
progress, or lack thereof.
.72
.78
.61
Item 14: Evaluating Progress and Success
TOM Item & Indicator Scores
All CSAs
VOICE AND CHOICE
2010
2011
Nat
Mean
Item 15: Youth and Family Voice
3.89
3.86
3.53
a. The team provides extra opportunity for caregivers to speak and
offer opinions, especially during decision making.
.98
.98
.95
b. The team provides extra opportunity for the youth to speak and
offer opinions, especially during decision making.
.93
.93
.91
c. Caregivers, parents, and family members are afforded
opportunities to speak in an open-ended way about current and
past experiences and/or about hopes for the future.
.98
.98
.95
d. The youth is invited to speak in an open-ended way about
current and past experiences and/or about hopes for the future.
.96
.93
.88
3.72
3.69
3.38
a. The youth prioritizes life domains, goals, or needs on which he
or she would like the team to work.
.78
.79
.76
b. The caregiver or parent prioritizes life domains goals, or needs
on which he or she would like the team to work.
.93
.96
.89
c. The family and youth have highest priority in decision making
.97
.95
.90
Item 16: Youth and Family Choice
TOM Item & Indicator Scores
All CSAs
STRENGTH BASED
2010
2011
Nat
Mean
Item 17: Focus on Strengths
3.31
3.47
2.89
a. Team members acknowledge or list caregiver/youth strengths.
.92
.95
.82
b. Team builds an understanding of how youth strengths contribute to
the success of team mission or goals.
.78
.85
.67
c. In designing strategies, team members consider and build on
strengths of the youth and family.
.82
.89
.72
d. Facilitator and team members analyze youth & family member
perspectives and stories to identify functional strengths.
.78
.87
.69
3.7
3.69
3.44
a. The team focuses on improvements or accomplishments throughout
the meeting.
.91
.92
.82
b. The facilitator directs a process that prevents blame or excessive
focus on or discussion of negative events.
.97
.95
.89
c. The facilitator encourages team culture by celebrating successes
since the last meeting
.88
.93
.78
d. There is a sense of openness and trust among team members.
.94
.93
.90
Item 18: Positive Team Culture
TOM Item & Indicator Scores
All CSAs
COMMUNITY BASED
2010
2011
Nat
Mean
Item 19: Community Focus
3.41
3.45
3.16
a. The team is actively brainstorming and facilitating community
activities for the youth and family.*
.82
.85
.71
b. The team prioritizes services that are community-based.
.82
.86
.79
c. The team prioritizes access to services that are easily accessible to
the youth and family.
.93
.94
.90
Item 20: Least Restrictive Environment
3.92
3.86
3.59
a. The team’s mission and/or identified needs support the youth’s
integration into the least restrictive residential and educational
environments possible.*
.99
.97
.98
b. When residential placements are discussed, team chooses
community placements for the child or youth rather than out-ofcommunity placements, wherever possible.
.87
.88
.94
c. Serious challenges are discussed in terms of finding solutions, not
placement in more restrictive residential or educational
environments.
.95
.94
.94
TOM – Summary
• Overall: Significantly higher than national mean
• Significant improvement from 2010
• Relative strengths:
– Nearly all principles. Particular strengths in:
• School/Agency reps present, effective teams and facilitation, facilitator
prep, Creative/individualized services, services for all family members,
crisis plans
• Relative weaknesses:
– Youth involvement/presence at meetings
– Natural supports present and involved
– Flex resources available and used when needed
TOM – Summary of trends from
2010
• Areas of improvement from 2010:
– Facilitator prep, team process
– Flexible resources available
– Balance of informal/formal services/supports
– Crisis planning and plans
– Use of objective measures and following up
– Tying strengths to planning
• Areas of decline:
– Youths being asked their priorities
– Youths present
Document Review Measure
Mass CBHI overall
Mass-DRM Principle Scores
Total Mass DRM improvement: 61% to 74%
DRM Item scores: Relative strengths
2010
2011
8
The family is present at all meetings.
2.83
2.91
4
There is a clearly articulated long range vision of the youth and family.
2.31
There is evidence that the family and/or youth are making decisions about the direction
and methods of the team.
2.53
There is evidence that all members on the CPT are in regular communication, to ensure
coordinated efforts on the youth and family's behalf.
2.71
Services, supports and tasks are based on the needs of the youth and family, not
availability of services or support that are needed.
2.34
The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and
interventions to deescalate crisis situations.
2.62
The family and youth's needs or concerns are identified across life domains.
2.13
The risk management/safety plan (RMSP) includes strengths of youth and family and
strategies for preventing potential crises.
2.51
There is evidence that the wraparound plan, including the RMSP, is updated at each
team meeting or as needed.
2.25
There is documentation that progress toward goals and tasks has been monitored.
2.29
2.9
17
11
15
23
2
22
12
18
5
25
16
Detailed examples of family and youth strengths, assets, resources, culture, values, and
beliefs are included to support planning for areas of priority need.
2.07
The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for
each step, including the youth's local ESP/MCI.
2.43
There is evidence that the team brainstorms more than 6 options to address goals.
1.99
2.84
2.74
2.73
2.68
2.62
2.59
2.56
2.56
2.52
2.39
2.33
DRM item scores: Needs for
improvement
2010
10
13
There is evidence that natural supports are actively involved in the planning and
implementation process or there are ongoing and persistent efforts to identify and
engage natural supports.
There is documentation that an attempt has been made to collaborate with the
primary care physician (PCP) or primary care clinician (PCC).
0.8
0.97
1.72
1.21
21
The wraparound plan includes opportunities for the youth to engage in community
activities that he or she likes and does well.
1.05
There is documentation of transition and sustainability planning that identifies needs,
services and support that will continue after formal wraparound ends or when the
youth transitions to the adult service system.
1.16
26
The RMSP is proactively shared with the MCI team.
20
24
6
7
14
The RMSP is updated as needed; always when a hub-dependent support joins the
youth and family team.
There is evidence that a strengths, needs, and culture discovery process was
completed within 10 days of enrollment of the youth.
The initial wraparound plan was developed within 35 days of enrollment with the
youth and family.
The plan includes strengths-based goals that are measurable with specific
measurement strategies.
2011
1.34
1.4
1.43
1.97
1.94
2.09
1.64
2.11
2.47
2.16
1.75
2.17
DRM Summary
• Greater correlation with WFI/TOM from 2010
• Large improvement from 2010, particularly:
– Vision statements
– Updated plans
– Needs ID’d across domains
– Examples of strengths, beliefs, culture
– SNCD within 10 days
• Declines in:
– Wrap plans w/in 35 days
– Collaboration with PCPs
DRM Psychometrics
• Total Alpha = .85
• Principal Components Analysis
(w/Varimax Rotation) found
Interpretable 8-Factor Solution
explaining 60.6% of variance
• Correlation with WFI = .326 (p<.05)
– Certain items negatively correlate with
WFI and/or have negligible correlation
with DRM total
6 “Bad” DRM items
Do not correlate with the WFI and have low
item-total correlations
There is evidence that a CANS and full assessment were completed with the
youth and family within 10 days of enrollment.
The family is present at all meetings.
There is evidence that natural supports are actively involved in the planning and
implementation process or there are ongoing and persistent efforts to identify
and engage natural supports.
There is evidence that all members of the CPT are in regular communication, to
ensure coordinated efforts on the youth and family's behalf.
There is documentation that an attempt has been made to collaborate with the
primary care physician (PCP) or primary care clinician (PCC).
The RMSP is proactively shared with the MCI team.
11 “Good” DRM Items
Significant correlation with WFI AND
high item totals
Documentation is strengths based and identifies and prioritizes the needs for
the youth and family.
Detailed examples of family and youth strengths, assets, culture, values and
resources are included to support planning for areas of priority need.
The initial ICP was developed within 28 days of enrollment with the youth and
family.
The ICP specifies the goals that will help the family reach their long-range vision.
There is evidence that the ICP is updated at each team meeting and as needed.
Services, supports and tasks are based on the needs of the youth and family and
not on the availability of services or support that are needed.
There is evidence that the CPT brainstorms more than six options to address
goals.
There is documentation that progress toward goals and tasks has been
monitored.
There is evidence that the youth and family are making progress toward their
goals.
The RMSP identifies triggers, signs or behaviors that indicate the crisis is
DRM Principal Components
Analysis
• Goals and Strategies (5 items)
• Strengths and Needs (4 items)
• Progress monitoring and timely updates (4 items)
• Crisis and safety plans present and of quality (3 items)
• Natural supports and community based services evident
(3 items)
• Families as primary decision makers (3 items)
• Meeting mandated timelines (2 items: CANS and ICP)
• RMSP is shared with MCI team (1 item)
DRM Principal Components
Analysis
Item
There is evidence that a CANS and full assessment were completed with the youth and family
within 10 days of enrollment.
The family and youth's needs or concerns are identified across life domains.
Documentation is strengths based and identifies and prioritizes the needs for the youth and family.
There is a clearly articulated long range vision of the youth and family.
Detailed examples of family and youth strengths, assets, culture, values and resources are included
to support planning for areas of priority need.
There is evidence that a strengths, needs, and culture discovery process was completed before the
first CPT meeting.
The initial ICP was developed within 28 days of enrollment with the youth and family.
The family is present at all meetings.
The ICP specifies the goals that will help the family reach their long-range vision.
There is evidence that natural supports are actively involved in the planning and implementation
process or there are ongoing and persistent efforts to identify and engage natural supports.
There is evidence that all members of the CPT are in regular communication, to ensure
coordinated efforts on the youth and family's behalf.
There is evidence that the ICP is updated at each team meeting and as needed.
There is documentation that an attempt has been made to collaborate with the primary care
physician (PCP) or primary care clinician (PCC).
The ICP includes strengths-based goals that are measurable with specific measurement strategies.
Services, supports and tasks are based on the needs of the youth and family and not on the
availability of services or support that are needed.
There is evidence that the CPT brainstorms more than six options to address goals.
There is evidence that the family and/or youth are making decisions about the direction and
methods of the team.
There is documentation that progress toward goals and tasks has been monitored.
There is evidence that the youth and family are making progress toward their goals.
The ICP includes opportunities for the youth to engage in community activities that he or she likes
and does well.
There is documentation of transition and sustainability planning that identifies needs, services and
supports that will continue after formal wraparound ends or when the youth transitions to the
adult service system.
The risk management/safety plan (RMSP) includes strengths of youth and family and strategies for
preventing potential crises.
The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and notes
interventions to deescalate crisis situations.
The RMSP is updated as needed, and reviewed at every CPT meeting.
The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for each step,
including the youth's local ESP/MCI.
The RMSP is proactively shared with the MCI team.
Str and
Needs
Progress
monitoring
Goals and and timely Crisis and
Strategies updates
safety
Natural
supports
and
Family de
community cision
MCI
based
making
sharing
Meet
mandated
timelines
0.098
0.840
0.745
0.245
-0.035
0.066
0.125
0.338
0.100
0.042
0.135
0.067
-0.075
-0.011
0.156
0.183
0.055
0.086
-0.057
-0.237
0.040
0.072
0.104
0.435
0.069
0.030
-0.080
-0.187
0.830
0.128
-0.060
-0.002
0.708
0.187
0.048
0.176
0.137
0.003
0.089
-0.051
0.614
0.026
0.079
0.169
-0.130
0.214
0.027
0.675
0.137
0.110
0.133
0.386
-0.124
0.208
0.024
-0.019
0.176
-0.048
-0.004
-0.051
0.042
-0.076
0.776
0.098
0.195
-0.073
0.145
0.006
0.359
0.677
0.022
0.020
0.125
-0.027
0.064
0.085
0.601
0.145
-0.271
0.155
0.088
0.091
0.478
0.209
0.074
0.680
0.141
0.044
0.165
0.064
0.301
0.232
-0.078
0.058
-0.067
0.161
0.027
0.049
0.242
0.676
0.011
0.020
-0.155
0.050
-0.037
0.101
0.088
-0.035
0.703
0.319
0.162
0.085
0.253
-0.066
0.600
0.566
0.403
0.037
0.018
0.083
-0.031
0.244
0.013
0.233
0.103
0.046
0.067
0.095
0.040
0.047
0.106
0.226
0.161
0.143
0.128
0.739
0.648
0.108
0.141
0.106
0.172
0.219
0.405
0.754
0.198
0.101
0.063
-0.003
0.017
-0.045
0.058
0.079
0.080
0.183
0.174
-0.028
0.705
0.001
-0.051
-0.073
0.066
0.114
0.078
0.190
0.667
-0.014
0.390
-0.032
0.145
-0.019
0.204
0.777
-0.008
-0.011
0.002
0.044
0.105
0.119
0.070
0.029
0.160
0.538
0.800
0.222
0.048
-0.145
0.026
-0.213
0.049
0.316
0.015
0.029
-0.051
0.077
0.169
-0.001
-0.077
0.147
0.689
0.137
0.189
-0.014
0.291
0.091
-0.018
0.745
0.038
-0.109
Summary of Findings
Summary of findings
• Continued strong fidelity results, especially for a large
statewide initiative
– WFI 3 points higher than national mean
• WFI scores stay stable, TOM shows small improvement,
DRM show large improvement
• Big improvements noted for:
– Crisis planning
– Transition planning
– Team process / Facilitator prep
– Availability of flexible resources (but is this an artifact
of TOM scoring clarification?)
– Documentation of strengths, needs, goals, crisis plans
Summary of findings
• Areas of strength continue to be:
– Managing the details of the wrap
process BUT ALSO
– Family voice and choice
– Collaborative teaming
– Individualized plans with balance of
formal svcs and informal supports
Looking deeper: Relative weaknesses
•
Certain bigger ideas of wraparound not yet happening consistently
– Natural supports and natural support plans
– Connection to community activities and informal supports
– Full youth engagement
– Youth and family self-reliance and self-efficacy
– Youths present at meetings
•
Better on some things, but still room for more improvement:
– Robust brainstorming
– Objective goal setting and monitoring of progress
– Transition planning
•
Structural barriers:
– Flexible funds
– Local community collaboratives
– Time to develop more creative plans linked to community supports
and natural supports
– Consistent understanding of how FPs and ICCs work together
Possible strategies: System level
• Fiscal supports
– Use of case rates
– Availability of flexible funds
– Methods to ensure less emphasis on productivity vs fidelity
and outcomes?
• Community supports, collaboration, transition planning:
– Encouragement to taking a community-level system of care
approach
– Hire resource development specialist(s)
– Training and support to community collaborative teams
– Continued social marketing of CBHI and cross agency
training
– Collecting info on and reviewing barriers faced by teams
and CSAs
• Youth engagement
– Use of youth peer to peer support partners?
Possible strategies: Organizational
level
• Natural supports:
– Specific strategies (family finding?), training, time,
supervision emphasis
– Role for family partner?
• Transition planning
– Clear understanding of interactions between medical
necessity determination and Unconditional Care
• Ensure clear expectations are delivered to families about transition
• Outcomes based:
– Greater attention to consistent tracking of goal attainment
and meeting needs
• Use a dashboard approach?
• Staff turnover/attrition:
– Training and guidance on staff recruitment and selection;
compensation, caseloads, expectations (e.g., taking
vacation to attend trainings)
Possible strategies: Practice level
• Invest in specific strategies for engaging youth
– E.g., Achieve My Plan! (AMP) from Portland State University
• Individualized care
– More attention to developing individualized, creative plans
in supervision, QI
• Emphasis on training and supervision on:
– Transition planning
– Individualized care plans
– Natural support action steps
– Teaching self-efficacy
Measurement issues
• WFI: Start to interview youths
– As a way of modeling a new era of engagement of Youth
– To find out what is going on from their perspective
• How to use the TOM?
– Ceiling effect now evident
– Doesn’t correlate with WFI in Mass
• Bias among raters/supervisors
– Recommend use of independent TOM raters for a
subsample of selected teams
Measurement issues
• Continue to use the DRM?
– Now clearly demonstrating utility as a fidelity tool
• Could be evidence for the benefit of its continued use
• OR…. Could mean don’t need to use it – duplicative
– Perhaps revise based on data and disseminate for use by
supervisors / CSA QA?
Measurement issues
•
Analytic questions: What influences fidelity?
– Established vs. new sites
– Caseload sizes
– Urban vs. rural
– Characteristics of families served
– Time enrolled in wraparound
– Agency culture
– Turnover
– Perception of Community supports
WHAT ARE THE OUTCOMES???
• Residential
• Medicaid expenditures/Fiscal
• CANS/Functional
Part 4: Massachusetts Data by
CSA
WFI CSA Results
N
Total Score
18 Lawrence
18
29 Springfield
20
0.86
0.82
0.82
24 Park
Street
14
20 Lynn
19
12 Gandara
14
2 Attleboro
18
23 N. Central
22
22 New
Bedford
21
5 Cape Ann
20
28 S. Central
17
0.82
0.82
0.81
0.81
0.81
0.81
0.80
0.77
Nat Mean
0.8
28 S Central
CSA
CSA
5 Cape Ann
0.81
22 New Bedford
0.81
23 N Central
0.81
2 Attleboro
0.81
12 Gandara
0.82
20 Lynn
0.82
24 Park Street
0.82
29 Springfield
0.82
0.86
18 Lawrence
0.7
0.75
0.8
0.85
Percent Fidelity
0.9
WFI CSA Results
N
Total
Score
10 Fall River
19
0.80
16 Holyoke
16
0.80
6 Cape and Isl
19
0.79
1 Arlington
18
0.79
25 Pittsfield
21
0.79
19 Lowell
20
0.78
0.77
Nat Mean
CSA
CSA
11 Framingham
0.76
31 Worcester E
0.76
14 Harbor
0.77
8 Coastal
0.77
32 Worcester W
0.78
19 Lowell
0.78
25 Pittsfield
0.79
1 Arlington
0.79
0.79
32 Worcester W
20
0.78
6 C and I
8 Coastal
21
0.77
16 Holyoke
0.8
14 Harbor
18
0.77
10 Fall River
0.8
31 Worcester E
18
0.76
11 Framingham
20
0.76
0.74
0.76
0.78
0.8
Percent Fidelity
0.82
WFI CSA Results
13 Greenfield
N
Total
Score
19
0.76
27 RVW
20
0.76
9 Dimock
20
0.75
21 Malden
22
0.75
15 Haverhill
20
0.74
7 CSR
11
0.74
0.77
Nat Mean
0.55
30 Walden
CSA
CSA
17 Hyde Park
0.69
4 Cambridge
0.7
26 Plymouth
0.72
3 Brockton
0.73
7 CSR
0.74
15 Haverhill
0.74
21 Malden
0.75
9 Dimock
0.75
3 Brockton
18
0.73
26 Plymouth
20
0.72
27 RVW
0.76
4 Cambridge
22
0.70
13 Greenfield
0.76
17 Hyde Park
20
0.69
30 Walden
3
0.55
0
0.5
Percent Fidelity
1
TOM CSA Results
N
Total
Score
Nat Mean
8 Coastal
14
0.93
20 Lynn
7 CSR
58
0.93
22 New Bedford
0.91
10 Fall River
32
0.93
26 Plymouth
0.91
11 Framingham
0.91
21 Malden
0.91
5 Cape Ann
38
0.92
30 Walden
4
0.92
21 Malden
44
11
Framingham
22
26 Plymouth
18
22 New
Bedford
23
20 Lynn
19
CSA
CSA
0.77
0.9
30 Walden
0.92
0.91
5 Cape Ann
0.92
0.91
10 Fall River
0.93
7 CSR
0.93
8 Coastal
0.93
0.91
0.91
0.7
0.90
Percent Fidelity
TOM CSA Results
CSA
N
Total
Score
6 C and I
32
0.89
16 Holyoke
14
9 Dimock
13
0.86
2 Attleboro
10
0.86
31 Worcester E
23
0.86
18 Lawrence
10
0.85
0.77
Nat Mean
0.83
13 Greenfield
CSA
0.86
15 Haverhill
0.85
4 Cambridge
0.85
23 N Central
0.85
1 Arlington
0.85
18 Lawrence
0.85
31 Worcester E
0.86
2 Attleboro
0.86
9 Dimock
0.86
0.86
1 Arlington
18
0.85
23 N Central
31
0.85
16 Holyoke
4 Cambridge
16
0.85
6 C and I
15 Haverhill
21
0.85
13 Greenfield
16
0.83
0.7
0.89
0.75
0.8
0.85
Percent Fidelity
0.9
TOM CSA Results
CSA
N
Total
Score
28 S Central
14
0.83
0.77
Nat Mean
0.73
12 Gandara
0.75
17 Hyde Park
8
0.83
25 Pittsfield
25
0.82
3 Brockton
11
0.81
14 Harbor
18
0.79
27 RVW
25
0.79
CSA
24 Park Street
29 Springfield
0.77
32 Worcester W
0.77
19 Lowell
0.79
27 RVW
0.79
14 Harbor
0.79
0.81
3 Brockton
19 Lowell
16
0.79
25 Pittsfield
32 Worcester W
13
0.77
17 Hyde Park
0.83
29 Springfield
31
0.77
28 S Central
0.83
24 Park Street
18
0.75
12 Gandara
6
0.73
0.65
0.82
0.7
0.75
0.8
Percent Fidelity
0.85
DRM CSA Results
CSA
N
Total
Score
28 S Central
10
0.82
6 Cape and Isl
10
11
Framingham
10
20 Lynn
10
0.80
18 Lawrence
10
0.80
MA Mean
0.74
0.78
5 Cape Ann
1 Arlington
0.79
4 Cambridge
0.79
0.82
14 Harbor
10
31 Worcester
E
10
4 Cambridge
10
0.79
1 Arlington
10
0.79
5 Cape Ann
10
0.78
CSA
0.82
31 Worcester E
0.8
14 Harbor
0.8
18 Lawrence
0.8
20 Lynn
0.8
0.80
0.80
11 Framingham
0.82
6 C and I
0.82
28 S Central
0.82
0.7
0.75
0.8
Percent Fidelity
0.85
DRM CSA Results
CSA
N
Total
Score
32 Worcester W
10
0.76
8 Coastal
10
10
0.76
17 Hyde Park
10
0.76
23 N Central
10
0.76
10 Fall River
10
0.75
CSA
0.76
3 Brockton
0.74
MA Mean
7 CSR
0.73
2 Attleboro
0.73
29 Springfield
0.73
22 New Bedford
0.74
26 Plymouth
0.74
0.75
10 Fall River
23 N Central
0.76
17 Hyde Park
0.76
3 Brockton
0.76
26 Plymouth
10
0.74
22 New Bedford
10
0.74
8 Coastal
0.76
29 Springfield
10
0.73
32 Worcester W
0.76
2 Attleboro
10
0.73
7 CSR
10
0.73
0.7
0.72
0.74
0.76
Percent Fidelity
0.78
DRM CSA Results
N
Total
Score
13 Greenfield
12
0.71
21 Malden
10
0.71
16 Holyoke
10
0.71
24 Park Street
10
0.70
15 Haverhill
10
0.69
25 Pittsfield
10
0.69
27 RVW
10
0.69
30 Walden
10
9 Dimock
0.74
MA Mean
0.64
12 Gandara
19 Lowell
0.65
9 Dimock
0.65
0.66
30 Walden
CSA
CSA
27 RVW
0.69
25 Pittsfield
0.69
15 Haverhill
0.69
24 Park Street
0.7
16 Holyoke
0.71
0.66
21 Malden
0.71
10
0.65
13 Greenfield
0.71
19 Lowell
10
0.65
12 Gandara
10
0.64
0.55
0.6
0.65
0.7
Percent Fidelity
0.75
Combined Scores
by CSA
1
0.9
0.8
0.7
0.6
0.5
0.4
18
20 Lynn Lawrenc
e
5 Cape
Ann
11
6 C and I Framing
ham
10 Fall
River
8 Coastal
28 S
Central
22 New
1
Nat Mean
Bedford Arlington
WFI
0.82
0.86
0.81
0.79
0.76
0.8
0.77
0.8
0.81
0.79
0.77
TOM
0.9
0.85
0.92
0.89
0.91
0.93
0.93
0.83
0.91
0.85
0.77
DRM
0.8
0.8
0.78
0.82
0.82
0.75
0.76
0.82
0.74
0.79
0.74
Combined Scores
by CSA
1
0.9
0.8
0.7
0.6
0.5
0.4
31
2
23 N
Worcest
Attlebor
Central
er E
o
7 CSR
26
4
29
32
21
16
14
Plymout
Cambrid Springfi Worcest
Malden Holyoke Harbor
h
ge
eld
er W
Nat
Mean
WFI
0.76
0.81
0.81
0.74
0.72
0.75
0.8
0.77
0.7
0.82
0.78
0.77
TOM
0.86
0.85
0.86
0.93
0.91
0.91
0.86
0.79
0.85
0.77
0.77
0.77
DRM
0.8
0.76
0.73
0.73
0.74
0.71
0.71
0.8
0.79
0.73
0.76
0.74
Combined Scores
by CSA
1
0.9
0.8
0.7
0.6
0.5
0.4
13
3
25
15
17 Hyde 24 Park
9
Greenfie Brockto Pittsfiel Haverhil
27 RVW
Park
Street Dimock
ld
n
d
l
19
12
30
Lowell Gandara Walden
Nat
mean
WFI
0.76
0.73
0.79
0.74
0.69
0.82
0.75
0.76
0.78
0.82
0.55
0.77
TOM
0.83
0.81
0.82
0.85
0.83
0.75
0.86
0.79
0.79
0.73
0.92
0.77
DRM
0.71
0.76
0.69
0.69
0.76
0.7
0.65
0.69
0.65
0.64
0.66
0.74
Percent improvement by CSA by tool
• WFI top 3 +
– 19 Lowell +.07
– 18 Lawrence +.05
– 25 Pittsfield and 22 New Bedford +.05
• WFI bottom 3 – 4 Cambridge -.09
– 17 Hyde Park -.14
– 30 Walden -.24 *only N = 3 in 2010
Percent improvement by CSA by tool
• TOM top 3 +
– 9 Dimcock +.34
– 8 Coastal + .12
– 5 Cape Ann + .12
– 25 Pittsfield +.12
– 6 C and I +.09
• TOM bottom 3 – 14 Harbor -.06
– 29 Springfield -.10
– 24 Park Street -.11
Percent improvement by CSA by tool
• DRM top 3 +
– 31 Worcester E +.43
– 32 Worcester W + .40
– 5 N. Cape Ann + .38
• DRM bottom 3 – 27 RVW -.10
– 19 Lowell -.13
– 30 Walden -.16