Transcript THE WASHINGTON INSTITUTE FOR MENTAL HEALTH RESEARCH & TRAINING Quality of Life Measures: Mental Health Services and Consumer Satisfaction For further information, please contact: Dennis McBride:

THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Quality of Life Measures:
Mental Health Services
and Consumer Satisfaction
For further information, please contact:
Dennis McBride: (253) 756-2335 | [email protected]
Catherine M. Wilson: (253) 761-7573 | [email protected]
Part 1
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer Satisfaction
Surveys
The Mental Health Statistical Improvement Program
(MHSIP)
For further information, please contact:
Dennis McBride: (253) 756-2335 | [email protected]
Catherine M. Wilson: (253) 761-7573 | [email protected]
MHSIP Emerges
 1976 – MHSIP Advisory Group (developed data standards for public
mental health systems)
 1987 – National Research Institute (NRI) to Support MHSIP
 1989 – MHSIP Policy Group (Refined Data Standards for Mental Health
Systems)
 1996 – MHSIP Consumer-Oriented Report Card (Developed base
survey we now use). National Association of State Mental Health
Program Directors (NASMHPD) was created to further develop the
Application of MHSIP.
 1997 – Five State Feasibility Project (refined survey)
 1998-2001 – 16 State Study on Mental Health Performance Measures
(refined survey)
 2001- 2014 – Current Survey
Consumer
Goal & Audiences
Satisfaction
Surveys 2013
The consumer surveys are designed to examine
service quality issues related to Washington State’s
delivery of state-funded mental health services.
Using Survey Data for:
Federal Reporting
Performance Indicator Work Groups
Provider Agencies and RSNs (e.g., EQROs)
Consumer Groups
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Federal Uses
•
•
•
•
Data Infrastructure Grants (DIGS)
National Outcome Measures (NOMS)
Government Performance and Results Act (GPRA)
Uniform Reporting System (URS).
The MHSIP Consumer Survey is now being
implemented in 55 states/territories for the adult
survey and 54 states/territories for youth surveys.
Consumer
Satisfaction
Surveys 2013
Surveys to Meet CMS
Requirements
Survey
Year
Statewide Sample Based Outcomes
Survey
1998
Children with Special Needs
1999
MHSIP Adult Consumer Survey
(10 surveys)
2002-2013
MHSIP Child and Family Consumer
Survey (9 surveys)
2002-2013
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction 2013 MHSIP Survey Samples
Surveys 2013
• Persons receiving at least one hour of service within
a 6 month time frame: May-Oct 2012
Adult Survey: 18+ years old
Youth Survey: 13-20 years old
Family Survey: <13 years old
• Stratified Random Sample (Stratified by RSN, Age,
Minority Status
• Probability Proportionate to Size (Size adjusted for
small RSNs)
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
WIMHRT’s CATI Lab System
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Sources of Data
ProviderOne
Data System
Contact
Information
ACES Barcode data
– Economic Services
Administration
(DSHS)
Research and
Data Analysis
(DSHS)
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Adult Consumer
Survey (ACS)
 Sample
Frame
Selecting Participants
49,746
5,646
 Drawn
Sample
 Respondent
Sample
1,334
Child & Family
Consumer
Survey (CFCS)
 Sample
Frame
 Drawn
Sample
30,311
3,924
924
 Respondent
Sample
F=604
Y=320
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
ACS & CFCS Disposition
Adult
Disposition
N (%)
Family/Youth
Disposition
N (%)
Completions
1344 (23.6)
924 (23.5)
Refusals
647 (11.5)
260 (6.6)
Incorrect Numbers
2073 (36.7)
1015 (25.9)
No MH Services
46 (0.8)
54 (1.4)
Language Barrier
114 (2.0)
9 (0.2)
Unavailable
310 (5.5)
86 (2.2)
Deceased
33 (0.6)
1 (0.03)
No Answer
932 (16.5)
1534 (39.1)
Other
157 (2.8)
41 (1.04)
TOTAL
5646
3924
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
100
90
80
70
60
50
40
30
20
10
0
Response & Cooperation
Rates Over Time
ACS %
Cooperation Rate
ACS % Response
Rate
CFCS %
Cooperation Rate
CFCS % Response
Rate
2004 Data
CFCS % Cooperation Rate N/A
CFCS % Response Rate N/A
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
States’ MHSIP
Completion Rates 2008
State
Sample
Method
Survey
Method
Adult N
Adult %
Y/F N
Y/F %
Texas
Stratified
Random
Mail Out
334
19%
389
18%
Wisconsin*
Random
Mail Out
803
36%
524
39%
Vermont
Random
Mail Out
737
36%
251
20%
Washington Stratified
State
Random
Telephone/M 1500
ail Out
24%
906
33%
Illinois
Random
Mail Out
561
19%
586
17%
Oregon
Total
Population
Mail Out
3240
24%
Combined
Source: Sampling and the MHSIP Consumer Surveys: Techniques, Models, Issues. Mary E.
Smith & Vijay Ganju (June, 2008).
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
*Removes non-contacts from the denominator.
Consumer
Satisfaction
Surveys 2013
2013 Calls & Completions
Total number of attempts/calls: 32,628
ACS: 22,023
CFCS: 10,605
Average number of attempts/calls per
completion: 14.45
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Total Calls & Completions
Total number of attempts/calls: 341,078
ACS: 231,995
CFCS: 109.083
Total number of completions: 24,005
ACS: 15,592
CFCS: 8,413
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Representativeness
ACS
CFCS
50
40
30
20
10
0
Respondent Sample
Drawn Sample
Sample Frame
Age
Hours
Age
Hours
80
60
40
20
0
Female % Minority % Female % Minority %
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction Adult/Youth Perception of Outcomes
Alpha = 0.90
Surveys 2013
As the result of services I received:
1.
2.
3.
4.
5.
6.
I am better at handling daily life.
I get along better with family members.
I get along better with friends and other people.
I am doing better in school and/or work.
I am better able to cope when things go wrong.
I am satisfied with my family life right now.
Scale
Scoring: 1-Strongly Disagree; 2-Disagree; 3-Undecided; 4-Agree; 5-Strongly Agree
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Scale
Scale Construction
ACS
CFCS
General Satisfaction
.87
.92
Appropriateness/Quality of Services
.87
.91
Participation in Treatment Goals
.56
.70
Perceived Outcomes
.91
.90
Perception of Access
.82
.48
Satisfaction with Staff
NA
.84
Cultural Sensitivity of Staff
NA
.84
NOMS Functioning*
.87
NA
NOMS Social Connectedness*
.79
.81
Stigma*
.86
.85
(Youth only)
*Added in 2007
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
ACS Scales
100%
5
4,5
4
Percent Satisfied
75%
3,5
3
50%
25%
Strongly Agree
Agree
2,5
Undecided
2
Disagree
1,5
Strongly Disagree
1
Average Score
0,5
0%
0
General
Satisfaction
Quality of
Service
Participation Perception of Perceived
in Treatment
Access
Outcomes
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
CFCS Scales
100%
5
4,5
4
Percent Satisfied
75%
3,5
3
50%
Agree
2,5
Undecided
2
Disagree
1,5
25%
Strongly Agree
1
Strongly Disagree
Average Score
0,5
0%
General Participation Perceived
Staff
Quality of
Staff
Access to
Satisfaction
in
Outcomes Satisfaction Service
Cultural
Services
Treatment
Competence
0
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Other Relationships
•
•
•
•
•
Demographics and Total Service Hours
Living Situation by Age, Gender, & Ethnicity
Employment Status by Age, Gender, & Ethnicity
Employment Status by Age, Gender, & Ethnicity
Medical/Insurance Information by Age, Gender,
& Ethnicity
• Satisfaction Scales by Age, Gender, & Ethnicity
• Satisfaction Scales by RSN
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
What two things do you like the most
Satisfaction
about the services you received?
Surveys 2013
ACS First Response
General Positive
1%
General Negative
1%
Services
Services
14%
Support
Therapy/Counseling
No Response
21%
Medications
Access
Staff
Environment
1%
Support
21%
Professional Staff
Environment
Professional Staff
22%
No Response
General Positive
General Negative
Access
6%
Staff
1%
Medications
1%
Therapy/Counseling
11%
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Open-Ended Responses:
Adult – Like Most
Support:
”The support that they give me. If I need someone to talk with, they are
there. If not, they call right back. They are good listeners and very
understanding .”
”You are not rejected on account of anything you might be
experiencing/feeling. If there’s one counselor that you’re just not clicking
with, they will rotate you to another. They’ll bear with you and just keep
going until you can identify your feelings and handle them.”
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
What two things do you like the least
Satisfaction
about the services you received?
Surveys 2013
CFCS First Response
General Positive
2%
0%
General Negative
1%
Services
Services
16%
Support
Therapy/Counseling
Medications
No Response
25%
Access
Staff
Support
21%
Professional Staff
No Response
Professional Staff
17%
General Positive
General Negative
Access
7%
Staff
2%
Therapy/Counseling
8%
Medications
1%
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Open-Ended Responses:
Family – Like Least
Access:
”The location and difficulty in getting after school appointments .”
”Long waiting list and it takes months to see a psychiatrist.”
”I liked least that they are short of time. They had so few choices
of time. My child would miss school for appointments.’
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
STIGMA
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction ACS Perceived Stigma by Gender/Age
Surveys 2013
100%
4
90%
3,5
Percent Stigmatized
80%
3
70%
60%
2,5
50%
2
40%
1,5
30%
1
20%
0,5
10%
0%
Most Stigmatized
(3.5-4)
More Stigmatized
(2.5-3.49)
Less Stigmatized
(1.5-2.49)
Least Stigmatized
(1-1.49)
Average Score
Female
Male
18 to <21
21 to <41
41 to <61
61 to <76
0
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
ACS Perceived Stigma by Gender/Ethnicity
Surveys 2013
Consumer
Satisfaction ACS Perceived Service Outcomes by Stigma
Surveys 2013
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
ACS Consumer
Satisfaction Surveys
2012 -13
Correlations: Service Perception
by Stigma
Service Perception Scores
Stigma Scale
Satisfaction with Services
-.124
Appropriateness and quality
of Services
-.175
Participation in Treatment
Goals
-.166
Perceived Outcome of
Service
-.241
Perception of Access to
Service
-.189
NOMS Functioning Scale
-.232
NOMS Social
Connectedness Scale
-.312
THE WASHINGTON INSTITUTE
N = 2578 p < .001
FOR MENTAL HEALTH RESEARCH & TRAINING
Part 2
Trends
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction ACS
Surveys 2013
Perceived Stigma by Year
100%
4
Percent Stigmatized
3,5
75%
3
2,5
50%
2
1,5
25%
Most Stigmatized
(3.5-4)
More Stigmatized
(2.5-3.49)
Less Stigmatized
(1.5-2.49)
Least Stigmatized
(1-1.49)
Average Score
1
0,5
0%
2007
2008
2009
2010
2011
2012
2013
0
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Trends in Mean ACS Satisfaction
Scores Over Time
4,2
4,1
4
3,9
3,8
General Satisfation
Quality of Service
3,7
Participation in Treatment
Perception of Access
3,6
Perceived Outcome
3,5
3,4
3,3
3,2
2002
2004
2006
2007
2008
2009
2010
2011
2012
2013
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Trends in Mean CFCS
Satisfaction Scores Over Time
4,4
4,2
General Satisfation
4
Satisfation with Staff
Perception of Access
Participation in Treatment
Cultural Sensitivity of Staff
3,8
Appropriateness of Services
Perceived Outcome
3,6
3,4
2002
2005
2007
2008
2009
2010
2011
2012
2013
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Functioning/Connectedness
Scores Over Time
4,3
4,2
4,1
4
ACS Social Connectedness
ACS Functioning
3,9
CFCS Social Connectedness
CFCS Functioning
3,8
3,7
3,6
3,5
2007
2008
2009
2010
2011
2012
2013
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
States’ Adult MHSIP Scores 2010
ACS Measures
Texas
Wisconsin Vermont
Washington Illinois
State
Oregon
Access to
Services
78.40%
74.40%
82.20%
68.20%
84.60% 74.00%
Quality/Appropri 83.70%
ateness of
Services
79.60%
84.60%
80.70%
83.20% 82.90%
Outcome of
Services
57.20%
57.20%
69.30%
61.50%
67.30% 57.20%
Participation in
Treatment
71.00%
67.30%
76.90%
71.90%
85.90% 66.80%
Overall
Satisfaction
89.00%
76.70%
84.40%
77.10%
85.70% 80.40%
Source: 2010 CMHS Uniform Reporting System Output Tables
Note: Scores represent the average of those positively agreeing or strongly agreeing with
each scale.
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Summary of Findings: ACS
• Satisfaction scores on most scales remain high, with at least 70% of
consumers on most scales saying they are satisfied or very satisfied
with services.
• Since 2002, satisfaction score trends have shown slight but steady
improvement over time.
• Across years, Perceived Outcomes has remained the lowest, with
more than one third of consumers being undecided or dissatisfied
in this area.
• Consumers continue to be most satisfied with Appropriateness and
Quality of Services.
• Perceived Stigma affects nearly half of consumers and has
remained relatively constant since 2007.
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
Summary of Findings: CFCS
• Satisfaction scores on most scales remain high, with at
least three quarters of consumers on most scales
saying they are satisfied or very satisfied with services.
• Since 2002, all areas of youth and family service
satisfaction have continued to improve.
• Cultural Sensitivity of Staff and Satisfaction with Staff
continue to be the highest rated areas.
• Perceived Outcomes is consistently the lowest rated
area.
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING
Consumer
Satisfaction
Surveys 2013
HTTPS://DEPTS.WASHINGTON.EDU/PBHJP/DEV/PROJECTSPROGRAMS/PAGE/MENTAL-HEALTH-STATISTICS-IMPROVEMENTPROGRAM-CONSUMER-SATISFACTION-SURVEYS
THE WASHINGTON INSTITUTE
FOR MENTAL HEALTH RESEARCH & TRAINING