Transcript Consumer Evaluation of Recovery

Consumer Involvement
in Evaluating a Recovery-based
Systems Change Initiative
Vicki Cousins, Director
Office of Consumer Affairs
South Carolina Department of Mental Health
Katherine M. Roberts, MPH
Coordinator, Consumer-to-Consumer Evaluation Team
South Carolina Department of Mental Health
Words to Live By
in South Carolina
“Nothing About Us
Without Us”
Consumer Involvement in Four Key
Areas




Planning
Policy-making
Service Provision
Program Evaluation
Consumer Involvement in Planning
and Policy-making
 Central Administration
Offices
 Regional Service
Sites
(Community Mental
Health Centers in 17
locations serving 46
counties)
Consumer Involvement in Planning
and Policy-making
New SCDMH
consumer-centered
policies/directives
being developed
as a result of
our system’s
emphasis on
recovery.
Consumer Involvement
in Service Provision
Following Larry Fricks’ Georgia model,
South Carolina has developed a Medicaid-billable
service description for peer support
 40-hr. certification training for consumers to
become credentialed as “CPSS”
 Includes oral and written examination
 Final approval from DHSS expected July 1,
2003
 10 graduates to date; trainings to be year twice
a year
Consumer Involvement in
Program Evaluation
SCDMH Consumer-to-Consumer Evaluation Team
Several Years in the Making
1996 …
… 2002
Found our place in the system’s
recovery initiative
The Purpose of
Consumer Involvement
Surgeon General’s Report
1999
Consumer staff, clients, and
the mental health system
all benefit from consumer
involvement.
The Purpose of
Consumer Involvement
a Mental Health System
Surgeon General’s Report 1999
Consumer Staff
 meaningful activity or
employment
 role models of recovery
 enhance sensitivity of
the service system
The Purpose of
Consumer Involvement
a Mental Health System
Service system benefits:



Increases in service satisfaction as
related to people’s recovery needs
Reduced hospitalizations
Cost-savings in community program
operating costs
(Chamberlin & Rogers, 1990; Carpinello & Knight, 1993)
SCDMH Consumer-to-Consumer
Evaluation Team
Project Mission
To involve individuals who receive services, family members, independent
advocates and citizens-at-large in program evaluation and continuous quality
improvement of the South Carolina Department of Mental Health.
SCDMH Consumer-to-Consumer
Evaluation Team
 Office of Consumer Affairs, Division of Health
Care Reform
 Consumer-driven evaluation survey process
 Primary consumer team members
 Paid employees of SCDMH
 Participates in instrument development
 Conducts interviews, compiles data, and
produces reports
History of Consumer Involvement in
Evaluation
 For several years in the late 1990s, the Team
conducted open-ended surveys with over 900
clients, striving to evaluate individual CMHC
programs.
 Starting in 2002, a fast evolution began as the
emphasis shifted from program evaluation to
system evaluation. Survey questions became
closed ended.
History of Recovery Measurement in
South Carolina
 March 2002: Formal planning process for
recovery system evaluation begins.
Expert researchers partner with the CCET.
 June 2002: Consumer focus groups, staff
surveys
 August–October 2002: Surveying to test
for validity and reliability
Instruments
 MHSIP
 Consumer-to-Consumer Evaluation Scale
(CCET)
 Recovery Questionnaire
South Carolina Consumers
Define Recovery
Focus Groups
 What does it mean to be in recovery or
recovered from mental illness?
 What helps a person recover from mental
illness?
 What prevents a person from recovering from
mental illness?
Recovery Questionnaire Domains
(10) from the Focus Groups
 Hope - Self-esteem - Housing
 Employment - Stigma - Self-management Skills
 Empowerment/Independence/Self
Responsibility/Control
 Relationships - Social
 Relationships - Family
 Spirituality/Higher Power/Religion
Sample Size
 Plan A:
Sample consisted of 50 adult clients with
open cases in all 17 community mental health centers.
Expectation: 25 (425) would complete the initial survey
and half of that number would agree to be resurveyed 2
to 4 weeks later.
Plan A Results: <10% available to participate.
 Plan B:
Asked for available adult clients with open
cases to volunteer.
Method
 Informed consent
attained.
 Initial face-to-face
interviews conducted by
CCET team members.
 Follow-up surveys were
conducted either face-toface or by phone. The
process was the choice of
the client.
Incentives for Client Participation
 Each client received a button, a $5
McDonalds gift certificate, and a certificate
of appreciation signed by the SC State
Mental Health Director, George P. Gintoli,
for the initial interview.
 On the second interview, clients received 2
more buttons and an additional gift
certificate.
Process
 400+ clients were surveyed/resurveyed from
mid-August through mid-October 2002.
 Scannable forms were reviewed and prepared
for data entry.
 All analysis was conducted in a de-identified
manner.
Analysis
The psychometric properties of the MHSIP,
the Recovery Questionnaire, and the
Consumer-to-Consumer Evaluation Scale
included:
 Item-to-Scale Correlations
 Test-Retest Reliability
Results
 Based on the research, changes were made to the
Recovery Questionnaire and to the Consumer-toConsumer Evaluation Scale.
 The Recovery Questionnaire is presently being used by
SC Share in South Carolina to evaluate their “Recovery for
Life” program.
 Plans are underway by South Carolina’s Columbia Area
MH Center, the largest community mental health center, to
utilize the Recovery Questionnaire in June 2003.
Lessons Learned
 Challenging to go to a full-system survey approach in a
limited time period – 2 times in 8 weeks - from
individual program site assignments.
 Dramatic increase in consumer hours, conflicted with
consumers’ Medicaid benefits, stressed Team
Members.
 But still … state car break-downs, a crash,
getting lost, getting sick, losing surveyors
… It was a terrific learning experience with
positive outcomes. We survived it all.
Consumer Involvement in Evaluating
a Recovery-based Systems Change Initiative
Vicki Cousins, Director
Office of Consumer Affairs
South Carolina Department of Mental Health
Katherine M. Roberts, MPH
Coordinator, Consumer-to-Consumer Evaluation Team
South Carolina Department of Mental Health