Transcript Achieving Our Mission The Role of Continuous Quality Improvement in Community Corrections and Public Safety Kimberly Gentry Sperber, Ph.D.

Achieving Our Mission
The Role of Continuous Quality Improvement in
Community Corrections and Public Safety
Kimberly Gentry Sperber, Ph.D.
Efforts To Date
“What Works” Literature
– Principles of Effective Interventions
– Growing evidence based on individual program
evaluations and meta-analyses
Continuing Gap Between Science and Practice
– Few programs score as satisfactory on CPAI
Why Isn’t “It” Working?
Latessa, Cullen, and Gendreau (2002)
Article notes 4 common failures of correctional
programs:
– Failure to use research in designing programs
– Failure to follow appropriate assessment and
classification practices
– Failure to use effective treatment models
– Failure to evaluate what we do
CPAI Data as Evidence of Fidelity
Lowenkamp and Latessa (2005)
Examined data from 38 residential correctional
programs for adults
Looked at relationship between program fidelity and
program effectiveness.
Program fidelity was assessed using the CPAI.
Found significant correlation between fidelity and
effectiveness
– CPAI scores correlated to reincarceration
Lowenkamp and Latessa Findings
Continued
Differences in recidivism rates based on CPAI
scores:
– Scores of 0-49% demonstrated 1.7% reduction
compared to comparison group.
– Scores of 50-59% demonstrated 8.1% reduction.
– Scores of 60-69% demonstrated 22% reduction.
CPAI Data Continued
Holsinger (1999)
Examined data from Adolescent Community
Correctional Facilities in Ohio
Looked at relationship between program fidelity and
program effectiveness.
Program fidelity was assessed using the CPAI.
Outcome measures examined included any court
contact, felony or misdemeanor, felony, personal
offense, and commitment to a secure facility
CPAI Data Continued
Total composite score significantly correlated with all
outcome measures.
Each individual domain of the CPAI also significantly
correlated with all of the outcomes
–
–
–
–
–
Program Implementation
Client Assessment
Program Characteristics
Staff Quality
Evaluation
Recent CPAI Results
Results from more than 550 programs:
–
–
–
–
7% HIGHLY EFFECTIVE
18% EFFECTIVE
33% NEEDS IMPROVEMENT
42% INEFFECTIVE
More Fidelity Research
Landenberger and Lipsey (2005)
– Brand of CBT didn’t matter but quality of implementation did.
– Implementation defined as low dropout rate, close monitoring of quality and fidelity,
and adequate training for providers.
Schoenwald et al. (2003)
– Therapist adherence to the model predicted post-treatment reductions in problem
behaviors of the clients.
Henggeler et al. (2002)
– Supervisors’ expertise in the model predicted therapist adherence to the model.
Sexton (2001)
– Direct linear relationship between staff competence and recidivism reductions.
More Fidelity Research Cont’d.
Schoenwald and Chapman (2007)
– A 1-unit increase in therapist adherence score predicted 38% lower rate of
criminal charges 2 years post-treatment
– A 1-unit increase in supervisor adherence score predicted 53% lower rate of
criminal charges 2 years post-treatment.
Schoenwald et al. (2007)
– When therapist adherence was low, criminal outcomes for substance abusing
youth were worse relative to the outcomes of the non-substance abusing
youth.
Washington State Example
(Barnoski, 2004)
• For each program (FFT and ART), an equivalent
comparison/control group was created
• Felony recidivism rates were calculated for each of three
groups, for each of the programs
• Youth who received services from therapists deemed ‘competent’
• Youth who received services from therapists deemed ‘not competent’
• Youth who did not receive any services (control group)
Functional Family Therapy Results:
% New Felony
70
60
50
40
32
10
27
25
30
20
FFT Not Competent
Control group
19
13
9 6
FFT Competent
17
11
0
6 Months
12 Months
18 Months
Results calculated using multivariate models in order to control for potential differences between groups
Washington State Study Continued
When FFT was delivered competently, the program reduced
felony recidivism by 38%
When considering how much the program costs, substantial
savings in ‘avoided crime’ were observed – particularly for the
competent therapists
When ART was competently delivered, felony recidivism was
reduced by 24%
Also resulted in substantial savings
Project Greenlight
Short-term prison-based reentry program in New York
–
–
–
–
–
–
–
–
–
CBT Skills Training
Employment Services
Housing Services
Drug Education and Awareness
Family Counseling
Practical Skills Training
Community-Based Networks
Familiarity With Parole
Individualized Release Plans
Project Greenlight Benefits
Participants received more service referrals
Participants reported more contacts with community
services after release
Participants demonstrated significantly more familiarity
with parole conditions
Participants were more positive about parole
But Did It Work?
34.1
35
30
26.8
24.2
25
20
15
17.2
14.4 13
Greenlight
No Services
10
Existing Trans
Services
5
0
Arrests at 6 Months
Arrests at 12
Months
What Went Wrong?
Violation of the risk principle
– Ceased use of risk assessment instrument when staff deemed process
too cumbersome
Violation of the need principle
– All offenders received same services whether needed or not
Violation of the fidelity principle
– Staff modified delivery of the CBT curriculum (shortened the duration,
increased frequency, increased class size)
Differential staff competence
– Certain case managers produced worse outcomes
UC Halfway House/CBCF Study in Ohio: A Look
at Fidelity Statewide
Average Treatment Effect was 4% reduction in
recidivism
Lowest was a 41% Increase in recidivism
Highest was a 43% reduction in recidivism
Programs that had acceptable termination rates, had been in operation for 3
years or more, had a cognitive behavioral program, targeted criminogenic
needs, used role playing in almost every session, and varied treatment and
length of supervision by risk had a 39% reduction in recidivism
2010 UC Halfway House/CBCF Study in Ohio:
Adherence to CBT in Groups and Changes in Recidivism
7
6
5
4
3
2
1
0
-1
-2
-1
0
1
What Do We Know About Fidelity?
Fidelity is related to successful outcomes (i.e.,
recidivism reductions).
Poor fidelity can lead to null effects or even
iatrogenic effects.
Fidelity can be measured and monitored.
Fidelity cannot be assumed.
Monitoring Fidelity Through a CQI
Process
QA versus CQI
CQI – What Is It?
Infrastructure
Peer Review
Indicators
Client Satisfaction
Action Planning
Process Evaluation
Outcome Evaluation
Benefits
What Is Quality?
Services are based on current professional
knowledge.
Services produce desirable outcomes.
CQI – What Is It?
A method of continuously examining processes and
making them better.
Key principles:
– Use of data and team approaches to improve decision
making
– Involvement of entire organization to improve quality
– Strong focus on customers
– Continuous improvement of all processes and outcomes
CQI versus QA
QA:
– Retrospective review process
– Emphasis on regulatory and contract compliance
– Catching people being bad leads to hide and seek
behavior
CQI versus QA
CQI:
– CQI is a prospective process
– Holds quality as a central priority within the organization
– Focus on customer needs; relies on feedback from internal
and external customers
– Emphasizes systematic use of data
– Not blame-seeking
– Trust, respect, and communication
– Move toward staff responsibility for quality , problem solving
and ownership of services
Objectives of CQI
To facilitate the Agency’s mission
To ensure appropriateness of services
To improve efficiency of services/processes
To improve effectiveness of directing services to client
needs
To foster a culture of learning
To ensure compliance with funding and regulatory
standards
Building a CQI Process
Formal infrastructure
Core Elements
– Documentation Review
– Indicators
• Process Versus Outcome
• Performance Goals
• Action Planning
– Customer Satisfaction
• Clients, Staff, Stakeholders
– Program Evaluation
Process Evaluation
Sample Projects
Process Evaluation
Are we serving our target population?
Are the services being delivered?
Did we implement the program as designed (tx
fidelity)?
Are there areas that need improvement?
Example 1
Review of LSI Scores
Reviewed all open cases at Facility A
Recorded LSI risk category, UC Risk category,
and name of interviewer
77.5% of cases reviewed did not have a match
between staff rating and UC rating
LSI Scores Post-Training
60
40
Weeks 1-2
20
0
Weeks 3-6
Match
Off By 1
Off By 2 or More
First 2 weeks after training – 0 matches
3-6 weeks after training – 46.2% matched
First 2 weeks after training – 50% were off by 2 risk categories
3-6 weeks after the training – 0% were off by 2 risk categories
Example2
CBIT Site Assessments
Cognitive Behavioral Implementation Team
Site visits for observation and rating
Standardized assessment process
Standardized reports back to sites
Combination of quantitative data and qualitative
data
Example 3
Review of Core Correctional Practices
Reviewed 6 programs
Live observation of treatment groups for use of core
correctional practices:
–
–
–
–
–
–
–
–
Anticriminal Modeling
Effective Reinforcement
Effective Disapproval
Problem-Solving
Structured Skill Building
Effective Use of Authority
Cognitive Restructuring
Relationship Skills
Documented strengths and opportunities for improvement
Example 4
Process Evaluation of 2 Adolescent Residential Programs
Collecting data on 128 youth admitted during FY10 and FY11 across 2
boys’ programs.
Goal – to identify and quantify changes in population and programming
in order to identify any necessary changes to service delivery.
Advantages:
– Allows the agency to gain an accurate assessment of current target
population and treatment needs
– Allows the agency to seek out evidence-based practices that best align
with current treatment needs of clients
– Positions the agency for future outcome evaluation
Example 5
Assessing Best Practices at 17 Sites
Use of ICCA Treatment Survey to establish
baseline
Complete again based on best practice
Perform Gap Analysis
Action Plan
Reassess
Outcome Evaluation
Sample Projects
Outcome Evaluation
Are our services effective?
Do clients benefit (change) from the services?
Intermediate outcomes
– Reduction in risk
– Reduction in antisocial values
Long-term outcomes
– Recidivism
– Sobriety
Example 1
Off-Site Non-Emergency Medical Visits for 6 Months
Goals of the project:
– Reduce the overall number of off-site visits for nonemergency medical care.
– Reduce the number of staff-escorted visits (impacts
dollars and coverage)
– Reduce the amount of behavioral treatment missed
– Increase the number of referrals for primary care
upon discharge
Example 1 Continued
Outcomes Data
80
70
60
50
40
6 months
12 months
30
20
10
0
Reduction in
Visits
Cost
Reductions
Reduction in
Missed Tx
Sample Cost Comparison
ER versus On-Site Care
ER VISIT COSTS
$ 400 Average Visit
$ 34 Average Staff Cost
$$$ Hospital Pharmacy
TOTAL
$ 434 not including pharmacy
savings
TALBERT HOUSE
$ 62.50 Average Visit
$ 0 off-site staff cost
Medication savings: samples, patient
assistance programs
TOTAL
$62.50 average cost/visit
Savings of $371.50/visit
Example 2
Outcomes Pre/Post TFM Implementation
Passages Program for Girls
Table 1. Chi-Square Significant Differences
% of Cases
Successful Program Completion
Reduced Endorsement of Antisocial Attitudes
Increased Self-Esteem
Increased Self-Efficacy
Positive Drug Test While in Residential Program
Positive Drug Test Post-Discharge
New Offense Post-Discharge
Technical Violations Post-Discharge
*Sig. at the .05 level. (p<.05)
Pre-TFM
TFM
χ2
p
74.50%
80.00%
70.30%
58.70%
3.10%
20.40%
32.80%
64.1%
56.80%
94.40%
81.00%
90.50%
1.20%
2.50%
25.00%
15.00%
6.234
2.06
0.907
7.143
0.677
13.239
0.435
14.705
0.013*
0.151
0.341
0.008*
0.411
.000*
0.509
.000*
Example 3
Outcome Evaluation of Enhanced Outpatient
Services
3 year SAMHSA grant to enhance drug court OP
program
Sample of 357 clients
Assessing intermediate outcomes (6 months
post-intake)
Assessing long-term outcomes (12 months post
discharge)
Comparing to clients receiving services prior to
enhancements
Benefits of Program Evaluation
Proof of effective services
• Maintain or secure funding
• Improve staff morale and retention
• Educate key stakeholders about services
Highlights opportunities for improvement
Data to inform quality improvement initiatives
Establish/enhance best practices
Monitor/ensure treatment fidelity
The Role of QA/QI in Community Corrections
(based on UC Halfway House and CBCF study)
% Change in Recidivism
8
6
6
4
2
1
0
Internal QA
No Internal QA
NPC Research on Drug Courts
Drug Court Uses Evaluation Feedback to Make Modifications
Percent Improvement in Outcome Costs*
50%
44%
40%
30%
20%
11%
10%
0%
Yes
N=4
No
N=6
* "Percent improvement in outcome costs" refers to the percent savings for
drug court compared to business-as-usual
Conclusions
Many programs are not implementing the principles of effective
intervention with strong fidelity.
Result is an ongoing gap between science and practice.
This gap often results in null or even iatrogenic effects.
Correctional organizations have a responsibility to ensure
effective services.
Monitoring fidelity is key to success.
Responsibility for EBP needs to be aligned at all levels –
administration, management, line staff.
Need to focus on creating formal infrastructure to support and
sustain evidence-based practices.