Transcript Ohio Scales Data - mental health recovery

Using Outcomes in the Quality
Improvement Process:
Setting System Level Standards
Erik R. Stewart, Ph.D.
Vice President, System Performance
[email protected]
Renee Kopache, M.S.
Recovery Coordinator
[email protected]
History/Purpose of Outcomes
• Ohio Mental Health Outcomes Task Force (OTF) –
1996
– Charge
• Develop a statewide approach to measuring consumer
outcomes in Ohio’s publicly-supported mental health
system
Vital Signs (1998)
• Outcomes Implementation Pilot Coordinating Group
– Final recommendations were made in 1999.
Adult Outcomes Domains
• Clinical Status
– looks at symptoms that a person may
experience from their illness and how much
they interfere with their daily living.
• Quality of Life
– examines how “good” a person’s life is, and
if their needs are being met. Also
examines how much control a
person has over the events in
their life (empowerment).
Adult Outcomes Domains
• Functional Status
– This domain identifies how well a person is
doing in the community including areas such
as work, school and social relationships.
• Safety & Health
– Addresses how a person is doing physically
and the amount of freedom they have from
psychological harm from self and others.
Adult Outcomes: Instruments
•
Adult Consumer Form A
– This form is used by consumers with a
severe mental illness. (case management)
– The survey consists of four parts: Quality
of Life, Safety and Health, Symptom
Distress and Empowerment.
Adult Outcomes: Instruments
•
Adult Provider Form A
– consists of two parts: Functional
Status and Safety & Health.
– Provider’s observations and clinical
judgments about the consumer’s
social & role functioning, housing
status, activities of daily living,
criminal justice involvement, harmful
behavior and victimization.
Adult Outcomes: Instruments
•
Adult Consumer Form B
– Consists of three parts: Quality of Life
(excluding empowerment, Safety &
Health and Symptom Distress.
– This shorter version of the Adult A form
is used for adult outpatient clients.
Providers of non-SMD consumers are
not required to complete a provider
survey.
Youth Outcomes Domains
• Four primary areas or domains of
assessment were selected for Ohio
Scales:
– Problem severity
– Functioning
– Hopefulness
– Satisfaction with behavioral health
services
Youth Outcomes: Instruments
• Ohio Scales
Three parallel forms:
– Y-form is completed by the youth (selfreport for ages 12 and older).
– P-form is completed by the youth’s parent
(or primary caretaker).
– W-form is completed by the youth’s
agency worker/case manager.
Administration Periods
•
•
•
•
Initial
6 Month
12 Month
Annually thereafter
(Adult Consumer Form B: Initial and
Discharge administrations only)
OCO History in Hamilton County
• Data collection begins – September, 2001
• Regular and varied production reporting
begins at same time
• Production performance abysmal, though
consistent with other participating areas of State
HCCMHB surveys agencies to
discern impediments to production
Agencies React
HCCMHB representatives visit
agencies to discern impediments to
production
Impediments
(in no particular order):
1) Excuses:
•
•
•
•
POV boxes don’t turn on (batteries)
Staff refuse to complete or administer
Multi-million $ agency has only 1 printer
POV box times out too quickly
Impediments
(in no particular order):
2) Collection technology:
• POV data system interface with other agency
data system(s)
Impediments
(in no particular order):
3) Lack of agency/staff “buy-in”
• We know we do good work
• We don’t care about consumer outcomes
Impediments
(in no particular order):
4) Lack of products/information to staff
following administrations (initially)
•
•
•
•
•
•
Staff frustration with initial software
Introduction of ARROW with use limited to POV
Immediate red flags printout
Weekly reports to staff
Consistent supervisor response to problems
Use of data in “assessment update printout”
Impediments
(in no particular order):
5) Diffusion of responsibility (a social phenomenon that
occurs in groups of people when responsibility is not explicitly assigned)
• Request made by QA/Outcomes member
to resurrect regular IT/IS staff meetings at
HCCMHB. Solicitation made. ONE
response.
• Lack of departmental ownership
Impediments
(in no particular order):
6) Agency-specific challenges
•
•
•
•
Central Point of Access
New administration at agency intake?
Paper and pencil administration
Program support staff enter data
QA Approach Introduced
• Quarterly indicator measuring Initial Adult
Provider compliance added to existing
indicators
ODMH Introduces Missing Data
Reports - Summer, 2003
• Hamilton County Performance
– 23% Adult OR/OE
– 34% Youth OR/OE
October, 2003 - HCCMHB and
Agencies Form PI Sub-committee
to Develop Production Logic
February, 2004 – Ohio Department
of Mental Health Presents State
Plan for Upcoming PI
Requirements
PDCA Cycle
Plan
Act
Do
Check
April, 2004 – Proposed Logic for
OCO Production Measures Shared
With Membership
Initial Adult Consumer
Production Analysis
Initial Adult Consumer Production Analysis
• Claims data
compiled for
quarter under
review
UCI
First Svc. Date
Min Age
UPID
Service Type
1000001
7/10/2005
42
10001
CPST
1000001
8/1/2005
42
10002
Counseling
1005002
9/2/2005
28
10001
CPST
2052520
7/20/2005
19
10005
Med Som
2052520
7/25/2005
19
10006
CPST
Initial Adult Consumer Production Analysis
• “Pre-quarter
period” is
determined
by
subtracting
120 days
from first
date of
service in
period
UCI
First Svc. Date
Pre-Quarter Period
1000001
7/10/2005
3/12/2005
1000001
8/1/2005
4/3/2005
1005002
9/2/2005
5/5/2005
2052520
7/20/2005
3/22/2005
2052520
7/25/2005
3/27/2005
Initial Adult Consumer Production Analysis
• Review of
claims
during prequarter
period is
conducted.
Those
exhibiting
claims are
removed
UCI
First Svc. Date
Pre-Quarter Period
Pre-Quarter Claims
1000001
7/10/2005
3/12/2005
No
1000001
8/1/2005
4/3/2005
No
1005002
9/2/2005
5/5/2005
Yes
2052520
7/20/2005
3/22/2005
No
2052520
7/25/2005
3/27/2005
No
Initial Adult Consumer Production Analysis
• Unduplicate
list by
selecting
case with
highest
order
service
UCI
First Svc.
Date
Pre-Quarter
Period
Min
Age
UPID
Service
Type
1000001
7/10/2005
3/12/2005
42
10001
CPST
1000001
8/1/2005
4/3/2005
42
10002
Counseling
2052520
7/20/2005
3/22/2005
19
10005
Med Som
2052520
7/25/2005
3/27/2005
19
10006
CPST
Initial Adult Consumer Production Analysis
• Create date
range for
acceptable
OCO
administrations
UCI
First Svc. Date
1st Svc. Minus 60
1st Svc. Plus 44
1000001
7/10/2005
5/11/2005
8/23/2005
2052520
7/25/2005
5/26/2005
9/7/2005
Initial Adult Consumer Production Analysis
• Merge Adult Consumer A and Adult
Consumer B OCO administrations
• Match by date when acceptable date
exists
• Determine production ratio/percentage
January 2005
Su Mo Tu We Th Fr Sa
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
February 2005
Su Mo Tu We Th Fr Sa
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28
March 2005
Su Mo Tu
1
6 7 8
13 14 15
20 21 22
27 28 29
We
2
9
16
23
30
Th
3
10
17
24
31
Fr
4
11
18
25
Sa
5
12
19
26
Period under review
(4th qtr. FY 2005)
April 2005
Su Mo Tu We Th Fr
1
3 4 5 6 7 8
10 11 12 13 14 15
17 18 19 20 21 22
24 25 26 27 28 29
Sa
2
9
16
23
30
May 2005
Su
1
8
15
22
29
Mo
2
9
16
23
30
Tu
3
10
17
24
31
We
4
11
18
25
Th
5
12
19
26
Fr
6
13
20
27
Sa
7
14
21
28
June 2005
Su Mo Tu We
1
5 6 7 8
12 13 14 15
19 20 21 22
26 27 28 29
Th
2
9
16
23
30
Fr
3
10
17
24
Sa
4
11
18
25
January 2005
Su Mo Tu We Th Fr Sa
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
February 2005
Su Mo Tu We Th Fr Sa
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28
April 2005
Su Mo Tu We Th Fr
1
3 4 5 6 7 8
10 11 12 13 14 15
17 18 19 20 21 22
24 25 26 27 28 29
Sa
2
9
16
23
30
May 2005
Su
1
8
15
22
29
Mo
2
9
16
23
30
Tu
3
10
17
24
31
We
4
11
18
25
Th
5
12
19
26
Fr
6
13
20
27
Sa
7
14
21
28
1st service in period
May 11, 2005
March 2005
Su Mo Tu
1
6 7 8
13 14 15
20 21 22
27 28 29
We
2
9
16
23
30
Th
3
10
17
24
31
Fr
4
11
18
25
Sa
5
12
19
26
June 2005
Su Mo Tu We
1
5 6 7 8
12 13 14 15
19 20 21 22
26 27 28 29
Th
2
9
16
23
30
Fr
3
10
17
24
Sa
4
11
18
25
January 2005
Su Mo Tu We Th Fr Sa
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
February 2005
Su Mo Tu We Th Fr Sa
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28
March 2005
Su Mo Tu
1
6 7 8
13 14 15
20 21 22
27 28 29
We
2
9
16
23
30
Th
3
10
17
24
31
Fr
4
11
18
25
Sa
5
12
19
26
Absence of service determined for
120 day period prior to first service
(ensures client is new or in new
episode of care)
April 2005
Su Mo Tu We Th Fr
1
3 4 5 6 7 8
10 11 12 13 14 15
17 18 19 20 21 22
24 25 26 27 28 29
Sa
2
9
16
23
30
May 2005
Su
1
8
15
22
29
Mo
2
9
16
23
30
Tu
3
10
17
24
31
We
4
11
18
25
Th
5
12
19
26
Fr
6
13
20
27
Sa
7
14
21
28
June 2005
Su Mo Tu We
1
5 6 7 8
12 13 14 15
19 20 21 22
26 27 28 29
Th
2
9
16
23
30
Fr
3
10
17
24
Sa
4
11
18
25
January 2005
Su Mo Tu We Th Fr Sa
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
February 2005
Su Mo Tu We Th Fr Sa
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28
March 2005
Su Mo Tu
1
6 7 8
13 14 15
20 21 22
27 28 29
We
2
9
16
23
30
Th
3
10
17
24
31
Fr
4
11
18
25
Sa
5
12
19
26
Outcome administration
expected/sought between March 12
and June 24
April 2005
Su Mo Tu We Th Fr
1
3 4 5 6 7 8
10 11 12 13 14 15
17 18 19 20 21 22
24 25 26 27 28 29
Sa
2
9
16
23
30
May 2005
Su
1
8
15
22
29
Mo
2
9
16
23
30
Tu
3
10
17
24
31
We
4
11
18
25
Th
5
12
19
26
Fr
6
13
20
27
Sa
7
14
21
28
June 2005
Su Mo Tu We
1
5 6 7 8
12 13 14 15
19 20 21 22
26 27 28 29
Th
2
9
16
23
30
Fr
3
10
17
24
Sa
4
11
18
25
HCCMHB OCO Production Performance
Improvement Measures
• Initial Measures
– Adult Consumer
HCCMHB OCO Production Performance
Improvement Measures
• Initial Measures
–
–
–
–
–
Adult Consumer
Adult Provider
Youth Consumer
Youth Provider
Youth Parent
• Six-Month Measures
–
–
–
–
–
Adult Consumer
Adult Provider
Youth Consumer
Youth Provider
Youth Parent
• 12-Month Measures
–
–
–
–
–
Adult Consumer
Adult Provider
Youth Consumer
Youth Provider
Youth Parent
• Long-Term Measures
–
–
–
–
–
Adult Consumer
Adult Provider
Youth Consumer
Youth Provider
Youth Parent
June, 2004 – Contract Language
Established Obligating Agencies to
70% Compliance on OCO
Administrations
July, 2004 – HCCMHB Issues
First “Test” Measures
PI Products Provided to
Agencies:
UPID
Type
Number
Type
Total
Expected
Submitt
ed
Not
Submitted
Ratio
expectation
Ratiocalc
Ratio
Met
10000
11
Adult Consumer initial
52
43
9
70
82.69230769
yes
10000
12
Adult Provider initial
43
29
14
70
67.44186047
no
10000
13
Child Provider initial
46
39
7
70
84.7826087
yes
10000
14
Child Parent initial
46
40
6
70
86.95652174
yes
10000
15
Child Consumer initial
18
14
4
70
77.77777778
yes
10000
21
Adult Consumer 06 month
20
5
15
70
25
no
10000
22
Adult Provider 6 month
20
4
16
70
20
no
10000
23
Child Provider 6 month
23
18
5
70
78.26086957
yes
10000
24
Child Parent 6 month
23
13
10
70
56.52173913
no
10000
25
Child Consumer 6 month
12
4
8
70
33.33333333
no
10000
31
Adult Consumer 12 month
7
2
5
70
28.57142857
no
10000
32
Adult Provider 12 month
7
3
4
70
42.85714286
no
10000
33
Child Provider 12 month
12
8
4
70
66.66666667
no
10000
34
Child Parent 12 month
12
8
4
70
66.66666667
no
10000
35
Child Consumer 12 month
5
0
5
70
0
no
10000
41
Adult Consumer Annual
274
231
43
70
84.30656934
yes
10000
42
Adult Provider Annual
274
214
60
70
78.10218978
yes
10000
43
Child Provider Annual
62
58
4
70
93.5483871
yes
10000
44
Child Parent Annual
60
57
3
70
95
yes
10000
45
Child Consumer Annual
28
24
4
70
85.71428571
yes
Agency A
100.00%
90.00%
82.76%
80.00%
83.13%
Percent Complete
70.00%
60.00%
50.67%
50.00%
40.00%
30.00%
31.58%
20.00%
10.00%
0.00%
FY 05 Qrt1 Initial Adult
Consumer
FY 05 Qrt2 Initial Adult
Consumer
FY 05 Qrt3 Initial Adult
Consumer
FY 05 Qrt4 Initial Adult
Consumer
Spreadsheet containing:
•
•
•
•
•
•
•
•
•
•
UCI
First service in period date
Last service in period date
OCO time period parameters
Submission status
Date of administration for submission
Age of client
Service type by numeric rank
Instrument type of submission
Admission date
Agencies may formally dispute
findings on a case-specific basis
Measures allow agencies to look closely at
specific cases missing administrations to
discern issues
Had to educate agencies on
intricacies of logic
Tracking based upon most recent
intake date
Had to cut through many “trivial”
issues that impacted few clients but
served as a distraction from dealing
with larger issues (those effecting
many expected administrations)
Had to deal with misunderstandings or
“loose” understandings of the
administration criteria (ex. Use of
“person unable to complete” disposition
for unplanned termination/discharges)
Had to move past focusing upon issues
related to the logic used in the PI
exercise to focus on agency issues
needing attention
Had to manage areas subject to
manipulation (ex. Use of “B” consumer
administrations when “A” is warranted)
….And in the End…..
(Measures of Progress Over Time)
The Heavens Opened!!!!
OC O Pr o d uct io n Pr o g r ess ( Out co mes R eceived / Out co mes Exp ect ed )
100. 00%
86. 10%
90. 00%
76. 60%
80. 00%
76. 30%
68. 44%
70. 00%
64. 98%
61. 68%
60. 00%
51. 57%
47. 67%
50. 00%
43. 00%
40. 00%
39. 60%
37. 80%
33. 60%
30. 00%
20. 00%
10. 00%
0. 00%
4/ 2003-3/ 2004
7/ 2003-6/ 2004
10/ 2003-9/ 2004
Youth
1/ 2004-12/ 2004
4/ 2004-3/ 2005
Adult
7/ 2004-6/ 2005
Measures of Progress
Relative to Other Board Areas
• Additional “learnings”
– 120 day absence without new OCO/tx. Plan
update – not clinically sound practice
– Delayed tx. Following assessment
Data Use
Ohio Scales Data
Ohio Scales
State Data Report
Youth Consumer Findings
Youth Consumer Problem Severity Scores Statewide
(Cross-sectional data)
35
33
31
29
Mean Score
27
25
23.89
23
21
19
18.86
18.68
18.99
18.56
18.96
18.98
18.37
17
15
Initial
90 Days
180 Days
270 Days
1 Year
2 Years
3 Years
4 Years
Youth Parent Findings
Youth Parent Problem Severity Scores Statewide
(Cross-sectional data)
35
33
31
29
28.57
Mean Score
27
25
23.64
23.14
23.12
24.09
23
22.21
22.15
22.98
21
19
17
15
Initial
90 Days
180 Days
270 Days
1 Year
2 Years
3 Years
4 Years
Youth Worker Findings
Youth Worker Problem Severity Scores Statewide
(Cross-sectional data)
35
33
31
29
28
Mean Score
27
25
22.76
23
21
21.54
21.41
21.51
21.14
21.43
21.13
19
17
15
Initial
90 Days
180 Days
270 Days
1 Year
2 Years
3 Years
4 Years
Does Hamilton County data
look like Statewide data?
Youth Consumer Findings
Youth Parent Findings
Youth Parent Functioning Scores Comparison
(Cross-sectional data)
65
Mean Score
60
55
50
49.34
48.95
48.16
48.29
47.9
46.91
46.81
45
45
40
Initial
90 Days
180 Days
270 Days
Statewide Youth
1 Year
Hamilton Youth
2 Years
3 Years
4 Years
Youth Worker Findings
Does a longitudinal design
change the picture?
Youth Consumer Findings
Youth Parent Findings
Youth Parent Functioning Scores Comparison
(Cross-sectional data)
65
Mean Score
60
55
50
49.34
48.95
48.16
48.29
47.9
46.91
46.81
45
45
40
Initial
90 Days
180 Days
270 Days
Statewide Youth
1 Year
Hamilton Youth
2 Years
3 Years
4 Years
Youth Worker Findings
Does looking at everyone
together make sense?
Should the data be apportioned
in some manner when
considering outcomes?
By Diagnosis?
HCCMHB expenditures for
treatment of youth with diagnosis
of 314.01 (AD/HD) from
September 2001 (inception of
OCO/OYS) to ~ July, 2005 =
$14,470,290 and provided care
to 2,652 individuals under the
age of 18.
Summary: Findings for all
groups (Youth, Parents, &
Agency Staff) indicate that
statistically significant gains are
made during the initial period (6
months) of treatment. Findings
did not discern statistically
significant differences beyond
this point.
Examination of CY 2001 data
indicates that of all children with
dx.=314.01, 45% receive greater
than 180 days of treatment. 25%
receive greater than 375 days of
treatment. 11% receive greater
than 2 years of treatment.
Considerable disparity exists
between agencies in regard to
the average period of treatment
for children with this diagnosis.
Ohio Consumer Outcomes
Data Mart
http://mhwwwtest.mh.state.oh.us/Screen1/odmhFirstScreen.jsp
Erik R. Stewart, Ph.D.
Vice President, System Performance
[email protected]
Renee Kopache, M.S.
Recovery Coordinator
[email protected]