Transcript Measurement for Improvement

Measurement for Improvement
For the Practice Change Fellows Program
September 28, 2007
Washington, DC
Dennis A. Ehrich, MD, FACC
Vice President for Medical Affairs
St. Joseph’s Hospital Health Center
Syracuse, New York
Agenda for the Morning
1-Why Do We Measure in Health care?
2-The Model for Improvement
3-Selecting one’s measures
7-Time ordered statistics and understanding variation
8-The run chart
10-Practical exercises
11-The control chart (if time permits)
Why We Measure in Health Care
Measuring for Research
Measuring for Judgment
Measuring for Improvement
Purpose
To discover new
knowledge
To compare to others, to rank
To bring new knowledge into daily
practice
Tests
One large trial
Public reporting quarterly or
with 12 month running
averages
Many sequential, observable tests
Biases
Control for as many as
possible
Severity or risk adjustment
where available
Stabilize the biases from test to test
Data
Gather as much data as
possible, just in case
Reports structure, process or
outcomes
Small tests of significant changes,
accelerates the rate of
improvement
Duration
Can require large numbers
of patients and long
periods of time to obtain
results
Ongoing data collection and
quarterly public reporting
Short iterative cycles in a limited
number of subjects, followed by
spread
The Model for Improvement
Set aims that are measurable, time-specific, and apply to a defined population
Establish measures to determine if a specific change leads to improvement
Select changes most likely to result in improvement
Test the changes
T. Nolan et al. www.ihi.org
The Use of Iterative PDSA Cycles
Implementing the Changes
“Rapid-cycle CQI”
T. Nolan et al. www.ihi.org
Spreading the Change
1-Planning and set-up
2-Spread within the target population
3-Continuous monitoring and
feedback on the spread process
T. Nolan et al. www.ihi.org
Donabedian’s Quality Triangle-It’s
Relevance to Process Improvement
-Avedis Donabedian, MD, MPH (1919-2000)
Donabedian’s Triad
 Structure
 Organization
 People
 Equipment/Technology
 Process
 The actual steps taken in accomplishing the change
 Results must be client-focused
 Must deliver results reliably
 Outcomes
 Clinical
 Client perception or satisfaction
 Financial
Selecting Your Measures
The Three Domains of
Measurement
 Structural Measures
 Process measures
 Outcomes Measures
 Balancing measures
Donabedian
The Three Domains of
Measurement
 Structural Measures
 Describe the environment. How many?
 Square footage of a clinical unit
 Number of staff
 Staff qualifications and competencies
 Presence or absence of technology and its characteristics
 Process Measures
 Process cycle time
 The percentage of patients for whom the process achieves its desired
result
Donabedian
The Three Domains of
Measurement
 Outcome Measures
 The impact of the change initiative on mortality, readmissions to the
hospital, ED visits
 The satisfaction scores of clients and staff
 The cost per case, average LOS, revenue per case
 Balancing Measures
 Unintended outcomes that are consequences of the new
program
 Unanticipated mortality, morbidity or cost
 Has the shifting of resources in an organization compromised
other client or patient populations?
Donabedian
Aim
Selecting A Measure
Operational Definitions
Data Collection Plan
The Quality
Measurement Roadmap
Data Collection
Data Analysis
ACTION
Modified from Lloyd, Robert: “Quality Health Care A Guide to Using Indicators”
Selecting a Measure:
-When selecting a measure, have clarity as to
whether the measure is one of structure, process
or outcome
-And select a balanced panel of indicators that
reflect the dimensions of performance being
evaluated and the change concept(s) being
employed
What Dimension of Performance
do You Want to Measure?
 Appropriateness
 Availability
 Continuity
 Effectiveness
 Efficiency
 Respect and caring
 Financial/Viability
 Safety
 Time lines
Joint Commission (1996)
What Dimension of Performance
do You Want to Measure?
 Safety
 Effectiveness
 Patient-centeredness
 Timeliness
 Efficiency
 Equity
IOM: Crossing the Quality Chasm (2001)
What is the “Change Concept”?
 Eliminate waste
 Improve work flow
 Shorten a waiting list
 Change the work environment
 Improve the Provider/Client interface
 Manage time
 Focus on variation
 Error proofing a process
 Focusing on product or service
The Improvement Guide by Langley, Nolan, Nolan, Norman and Provost. Jossey-Bass
Relating a Change Concept to a
Specific Measure
Concept
Potential Indicators for this process
Patient scheduling
•The average number of days between the call for an
appointment and the actual appointment date
•The percentage of appointments made within 3 days of the
call for an appointment
•The number of appointments scheduled each day
Home care visits
•The number of home care visits
•The average time spent during a home care visit
•The percentage of time spent traveling during each home care
visit
•The number of visits per home care nurse
CQI Training
•The number of participants attending a class
•The percentage of cancellations
•The percentage of no-shows
•The information recall scores at 30 and 60 days
Establishing Operational Definitions That
Are Agreed Upon By All Stakeholders
Operational Definitions
 Is clear and unambiguous
 Specifies the measurement method, procedures and
equipment when appropriate
 Clinical data (chart reviews) vs. administrative data
 Client logs vs. a computer database
 Define specific criteria for the data to be collected
 Define all inclusions and exclusions
 For percentages or rates, or ratios, define the criteria for
inclusion in the numerator and denominator
 Always ask “How might somebody be confused by this
definition?”
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
Examples of Unclear Definitions
 Timely completion of the screening process
 A complete medication list
 The readmission rate
 Medication error
 Cost impact
 From the acute care hospital
 A patient fall
 Surgical start time
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
Data Analysis
 What descriptive statistics will be used?
 Mean, median, mode
 Minimum, maximum, range, standard deviation
 Quantities, proportions (percentages), rates
 How will data be displayed?
 Bar chart, histogram, line chart, pie chart, Pareto diagram
 Run chart, control chart
External Benchmarking
Joint Commission
CMS
Data Reporting
 Data reporting plan
 Who will receive the results
 How often will they receive the results
 How will the data be disseminated?
 Printed reports
 Email
 Dashboard
 Spider diagram
Displaying Time-Ordered Statistics
and Understanding Variation
Tools for Displaying Time-ordered Data
 Run charts
 Plot of data over time with the median of the data set plotted as
a center line
 Control charts
 Plot of data over time with the mean as the center line and with
upper and lower control limits
Run Charts
 Easily constructed by hand or in available spreadsheet
programs
 Provides a good idea of improvement in a change initiative
 Less sensitive to significant changes (special cause variation)
than the control chart
Control Charts
 More sensitive to special cause variation than a run
chart
 Requires specialized computer software to create
 There are 9 types of control charts used in health care,
depending upon whether the data collected is
distributed normally, is continuous (numerical) or
discreet (attributes) and whether the events measured
are frequent or infrequent
 Have their own set of rules to identify special cause
variation
Understanding Variation
 All data, collected over time, varies
 Random variation (common cause)
 The changes occurring are intrinsic to the process being
measured
 Non-random variation (special cause)
 The changes are being imposed on the system by some external
factor
 May be unintended and un anticipated or may be by design
Common Cause (Random)
Variation
Average WBC of Patient With Neutropenia
2.5
2
1.5
1
0.5
0
Value
Median
Special Cause Variation
Creating a Run Chart
Hand-Drawing a Run Chart
 Plot data points as a line graph on x-y axes, where “x” is the
increment of time and “y” is the measurement.
 Calculate the median value of the data set and draw that line on
the chart
 Sort the data from smallest to largest value
 Count the data points. That count=N
 If “N” is an odd number: Median=N+1/2. Begin counting from
smallest to the largest number. When the count reaches N+1/2, that
number is the median
 If “N” is an even number: Median=The average of N/2 and the next
number in the series. Begin counting from smallest to the largest
number. When the count reaches N/2, stop and take the average of
that number and the next number in the series. That average is the
median
Calculating the Median
Odd Number of Data Points
1
N=7
2
2
Median=N+1/2
3
=7+1/2=4
7
The median is the 4th
11
number in the series,
12
which is 3
Even Number of Data Points
1
N=6
3
4
Median=The avg. of the
5
number that is N/2 and the
5
next number in the series.
8
=[4 (the third number in
the series) +5 (the next
number in the series)] /
2=4.5
Balestracci, D., and Barlow, J, Quality Improvement. 1998 Center for Research in Ambulatory Health Care Administration
Definitions
 A run is 1 or more consecutive data points on the same side of the median line
 A useful observation is one that does not fall on the median line
•Sixteen of the eighteen observations are useful
•There are 10 runs on this run chart
Testing for Special Cause Variation on
a Run Chart
Test 1. Are any runs longer than expected? If so, then that run
represents a special cause.
 If there are fewer than 20 useful observations, then 7 or more
data points in a run indicate a special cause.
 If there are 20 or more useful observations, then 8 or more data
points in a run indicate a special cause.
Testing for Special Cause Variation on
a Run Chart
Test 2. Is there a trend? A trend is an excessively long series of
consecutive increases or decreases in the data.
Total Number of
Data Points on the Chart
Number of Consecutive
Ascending or Descending Points
Indicating a Special Cause
5 to 8
9 to 20
21 to 100
5 or more
6 or more
7 or more
Applying Tests 1 and 2
Total number of data points=18 Number of useful observations=16
Test 1-Since there are < 20 useful observations it will take ≥ 7 data points in
a run to cause a run to be “too long” defining special cause variation
Test 2-Is there a trend? For 18 total data points, it will take ≥6 consecutive
ascending or descending data points to define a trend.
Testing for Special Cause Variation on
a Run Chart
Test 3. Are there too few or too many runs in the data?
 Determine the number of useful observations in your data set.
 Use the following table to determine whether the number of
runs in your data are within the expected range. If the number
of runs is above or below the expected range, the data suggest
special cause variation
Expected Number of Runs
Useful
Observations
Lower
Limit
Upper
Limit
Useful
Observations
Lower
Limit
Upper
Limit
15
16
17
18
19
20
21
22
23
24
25
26
27
28
4
5
5
6
6
6
7
7
8
8
9
9
9
10
12
12
13
13
14
15
15
16
16
17
17
18
19
19
29
30
31
32
33
34
35
36
37
38
39
40
10
11
11
11
11
12
13
13
13
14
14
15
20
20
21
22
22
23
23
24
25
25
26
26
Applying Test 3
Are there too many runs? Useful observations= 24. Number of runs= 8. Expected number of runs
= 8-17. Therefore there is no evidence for special cause variation.
Testing for Special Cause Variation on
a Run Chart
Test 4. Fourteen or more points alternating above and below
the median line is a saw tooth pattern indicate a special cause.
KQC=Key Quality
Characteristic
When this pattern is seen, it indicates either that two different
processes are operating at the same time and have been
measured together; in which case stratification of the data
would be helpful. Or, it may indicate tampering