Measurement for Improvement
Download
Report
Transcript Measurement for Improvement
Measurement for Improvement
For the Practice Change Fellows Program
September 28, 2007
Washington, DC
Dennis A. Ehrich, MD, FACC
Vice President for Medical Affairs
St. Joseph’s Hospital Health Center
Syracuse, New York
Agenda for the Morning
1-Why Do We Measure in Health care?
2-The Model for Improvement
3-Selecting one’s measures
7-Time ordered statistics and understanding variation
8-The run chart
10-Practical exercises
11-The control chart (if time permits)
Why We Measure in Health Care
Measuring for Research
Measuring for Judgment
Measuring for Improvement
Purpose
To discover new
knowledge
To compare to others, to rank
To bring new knowledge into daily
practice
Tests
One large trial
Public reporting quarterly or
with 12 month running
averages
Many sequential, observable tests
Biases
Control for as many as
possible
Severity or risk adjustment
where available
Stabilize the biases from test to test
Data
Gather as much data as
possible, just in case
Reports structure, process or
outcomes
Small tests of significant changes,
accelerates the rate of
improvement
Duration
Can require large numbers
of patients and long
periods of time to obtain
results
Ongoing data collection and
quarterly public reporting
Short iterative cycles in a limited
number of subjects, followed by
spread
The Model for Improvement
Set aims that are measurable, time-specific, and apply to a defined population
Establish measures to determine if a specific change leads to improvement
Select changes most likely to result in improvement
Test the changes
T. Nolan et al. www.ihi.org
The Use of Iterative PDSA Cycles
Implementing the Changes
“Rapid-cycle CQI”
T. Nolan et al. www.ihi.org
Spreading the Change
1-Planning and set-up
2-Spread within the target population
3-Continuous monitoring and
feedback on the spread process
T. Nolan et al. www.ihi.org
Donabedian’s Quality Triangle-It’s
Relevance to Process Improvement
-Avedis Donabedian, MD, MPH (1919-2000)
Donabedian’s Triad
Structure
Organization
People
Equipment/Technology
Process
The actual steps taken in accomplishing the change
Results must be client-focused
Must deliver results reliably
Outcomes
Clinical
Client perception or satisfaction
Financial
Selecting Your Measures
The Three Domains of
Measurement
Structural Measures
Process measures
Outcomes Measures
Balancing measures
Donabedian
The Three Domains of
Measurement
Structural Measures
Describe the environment. How many?
Square footage of a clinical unit
Number of staff
Staff qualifications and competencies
Presence or absence of technology and its characteristics
Process Measures
Process cycle time
The percentage of patients for whom the process achieves its desired
result
Donabedian
The Three Domains of
Measurement
Outcome Measures
The impact of the change initiative on mortality, readmissions to the
hospital, ED visits
The satisfaction scores of clients and staff
The cost per case, average LOS, revenue per case
Balancing Measures
Unintended outcomes that are consequences of the new
program
Unanticipated mortality, morbidity or cost
Has the shifting of resources in an organization compromised
other client or patient populations?
Donabedian
Aim
Selecting A Measure
Operational Definitions
Data Collection Plan
The Quality
Measurement Roadmap
Data Collection
Data Analysis
ACTION
Modified from Lloyd, Robert: “Quality Health Care A Guide to Using Indicators”
Selecting a Measure:
-When selecting a measure, have clarity as to
whether the measure is one of structure, process
or outcome
-And select a balanced panel of indicators that
reflect the dimensions of performance being
evaluated and the change concept(s) being
employed
What Dimension of Performance
do You Want to Measure?
Appropriateness
Availability
Continuity
Effectiveness
Efficiency
Respect and caring
Financial/Viability
Safety
Time lines
Joint Commission (1996)
What Dimension of Performance
do You Want to Measure?
Safety
Effectiveness
Patient-centeredness
Timeliness
Efficiency
Equity
IOM: Crossing the Quality Chasm (2001)
What is the “Change Concept”?
Eliminate waste
Improve work flow
Shorten a waiting list
Change the work environment
Improve the Provider/Client interface
Manage time
Focus on variation
Error proofing a process
Focusing on product or service
The Improvement Guide by Langley, Nolan, Nolan, Norman and Provost. Jossey-Bass
Relating a Change Concept to a
Specific Measure
Concept
Potential Indicators for this process
Patient scheduling
•The average number of days between the call for an
appointment and the actual appointment date
•The percentage of appointments made within 3 days of the
call for an appointment
•The number of appointments scheduled each day
Home care visits
•The number of home care visits
•The average time spent during a home care visit
•The percentage of time spent traveling during each home care
visit
•The number of visits per home care nurse
CQI Training
•The number of participants attending a class
•The percentage of cancellations
•The percentage of no-shows
•The information recall scores at 30 and 60 days
Establishing Operational Definitions That
Are Agreed Upon By All Stakeholders
Operational Definitions
Is clear and unambiguous
Specifies the measurement method, procedures and
equipment when appropriate
Clinical data (chart reviews) vs. administrative data
Client logs vs. a computer database
Define specific criteria for the data to be collected
Define all inclusions and exclusions
For percentages or rates, or ratios, define the criteria for
inclusion in the numerator and denominator
Always ask “How might somebody be confused by this
definition?”
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
Examples of Unclear Definitions
Timely completion of the screening process
A complete medication list
The readmission rate
Medication error
Cost impact
From the acute care hospital
A patient fall
Surgical start time
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
Data Analysis
What descriptive statistics will be used?
Mean, median, mode
Minimum, maximum, range, standard deviation
Quantities, proportions (percentages), rates
How will data be displayed?
Bar chart, histogram, line chart, pie chart, Pareto diagram
Run chart, control chart
External Benchmarking
Joint Commission
CMS
Data Reporting
Data reporting plan
Who will receive the results
How often will they receive the results
How will the data be disseminated?
Printed reports
Email
Dashboard
Spider diagram
Displaying Time-Ordered Statistics
and Understanding Variation
Tools for Displaying Time-ordered Data
Run charts
Plot of data over time with the median of the data set plotted as
a center line
Control charts
Plot of data over time with the mean as the center line and with
upper and lower control limits
Run Charts
Easily constructed by hand or in available spreadsheet
programs
Provides a good idea of improvement in a change initiative
Less sensitive to significant changes (special cause variation)
than the control chart
Control Charts
More sensitive to special cause variation than a run
chart
Requires specialized computer software to create
There are 9 types of control charts used in health care,
depending upon whether the data collected is
distributed normally, is continuous (numerical) or
discreet (attributes) and whether the events measured
are frequent or infrequent
Have their own set of rules to identify special cause
variation
Understanding Variation
All data, collected over time, varies
Random variation (common cause)
The changes occurring are intrinsic to the process being
measured
Non-random variation (special cause)
The changes are being imposed on the system by some external
factor
May be unintended and un anticipated or may be by design
Common Cause (Random)
Variation
Average WBC of Patient With Neutropenia
2.5
2
1.5
1
0.5
0
Value
Median
Special Cause Variation
Creating a Run Chart
Hand-Drawing a Run Chart
Plot data points as a line graph on x-y axes, where “x” is the
increment of time and “y” is the measurement.
Calculate the median value of the data set and draw that line on
the chart
Sort the data from smallest to largest value
Count the data points. That count=N
If “N” is an odd number: Median=N+1/2. Begin counting from
smallest to the largest number. When the count reaches N+1/2, that
number is the median
If “N” is an even number: Median=The average of N/2 and the next
number in the series. Begin counting from smallest to the largest
number. When the count reaches N/2, stop and take the average of
that number and the next number in the series. That average is the
median
Calculating the Median
Odd Number of Data Points
1
N=7
2
2
Median=N+1/2
3
=7+1/2=4
7
The median is the 4th
11
number in the series,
12
which is 3
Even Number of Data Points
1
N=6
3
4
Median=The avg. of the
5
number that is N/2 and the
5
next number in the series.
8
=[4 (the third number in
the series) +5 (the next
number in the series)] /
2=4.5
Balestracci, D., and Barlow, J, Quality Improvement. 1998 Center for Research in Ambulatory Health Care Administration
Definitions
A run is 1 or more consecutive data points on the same side of the median line
A useful observation is one that does not fall on the median line
•Sixteen of the eighteen observations are useful
•There are 10 runs on this run chart
Testing for Special Cause Variation on
a Run Chart
Test 1. Are any runs longer than expected? If so, then that run
represents a special cause.
If there are fewer than 20 useful observations, then 7 or more
data points in a run indicate a special cause.
If there are 20 or more useful observations, then 8 or more data
points in a run indicate a special cause.
Testing for Special Cause Variation on
a Run Chart
Test 2. Is there a trend? A trend is an excessively long series of
consecutive increases or decreases in the data.
Total Number of
Data Points on the Chart
Number of Consecutive
Ascending or Descending Points
Indicating a Special Cause
5 to 8
9 to 20
21 to 100
5 or more
6 or more
7 or more
Applying Tests 1 and 2
Total number of data points=18 Number of useful observations=16
Test 1-Since there are < 20 useful observations it will take ≥ 7 data points in
a run to cause a run to be “too long” defining special cause variation
Test 2-Is there a trend? For 18 total data points, it will take ≥6 consecutive
ascending or descending data points to define a trend.
Testing for Special Cause Variation on
a Run Chart
Test 3. Are there too few or too many runs in the data?
Determine the number of useful observations in your data set.
Use the following table to determine whether the number of
runs in your data are within the expected range. If the number
of runs is above or below the expected range, the data suggest
special cause variation
Expected Number of Runs
Useful
Observations
Lower
Limit
Upper
Limit
Useful
Observations
Lower
Limit
Upper
Limit
15
16
17
18
19
20
21
22
23
24
25
26
27
28
4
5
5
6
6
6
7
7
8
8
9
9
9
10
12
12
13
13
14
15
15
16
16
17
17
18
19
19
29
30
31
32
33
34
35
36
37
38
39
40
10
11
11
11
11
12
13
13
13
14
14
15
20
20
21
22
22
23
23
24
25
25
26
26
Applying Test 3
Are there too many runs? Useful observations= 24. Number of runs= 8. Expected number of runs
= 8-17. Therefore there is no evidence for special cause variation.
Testing for Special Cause Variation on
a Run Chart
Test 4. Fourteen or more points alternating above and below
the median line is a saw tooth pattern indicate a special cause.
KQC=Key Quality
Characteristic
When this pattern is seen, it indicates either that two different
processes are operating at the same time and have been
measured together; in which case stratification of the data
would be helpful. Or, it may indicate tampering