Transcript Slide 1

The Importance of Measurement
in Health Care
For the Practice Change Fellows Program
September 25, 2008
Washington, DC
Dennis A. Ehrich, MD, FACC
Vice President for Medical Affairs
St. Joseph’s Hospital Health Center
Syracuse, New York
Agenda for the Afternoon
1-Why we measure in health care?
2-The Model for Improvement
3-Selecting one’s measures
4-Time ordered statistics and understanding
variation
5-Displaying and tracking results
6-Deciding whether To design a new process or
improve an existing process
Why We Measure in Health Care
Measuring for
Research
Measuring for
Judgment
Measuring for
Improvement
Purpose
To discover new
knowledge
To compare to others,
to rank
To bring new knowledge
into daily practice
Tests
One large trial
Public reporting
quarterly or with 12
month running
averages
Many small, sequential,
observable tests
Bias
Control for as many
as possible
Severity or risk
adjustment where
available
Stabilize the biases from
test to test
Data
Gather as much data
as possible, just in
case
Measures structure,
process or outcomes
Usually applied to process
Duration
Can require large
numbers of patients
and long periods of
time to obtain
results
Ongoing data collection Short iterative cycles in a
and periodic public
limited number of subjects,
reporting
followed by spread
The Model for Improvement
Set aims that are measurable, time-specific, and apply to a defined
population
Establish measures to determine if a specific change leads to
improvement
Select changes most likely to result in improvement
Test the changes
T. Nolan et al. www.ihi.org
The Use of Iterative PDSA Cycles
“Rapid-cycle CQI”
T. Nolan et al. www.ihi.org
Implementing the
Changes
Multiple Simultaneous Tests of Change
Spreading the Change
1-Executive sponsorship
2-Planning and set-up
3-Spread within the target
population-social network theory
4-Continuous monitoring and
feedback during the spread
process
5-Capturing and sharing
organizational learning
T. Nolan et al. www.ihi.org
Donabedian’s Quality Triangle-It’s
Relevance to Process Improvement
-Avedis Donabedian, MD, MPH (1919-2000)
Donabedian’s Triad
 Structure
 Organization
 People
 Equipment/Technology
 Process
 The steps taken in accomplishing the change and achieving the
outcome
 Results must be client-focused
 Must deliver results reliably
 Outcomes
 Clinical (mortality, complications)
 Client perception or satisfaction
 Financial
Selecting Your Measures
The Three Domains of Measurement
• Structural Measures
• Process measures
• Outcomes Measures
– Balancing measures
Donabedian
The Three Domains of Measurement
• Structural Measures
–
–
–
–
–
Describe the environment. How many?
Square footage of a clinical unit
Number of staff
Staff qualifications and competencies
Presence or absence of technology and its
characteristics
• Process Measures
• Process cycle time
• The percentage of patients for whom the process achieves
its desired result
Donabedian
The Three Domains of Measurement
• Outcome Measures
• The impact of the change initiative on mortality,
readmissions to the hospital, ED visits
• The satisfaction scores of clients and staff
• The cost per case, average LOS, revenue per case
• Balancing Measures
– Unintended outcomes that are consequences of the
new program
– Unanticipated mortality, morbidity or cost
– Has the shifting of resources in an organization
compromised other client or patient populations?
Donabedian
Aim
Selecting A Measure
Operational Definitions
Data Collection Plan
The Quality
Measurement Roadmap
Data Collection
Data Analysis
ACTION
Modified from Lloyd, Robert: “Quality Health Care A Guide to Using Indicators”
Selecting a Measure:
-When selecting a measure, have clarity as to
whether the measure is one of structure, process or
outcome
-And select a balanced panel of indicators that
reflect the dimensions of performance being
evaluated and the change concept(s) being
employed
What Dimension of Performance
Do You Want to Measure?
•
•
•
•
•
•
•
•
•
Appropriateness
Availability
Continuity
Effectiveness
Efficiency
Respect and caring
Financial/Viability
Safety
Time lines
Joint Commission (1996)
What Dimension of Performance do
You Want to Measure?
•
•
•
•
•
•
Safety
Effectiveness
Patient-centeredness
Timeliness
Efficiency
Equity
IOM: Crossing the Quality Chasm (2001)
What is the “Change Concept”?
•
•
•
•
•
•
•
•
•
Eliminate waste
Improve work flow
Shorten a waiting list
Change the work environment
Improve the Provider/Client interface
Manage time
Focus on variation
Error proofing a process
Focusing on product or service
The Improvement Guide by Langley, Nolan, Nolan, Norman and Provost. Jossey-Bass
Relating a Change Concept to a
Specific Measure
Concept
Potential Indicators for this process
Patient scheduling
•The average number of days between the call for an
appointment and the actual appointment date
•The percentage of appointments made within 3 days of the call
for an appointment
•The number of appointments scheduled each day
Home care visits
•The number of home care visits
•The average time spent during a home care visit
•The percentage of time spent traveling during each home care
visit
•The number of visits per home care nurse
CQI Training
•The number of participants attending a class
•The percentage of cancellations
•The percentage of no-shows
•The information recall scores at 30 and 60 days
Establishing Operational Definitions That
Are Agreed Upon By All Stakeholders
Operational Definitions
• Is clear and unambiguous
• Specifies the measurement method, procedures and
equipment when appropriate
– Clinical data (chart reviews) vs. administrative data
– Client logs vs. a computer database
• Define specific criteria for the data to be collected
– Define all inclusions and exclusions
– For percentages or rates, or ratios, define the criteria
for inclusion in the numerator and denominator
• Always ask “How might somebody be confused by this
definition?”
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
Examples of Unclear Definitions
•
•
•
•
•
•
Timely completion of the screening process
A complete medication list
The readmission rate
Medication error
Cost impact
From the acute care hospital
– A patient fall
– Surgical start time
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
Data Analysis
• How will the measurements be expressed?
– Quantities, rates, ratios, proportions, percentages
• What type of statistics will be used?
– Descriptive statistics
• Measures of central tendency
– Mean, median, mode
• Measures of variation or spread
– Minimum, maximum, range, standard deviation
– Inferential statistics
• t-tests
• ANOVA
• Chi Square
Data Display
•
•
•
•
•
•
•
Table
Bar chart
Histogram
Line chart
Pie chart
Pareto diagram
Time-ordered data
• Run chart
• Control chart
Comparative Data
• Internal targets-trended data
• External comparisons-benchmarking
– Best practices
– National or regional population averages
External Benchmarking
Joint Commission
Calculation of the
Confidence Interval
Estimates
± t * σ/ √n
CMS
Where
t= 3 (the sigma number for
99% confidence interval)
σ =The hospital’s standard
error of the mean and
n = The number of patients
in the hospital’s
denominator
Data Reporting
• Data reporting plan
– Who will receive the results?
– How often will they receive the results?
– How will it be formatted?
• Dashboard
• Paper reports
• Spider diagram
– How will the data be disseminated?
• E mail
• Internet
• Intranet
Displaying Time-Ordered Statistics
and Understanding Variation
Tools for Displaying Time-ordered
Data
• Run charts
– Plot of data over time with the median of the data
set plotted as a center line
• Control charts
– Plot of data over time with the mean as the center
line and with upper and lower control limits
Run Charts
• Easily constructed by hand or in available
spreadsheet programs
• Provides a good idea of improvement in a
change initiative
• Less sensitive to significant changes (special
cause variation) than the control chart
Control Charts
 More sensitive to special cause variation than a run
chart
 Requires specialized computer software to create
 There are 9 types of control charts used in health
care, depending upon whether the data collected is
distributed normally, is continuous (numerical) or
discreet (attributes) and whether the events
measured are frequent or infrequent
 Have their own set of rules to identify special cause
variation
Understanding Variation
• All data, collected over time, varies
• Random variation (common cause)
– The changes occurring are intrinsic to the process being
measured
• Non-random variation (special cause)
– The changes are being imposed on the system by some external
factor
– May be unintended and un anticipated or may be by design
• Before process improvement can be implemented, the
process must be in control (free of special cause variation)
Common Cause (Random) Variation in a
Run Chart
Average WBC of Patient With Neutropenia
2.5
2
1.5
1
0.5
0
Value
Median
Special Cause Variation in a Run Chart
Special Cause Variation in a Control Chart
Daily record of Blood Pressure
Upper Control Limit 205 mmHg
Mean 173 mmHg
Lower Control Limit 142 mmHg
Special Cause Variation 138 mm Hg
Special Cause Variation in a Control Chart
Deciding Whether To Design A
New Process or Improve An
Existing Process
Initial Considerations
• Is the process under consideration local?
– Within a department
– On a clinical unit
• Is the project organization wide?
– A process change in a work system that impacts
the entire organization
– Requires commitment of people, funds, or new
technologies
Organization-Wide Initiatives
• Must be consistent with the organization’s
Mission, Vision, and Values
• Must be aligned with the organization’s
strategic plan
Strategic Goals
Q
U
A
L
I
T
Y
+
P
A
T
I
E
N
T
s
+
P
E
O
P
L
E
=
M
A
R
G
I
N
+
G
R
O
W
T
H
Measurement and the Strategic Plan
Obtain
Inputs
Analyze the inputs
Determine the organizational
strategies for each strategic goal
Determine the organizational
measures, performance Targets
and benchmarks
Determine the departmental tactics,
measures, and targets
Formulate the IT Capital Budget
Determine HR Requirements
Map the data source
Locate or design the system
Write the interfaces
Populate the dashboards
Staffing requirements
Grow or Purchase
Training requirements