Transcript Quality Indicators

Quality Indicators:
Past and Present
Michael A Noble MD FRCPC
Professor
Medical Microbiology and Infection Control, Vancouver Coastal Health
and
Chair, Clinical Microbiology Proficiency Testing program,
Chair, Program Office for Laboratory Quality Management
Department of Pathology and Laboratory Medicine
University of British Columbia
Quality Indicators:
Past and Present
•
•
•
•
•
•
History
Quality Indicators and ISO
Characteristics of Indicators – strong and weak
Quality Indicator Inventories – USA and BC
Examples of Quality Indicators
Summary
Amazon.com
$30.00
A really
good,
inexpensive
reference
book
Two excellent (essential) references from CSA
A Short History of Metrics in
Quality Management
Innovator
Walter A Shewart
Date
Cycle
1920’s
Plan-Do-SEE
J Edwards Deming 1940’s
Bob Galvin
1980’s
Plan-Do-CHECK-Act
Define-MEASUREAnalyze-ImproveControl
The Quality Cycle
Plan
Act
Do
CHECK
Each step is
essential to keep
the quality cycle
cycling
Quality Indicators
A workable definition
• Established measures used to
determine how well an organization
meets needs and operational and
performance expectations.
– Objective
– Measurable
– Repeatable
Metrics and ISO 9001:2000
Factual Approach to Decision Making
5.4.1
• Top management should ensure that quality objectives,
including those needed to meet requirements for product,
are established at relevant functions and levels within the
organization. The quality objectives shall be measurable
and consistent with the quality policy.
Metrics and ISO 9001:2000 (2)
8.4
The organization shall determine, and collect and analyze appropriate
data to demonstrate suitability and effectiveness of the quality
management system and evaluate where continual improvement of the
effectiveness of the quality management system can be made. This
shall include data generated as a result of monitoring and measurement
and from other relevant sources.
The analysis of data shall provide information relating to:
– Customer satisfaction
– Conformity to product requirements
– Characteristics and trends of processes and products including
opportunities for preventive actions, and
– suppliers
Metrics and ISO 15189:2003
• 4.12.4
Laboratory management shall implement quality
indicators for systematically monitoring and
evaluating the laboratory’s contribution to patient
care. When this program identifies opportunities
for improvement, laboratory management shall
address them regardless of where they occur.
Laboratory management shall ensure that the
medical laboratory participates in quality
improvement activities that deal with relevant
areas and outcomes of patient care.
So…
Quality Indicators are measured information that
•
•
•
•
Indicates the performance of a process.
determines quality of services.
highlights potential quality concerns,
identifies areas that need further study and
investigation, and
• track changes over time.
Measuring Performance
Mark Graham Brown
• Fewer is better.
• Link measures to the factors needed for success.
• Measures should be based around customer and stakeholder
needs.
• Measures should start at the top and flow down to all levels
of employees.
• Measures should change as the environment and strategy
changes
• Measures should have targets or goals established that are
based on research rather than arbitrary values.
Keeping Score
Using the Right Metrics to Drive World Class
Performance
1996
Many organizations spend thousands of
hours collecting and interpreting data.
However many of these hours are nothing
more than wasted time because they
analyze the wrong measurements, leading
to inaccurate decision making.
– Mark Graham Brown.
Indicators?
You want Indicators?
We’ve got LOTS of Indicators!
AHRQ
RAND
IQLM
JCAHO
OECD
WHO
Leapfrog
American Nurses Association
American Psychiatric Association
ASQ
National Quality Forum
ISQua
Characteristics of Good Metrics
Measurable
objective
Achievable
contained
Timed
short and long term
Engaging
all levels
Good
Metrics
specific
Balanced
full cycle
Interpretable
Actionable
action oriented
Indicators of Good Indicators
Measurable
Can you count it, time it, record it?
Achievable
Can you actually capture it?
Interpretable When you’ve got it, what does it mean?
Actionable
Can you do something about it?
Timed
Does your set cover both the short and long term?
Engaging
Does your set involve all laboratory personnel?
Balanced
Does your set cover the full cycle of events?
Assessing Quality Indicators
•
•
•
•
Importance
Scientific Acceptability
Feasibility
Usefulness
Potential for Improvement
Reliability and Validity
Implementation and cost
Comprehensive
Having Quality Quality Indicators
Total Testing Cycle for
Medical Laboratories
Menu
Report Interpretation
Collection
Transport
Pre-Analytic
Patient ID
Post-Analytic
Ordering Rules
Acceptance Criteria
Report Transport
Report Creation
Data Capture
Analytic
Analysis
Quality Control
Baldrige Award Criteria
• Balanced Metrics
–
–
–
–
–
–
–
Customer satisfaction
Employee satisfaction
Financial performance
Operational performance
Product and Service quality
Supplier performance
Safety and environment and public responsibility
Most organizations focus 80% of metrics on
finance and operations.
IQLM Indicators
•
•
•
Diabetes monitoring (system)
Hyperlipidemia screening (system)
Test Order Accuracy and Appropriateness
•
•
•
Patient Identification (pre-analytic)
Adequacy and Accuracy of Specimen Information (pre-analytic)
Blood Culture Contamination (pre-analytic / system)
•
•
Accuracy of point-of-care testing (analytic)
Cervical cytology/biopsy correlation (analytic)
•
•
•
•
Critical Values Reporting
Turnaround time (postanalytic)
Clinician satisfaction (system/postanalytic)
Clinician followup (system/postanalytic)
CMPT Metrics Scorecard
• Balanced Metrics
–
–
–
–
–
–
–
Customer satisfaction
Employee satisfaction
Financial performance
Operational performance
Product and Service quality
Supplier performance
Safety /environment /
public responsibility
Percent
25
5
10
20
30
5
5
Characteristics of Weak
Metrics
•
•
•
•
Focus only on measures easy to count
Focus only on measures easy to achieve.
Metrics with arbitrary targets.
Measures that don’t change with experience
Computer Nonsense Metrics
[urine culture] * [glucose] * [INR]
X100
[NUPA hr] * [Telephone minutes]
Just because a computer
can calculate a value,
doesn’t mean that it
should.
Computerese Quality Indicators
•
•
•
•
•
•
Unit Producing Activity per Paid Hour
Unit Producing Activity per Worked Hour
Unit Producing Activity per Total FTE
Non-Unit Producing Activity per Paid Hour
Non-Unit Producing Activity per Worked Hour
Non-Unit Producing Activity per Total FTE
• Crude Turn-Around-Time
A Cautionary Note
• Measures that drive the wrong performance.
Measuring professionals is tough because
intellectual work is difficult to measure
objectively. Looking for factors that can be
counted may not be what is really
important. Meaningful outputs such as
ideas, information, and problems avoided
may be difficult but more relevant.
Mark Graham Brown
Caution about
patient outcome indicators
Theoretically, outcomes best assess quality,
but they are the most difficult to measure
– too many variables and confusers
• Age, underlying conditions, therapy, circumstance
– require high volumes of detailed data
– Need long collection periods.
David Hsia
Medicare Quality Improvement Bad Apples or
Bad Systems?
JAMA. 2003;289:354-356.
Are you an Indicator Glutton?
Monitoring more
than 10-12
indicators is rarely
successful
Mark Graham Brown
1996
Quality Inventory:
US Medical Laboratories
2004
• In 2004 the Institute for Quality in Laboratory
Medicine (IQLM) and the Clinical Laboratory
Managers Association (CLMA) undertook an online quality inventory of laboratories with CLMA
members.
• Approximately 400 laboratories responded.
• The study was voluntary, self-reported, with a
validated questionnaire.
• Information provided was not verified by a second
method
In British Columbia…
The Program Office for Laboratory Quality
Management and the Provincial Laboratory
Coordinating Office have organized to
perform a similar, but improved inventory
in 2005.
QC
EQ
A
Co
PT
nta
ID
Sa
i
n
e
mp
l e I r OK
nte
Sa grity
m
La pl e I
Wr b Inj D
uri
Co i tten
e
ll ec
Or s
Sp tion ders
eci
me Timi
n
nS
t or g
age
10 Most Common Procedures Monitored
BC Quality Inventory 2005
96
94
92
90
88
86
84
82
80
78
76
Pre-Analytic
System
Analytic
Post Analytic Procedures Monitored
BC Quality Inventory 2005
78
76
74
72
70
68
66
64
62
r
po
e
R
acy
r
cu
c
tA
p
Re
Cr
t
or
a
itic
es
u
l
a
lV
rn
u
T
A
nd
u
ro
e
m
i
T
Satisfaction Monitoring
BC Quality Inventory 2005
Emp
loyee
Phys
ician
omy)
lebot
Patie
nt (P
Patie
nt
60
50
40
30
20
10
0
Other Achievable Indicators
• Blood culture volumes:
Blood culture false negative results occur
when bottles contain insufficient (<3 mL)
or excessive (>15 mL) blood.
Insufficient or excessive blood collection is
a collection non-conformity.
Under and Overfill Blood Cultures
2001-2004
5.0%
4.5%
4.0%
3.5%
3.0%
Percent Over
2.5%
Percent Under
2.0%
1.5%
1.0%
0.5%
0.0%
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
Underfill Blood Collections
(As a percent of collections per site)
16.0
14.0
12.0
RF2
10.0
RF1
8.0
ER
ICU
6.0
Phlebotomists
4.0
2.0
0.0
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
Baldrige Award Criteria
• Balanced Metrics
–
–
–
–
–
–
–
Customer satisfaction
Employee satisfaction
Financial performance
Operational performance
Product and Service quality
Supplier performance
Safety and environment and public responsibility
Most organizations focus 80% of metrics on
finance and operations.
Eight Steps to Developing
Successful Indicators
1.
2.
3.
4.
5.
6.
7.
8.
Objective
Methodology
Limits
Interpretation
Limitations
Presentation
Action plan
Exit plan
Developing Indicators
Objective
Methodology
What are you trying to measure
Limits
Presentation
Interpretation
Acceptable, Concern, Unacceptable Critical
Limitations
Action Plan
Unintended variables
Exit Plan
When can I stop measuring?
1.
2.
3.
How to capture the data
Who (or what) to capture the data
How often to capture the data
Graphic or Text
What does it mean?
Does it reflect on YOUR quality?
What will I do if it indicates acceptable performance?
What will I do if it does not?
Presenting
Quality Indicator Information
80
70
60
50
High
40
Low
30
Average
20
10
0
1st Qtr
2nd Qtr 3rd Qtr
4th Qtr
The BIG SECRET for
Quality Indicator Team
Engage the folks
who do the work,
because they
know what
they do!
Microbiology Indicators
Collected and Monitored by Vancouver
General Hospital Division of Medical
Microbiology and Infection Control
Many thanks to:
Diane Roscoe
Anita Kwong
Medical Microbiology team
Contmination Rate: Blood Culture Sets
4.0%
Percent
3.0%
2.0%
1.0%
0.0%
1
2
3
4
5
6
7
8
9
10 11 12 13 14
Time Period
2002-2003
2003-2004
2004-2005
Contmination Rate: Blood Culture Sets
4.0%
Percent
3.0%
Objective:
to ensure that blood
culture results reflect
sepsis.
2.0%
Methodology: Count single bottle
1.0%
0.0%
1
2
3
4
5
6
7
8
9 10 11 12 13 14
positives of common skin
flora/ total sets
Time Period
2002-2003
2003-2004
2004-2005
Limits:
Below 2%
Interpretation: Meeting accepted limits all the time
Limitations Definition may include some true infections and
may miss others
Presentation: Linear time graph
Action plan: Identify and educate blood collector group.
Exit plan:
Reactivate with cause
Underfill Blood Collections
(As a percent of collections per site)
16.0
14.0
12.0
RF2
10.0
RF1
8.0
ER
ICU
6.0
Phlebotomists
4.0
2.0
0.0
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
Underfill Blood Collections
(As a percent of collections per site)
Objective:
16.0
14.0
to ensure that blood
culture are properly filled.
12.0
RF2
10.0
RF1
8.0
ER
ICU
6.0
Phlebotomists
Methodology: Count underfilled bottles /
total bottles collected
4.0
2.0
0.0
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
Limits:
Below 2% (?)
Interpretation: Wards with inexperienced collectors have problems
Limitations Some frail and elder people have very weak veins and may be
impossible to collect
Presentation: Linear time graph
Action plan: Identify and educate blood collector group.
Exit:
Continue on selective basis
Certification Performance
Year
Event
Measures
MAJOR
NC
Minor
NC
2002
Pre-Certification (EI)
100
1
2
2002
Certification (E)
100
0
2
2003
Pre-Certification (EI)
100
0
0
2003
Certification (E)
100
0
0
2004
Certification (E)
100
0
0
2005
Pre-certification (EI)
100
0
1
2005
Re-Certification (E)
100
0
0
Year
Event
Measures
MAJOR
NC
Minor
NC
2002
Pre-Certification (EI)
100
1
2
2002
Certification (E)
100
0
2
2003
Pre-Certification (EI)
100
0
0
2003
Certification (E)
100
0
0
2004
Certification (E)
100
0
0
2005
Pre-certification (EI)
100
0
1
2005
Re-Certification (E)
100
0
0
Objective:
to monitor CMPT quality
preparedness
Methodology: Monitoring External
assessment values
Limits:
No Major above 1Below 2%; No Minor above 3
Interpretation: Meeting accepted limits all the time
Limitations May indicate things are better than they are if
inspector is not diligent
Presentation: Linear time table
Action plan: Maintain program, respond through OFI and Corrective
Actions
Exit plan:
Compile with each inspection
Composite Indicators
• Reflecting a single subject with a number of
sub-components
When the finished value is greater than just the
sum of the parts
Creating Composite
Quality Indicators
25
1
2
üIdentify individual components
üWeight the components
üDefine Limits
üMeasure and Combine
üMonitor for trend
3
4
5
CMPT Client Satisfaction
Composite Score
Factor
Weighting
Survey Score
+10
Open Comments –Positive
+5
Open Comments – Negative
-10
New Contracts
+10
Contract Renewals
+25
Contract Cancellations
-100
Consults
+5
Complaints
-10
Survey
Positive Opinions
Negative Opinions
New Contracts
Contract Renewals
Contract Cancellations
Consults
Complaints
VALUE
30
5
1
1
0
1
0
105
80
30
16
0
0
0
0
1
76
Limits
90
Negative Opinions
New Contracts
Contract
Renewals
Contract
Cancellations
Consults
Complaints
90
24
6
0
0
0
5
2
2003-2004
85
22
10
4
0
0
5
0
2004-2005
85
22
6
6
0
0
3
0
2005-2006
85
20
2
2
1
0
4
1
Year
Survey
Positive Opinions
2002-2003
CMPT Client Satisfaction
Composite Score
110
103.5
100
100
96.5
96.5
2002-2003
2003-2004
90
80
70
2004-2005
2005-2006
CMPT Composite Satisfaction Score
•
•
•
•
•
•
•
•
Objectives: To indicate customer satisfaction
Methodology: Examination of 5 independent variables
Presentation: Composite score
Interpretation: Score associated with satisfaction
Limits:
76-105 calculated weighted score
Limitations: Arbitrary
Action plan: Root Cause Analysis of deficiencies
Exit:
Annual for 5 years and evaluate
In Summary
• Quality Quality Indicators are a required
component of a quality management system.
• Quality Quality Indicators can be characterized
and distinguished from Weak and Terrible Quality
Indicators.
– Watch out for the weak ones
– Avoid the terrible ones
• Quality Quality Indicators provide the information
and opportunity essential for POSITIVE action.
Setting Relevant Ranges
•
•
•
•
•
•
Set Objectively
Validate by Study
Clinical Relevancy
Customer Expectation
Matched Benchmarks
Regulation
60 minutes
Relevant or Easy?
Quality Indicators and Timing
Use an indicator
only as long as
it provides
you with
useful
information.
Don’t get tied to
your indicators
Caution about
patient outcome indicators
Theoretically, outcomes best assess quality,
but they are the most difficult to measure
– too many variables and confusers
• Age, underlying conditions, therapy, circumstance
– require high volumes of detailed data
– Need long collection periods.
David Hsia
Medicare Quality
Improvement Bad Apples
or Bad Systems?
JAMA. 2003;289:354356.
The BIG SECRET for
Quality Indicator Team
Engage the folks
who do the work,
because they
know what
they do!
Timed