Transcript An overview of public health and its practice

Performance Monitoring
Day 4 EM Analyst Training Course
02.10.12
Learning Objectives
• At the end of the course, participants will be
able to ...
– explain what is meant by performance monitoring
– identify the drivers, purpose and applications of
performance monitoring
– illustrate how performance monitoring is
undertaken in practice
Performance monitoring
Executives worth their salt will be asking ...
•
are we achieving our objectives?
•
are we meeting our targets?
•
what should our priorities be?
•
what’s our strategy for ....?
•
what changes do we need to make to local services?
•
where and how should we target our resources?
•
are our initiatives and service changes having the intended effect?
Drivers and purpose
• Demand for accountability of public services;
“Clinical governance”.
• Quantifying and qualifying the results of
surveillance.
• Encourage best practice; identify opportunities
for development and shared learning.
• Clarify an organisation’s objectives and priorities.
• Measuring progress against targets.
ASSESSING PERFORMANCE
5
MONITORING AND EVALUATING PLAY AN
IMPORTANT ROLE WITHIN THE NHS
Definition?
 Measuring and comparing observed data
with a benchmark or baseline
Comparisons can be




over time (e.g. one year to the next)
between areas (e.g. local authorities)
against a target (e.g. 98%)
against an expected value
CHOOSING AN INDICATOR
THE GOOD INDICATORS GUIDE
A poorly designed indicator with reliable
data, or a well designed indicator with
unreliable and/or untimely data, has very
little value (and is sometimes dangerous).
THE GOOD INDICATORS GUIDE
10 KEY QUESTIONS
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
What is being measured?
Why is it being measured?
How is the indicator defined?
Who does it measure?
When does it measure it?
Will it measure absolute numbers or proportions?
Where do the data actually come from?
How accurate and complete will the data be?
Are there any caveats/warnings/problems?
Are particular tests needed (e.g. standardisation, significance tests
or SPC) to test the meaning of the data and the variation they
show?
THE GOOD INDICATORS GUIDE
•
Selection of indicators and the data that feed into them will
often involve a trade off between what is convenient (and
possible) and what is ideal.
•
The right indicator at the right time and in the right context
can stimulate huge change where it might have no effect
in other contexts and at other times. Change hearts and
minds
•
Try not just to present the data – look for the story it is
telling.
DOES THIS TELL YOU ENOUGH?
Sources: Office for National Statistics and Teenage Pregnancy Unit
COMPARATORS GIVE CONTEXT
Sources: Office for National Statistics and Teenage Pregnancy Unit
Ways of evaluating performance
•
•
•
•
Spine chart
Tartan rug
Benchmarking
Statistical Process Control
– Control and run charts
– Funnel plots
12
Ways of evaluating performance
Number of indicators
Many/one
Spine chart
Tartan rug
Benchmarking
Statistical Process
Control (SPC) Control
chart / Run chart
SPC funnel plot
Number of organisations
Many/one
Over time?
Yes/no
Spine Chart
So you want to make a spine chart?
http://www.wmpho.org.uk/tools/
ERPHO Spine chart video
http://www.youtube.com/watch?v=480Mswgcg8M&feature=plcp
14
Tartan rug
15
Benchmarking
• Contextualise performance
• Encourages best practice
• Shared learning and areas for development
Benchmarking
• Which local authority would you say had an issue with
perinatal mortality? Can you think of any issues regarding
presenting data in a league table such as this?
East of England average
Confidence
interval
Perinatal mortality rate by local authority, East of England 2001–2005, INpho 26
17
Statistical Process Control
‘common cause variation’
the normal, everyday, inevitable (and usually
unimportant), variation which is intrinsic and
natural to any system
‘special cause variation’
the more important variation which is indicative of
something special happening and which calls for
a fuller understanding and often action
For more info see APHO technical briefing
http://www.apho.org.uk/resource/view.aspx?RID=39306
18
Types of control chart
• Run chart – displays any data over time against its
median
• Control chart – looks similar to a run chart but plots
data against its mean and the appropriate control limits
• Good for monitoring trends over time
• Good for monitoring data against a specific target
– A&E 4 hour waits
– Childhood obesity
Control charts explained to music
– http://www.youtube.com/watch?v=1XM3DpyBHss
Example of a control chart
•Control charts are plots of
the data with lines
indicating the target value
(mean, median) and
control limits
superimposed.
Types of control chart con..
Funnel plots
– A type of chart where the indicator of interest is
plotted against the denominator or sample size.
– This gives it the characteristic funnel shape
ERPHO funnel plot video
http://www.youtube.com/watch?v=480Mswgcg8M&feature=plcp
So you want to create a funnel plot?
APHO provides tools for creating funnel plots for
different kinds of data
http://www.apho.org.uk/default.aspx?RID=39403
Example of funnel plot
Using control charts and SPC methods
• Control charts can help us to present and interpret our
information more intelligently.
• They can be used
– To detect unusual or outlying patterns, e.g. poor
performance, outbreaks or unusual patterns of disease
– In health profiling and assessing levels of performance
– To decide whether or not targets are being met
– In assessing health inequalities (i.e. funnel plots)
Control chart A&E 4 hour waits:
Change in target
Green line = target
Red lines = CIs
24
Funnel plot
Perinatal Mortality
25
Breast cancer mortality
26
APHO technical briefing 2: Statistical process control methods in public health intelligence
Lung cancer mortality
27
APHO technical briefing 2: Statistical process control methods in public health intelligence
Summary
• Good monitoring and benchmarking requires use of
comparators
• Good data visualisation is helpful
• Statistical process control allows you to distinguish
between common cause variation and special cause
variation, which might indicate a problem or an
exceptional performance within the system.
28
FURTHER INFORMATION
The Good Indicator Guide: how to use and choose performance indicators
http://www.apho.org.uk/resource/item.aspx?RID=44584
APHO Technical Briefings:
http://www.apho.org.uk/resource/view.aspx?RID=39306
No. 2: Statistical process control methods in public health intelligence
No. 3: Commonly used public health statistics & their confidence intervals
No. 4: Target setting in a multi-agency environment
Health Profiles: http://www.healthprofiles.info
WMPHO spine chart generator:
http://www.wmpho.org.uk/resources/WMPHO_Spine_chart.xlsm
ERPHO tool to produce run charts (you may need to log onto the ERPHO website)
http://www.erpho.org.uk/viewResource.aspx?id=14762
Create a p chart in excel
http://www.ehow.com/how_8039012_create-chart-using-samples.html
Learning Objectives
• At the end of the course, participants will be
able to ...
– explain what is meant by performance monitoring
– identify the drivers, purpose and applications of
performance monitoring
– illustrate how performance monitoring is
undertaken in practice