Transcript Alastair Fischer: Centre for Public Health, NICE Modelling in UK Health Policy Decision Making. Newton Institute, Cambridge, June 2014 Running Order • • • • • • • • Objective function Before NICE After.

Alastair Fischer: Centre for Public Health, NICE
Modelling in UK Health Policy Decision Making.
Newton Institute, Cambridge, June 2014
Running Order
•
•
•
•
•
•
•
•
Objective function
Before NICE
After NICE
Social value judgements
The 4 pillars of the evaluation process
About Public Health
How Public Health is not Technology Appraisal
A decision theory approach versus a hypothesis-testing
approach
• Use of ‘what-if?’ or threshold analysis
• New issues: cost effectiveness and infectious disease
modelling
Objective Function
• If the objective is to maximise the health gain from
interventions, there needs to be a generic expression for
health
• The QALY: quality-adjusted life year. 1 QALY= 1 year at full health
– Pragmatic
– Relatively simple
– It is measured using a simple questionnaire. The UK uses the
simplest – the EQ-5D
•
•
•
5 dimensions (reasonably orthogonal), 3 states for each dimension
‘Utility’ (measuring quality of life) from EQ-5D is from 1(full health) to 0
(dead) but it is possible to go below 0
A QALY is the time integral of utility.
• Gain QALYs by either living longer or at a higher QoL.
Problems with QALYs
• You name them, QALYs have them
• But as a quick and dirty solution to being a generic
measure, they work well enough
• And in a lot of cases, the same decision would be made
whether QALYs or Life Years Gained (i.e. ignoring quality of
life) were measured.
• From a purist (academic) viewpoint, QALYs fail, but from a
pragmatic decision-making viewpoint, they are better for the
decision-maker than not having them (which is the realistic
alternative).
Back to the Objective function
• Maximise the QALY gain from a given budget
–
–
–
–
Classic optimisation problem
All the classic theorems apply (MUX/pX = MUY/pY)
This allows us to be completely efficient
Er…what about equity (= fairness)?
•
•
•
•
•
•
•
A: Well, we are prepared to trade off some efficiency to be fairer.
Q: So we really should be maximising the gain in a social welfare
function that consists of efficiency and equity arguments?
A: Yes
Q: But equity is subjective, isn’t it? So how do we formulate a SWF?
A: We don’t know if this will do more harm than good, because
fairness depends on context. So we use the GOBSAT principle.
Q: I’ve never heard of that. What is it?
A: (sheepish) …Good Old Boys Sat Around A Table.
– And we may have multiple objectives, not just equity and
efficiency. Again, done by deliberation.
The threshold
• So for efficiency, we buy the cheapest QALYs first.
• And keep on buying the next cheapest, and the next, till we
run out of money. That maximises the number of QALYs we
can gain.
• Where we run out of money is called the threshold.
– But NICE only controls a relatively small part of the budget.
– Other parts of the NHS use time-honoured deliberation, just like
most decisions made in other parts of the Civil Service, in every
country of the world.
– So the threshold was determined essentially by trial and error in
the first instance.
– Originally £30,000 per QALY, and then more recently in the range
£20,000 to £30,000 per QALY.
– More recently, a more objective measure has been found that is
good in theory but has many problems. It estimates the threshold
to be about £15,000 per QALY.
Value based pricing (VBP)
• A £15,000 per QALY threshold would mean few new drugs
would be approved at their current prices. The drug
companies would be affronted. The threshold is based on
what the NHS could afford to pay.
• So another mechanism to pay higher prices for drugs than the
threshold would allow has been sought. The DH has looked at
VBP, which would include other benefits not connected to the
NHS, such as whether a recipient of a drug could go back to
work, and if so, go back sooner. The value of the additional
production would be added into the ‘threshold’ for that drug.
• NICE has questioned this approach on the grounds that it
would discriminate against the old, the young, and the
unemployed and low paid workers. So equity has been
invoked.
Before NICE
• There was no single place where doctors could get advice on
the best ways of treating patients, including getting unbiased
advice on drugs and medical devices such as pacemakers.
• Some PCTs would allow a particular drug to be paid for by the
NHS but others would not. Postcode prescribing was rife.
Drug and device companies went from hospital to hospital, GP
practice to GP practice, hawking their wares. Information was
inefficiently broadcast.
• Clinical guidelines existed, but were relatively haphazardly
produced. They had no set timetable for completion,
somewhat scatty searches for the best evidence, often
unrepresentative committees to determine the decisions, a
lack of transparency about how a decision was made and
patchy consultation.
After NICE
• All that changed with the advent of NICE.
– Draconian timelines:
•
we know when a decision will be made, sometimes years in advance.
– Transparent process.
•
–
–
–
–
–
Many meetings are held in public.
Strict rules of engagement (methods manuals)
Representative committees
Independent advice
Extensive consultation
Citizen’s Council to advise on Social Value Judgements
The four pillars
•
•
•
•
Efficiency
Social Value Judgements
Process (transparency, timeliness, etc)
Policy Perspective (What will change a decision?)
• SVJs sometimes override efficiency
– The right to vote is inalienable. It cannot be traded.
• Additionally, other SVJs can be traded
– Equity offsets efficiency to some degree.
• Modellers need to be aware of all these things as context to
their modelling.
About Public Health
• Very many interventions in public health are cost effective
– Models can often be very simple and do not require fancy
extensions to be able to determine cost effectiveness.
– PH committees don’t like it when the NICE PH health economist
says that an intervention will be in the right direction and will be
cost saving. Therefore by definition cost effective!!
•
•
•
The committees seem to feel cheated, and hold out for numbers. Until
they get numbers, they appear to fret and to worry about cost
effectiveness.
But put a number in front of them – however great the uncertainty – if it
comes in far enough below £20,000 (such as £15,000) they take cost
effectiveness for granted and move onto more interesting discussions.
A caricature? Decidedly so, but as with all good caricatures, the
exaggeration is based on enough grains of truth for people to
recognise what is being lampooned.
• We use the example of a small reduction in salt
consumption to illustrate the point more fully
Small reductions in salt
• We already know the direction of change from our prior
knowledge. We know the effect of large reductions in salt,
and we also realise that the relationship will not change
sign when the ‘dose’ is reduced.
• We also know that salt is very cheap, and that reductions in
it will cause savings of future treatment costs.
• End of story. Cost effective. Better than that – cost saving
and positive health benefit.
• But RCTs have been done on this topic.
– The effect size on blood pressure is very small at an individual
level
– Blood pressure is devilishly hard to measure accurately
– Keeping people to a lower-salt diet is well-nigh impossible
– So the trials were all underpowered, and showed non-significant
results in the ‘right’ direction
More about salt
• A meta-analysis of the trials (n = 6,489) showed:
– Cardiovascular morbidity in people with normal blood
pressure (longest follow-up RR 0.71, 95% CI: 0.42 to 1.20,
200 events) or raised blood pressure at baseline (end of
trial RR 0.84, 95% CI: 0.57 to 1.23, 93 events)
• This prompted a Cochrane press release, 5 July 2011
– Moderate reductions in the amount of salt people eat
doesn’t reduce their likelihood of dying or experiencing
cardiovascular disease. This is the main conclusion from a
systematic review published in the latest edition of the
Cochrane Library.
• The study has since been withdrawn
The currently-used paradigm
• We establish an effect by conducting an RCT. The
hypothesis-testing approach uses a frequentist
paradigm and rules out chance by means of a ttest or similar using p-values.
• If significance is achieved, a cost effectiveness
analysis is conducted as a second stage. This
uses a decision-theoretic paradigm. It establishes
cost effectiveness by looking at the size of the
estimated mean ICER, and does not consider the
ICER variance.
Decision theory versus frequentist hypothesis-testing
Decision theory
Frequentist
•
•
•
•
•
•
Subjective probability
Prior beliefs
To maximise, variance of
effect estimate is ignored
–
•
•
•
This is as if the decisionmaker is risk-neutral
Makes sense if a large
number of independent
projects are considered.
Used routinely in the
business world for
maximising profits
Objective probability
No prior beliefs
Does not maximise, and is
rather conservative
•
•
Decision-maker is riskaverse
Does not consider other
projects (maybe shouldn’t if
health is concerned?)
Used routinely in medical
research for effectiveness.
The effect of bias
• Decision theory and frequentist hypothesis-testing
do not deal well with bias.
• The RCT avoids bias internally. A frequentist
approach further avoids bias by ignoring prior
beliefs. So where bias is important and can be
avoided, use a frequentist approach. As in TA
• The underpowered RCT is very prone to
publication bias. RCTs will often be underpowered
in public health when the individual effect size is
very small. Prior beliefs, however, will often not be
subject to much bias, especially if they are firmly
held. So use decision theory for effectiveness in PH
Technology appraisal
RCT: Estimated distribution of the mean effect
Posterior distribution combining prior and RCT
Vague prior belief about mean effectiveness
• Posterior distribution is influenced almost entirely by the RCT
• If the prior distribution is biased, it might detract from accuracy and not enhance it.
• So the decision-maker may wish to remain with the red curve (RCT only) as the
most accurate.
Public health population intervention
Non-vague prior belief about mean effectiveness
Posterior distribution combining prior and RCT
RCT: Estimated distribution of the mean effect
• Posterior distribution is influenced almost entirely by the prior belief curve.
• The RCT is mostly a distraction, and it is likely to be biased upwards because of
publication bias
• So the decision-maker may wish to ignore the red curve (RCT) because it is not
helpful for the decision-maker. Because we know only the direction of change
with the yellow curve, we must use “what if” analysis to inform cost effectiveness.
• To get an effect size (yellow curve doesn’t give it) we use estimated mean RCT
effect as an upper bound, and do ‘what-if?’ analysis
More on a full decision-theory approach
• We have published one paper on this topic (JPH Dec 2013)
and have 2 more under submission, and a fourth paper is
being written.
• NICE processes already reflect this approach, but are not
given the gravitas that accords to a fully accepted paradigm
like decision theory. (“We must not allow special pleading
for Public Health. If the interventions do not satisfy 95%
confidence intervals, we cannot accept them.” This comes
from the strictest adherents to the frequentist school, but
those are the voices that are the most respected.)
• In round figures, we believe that the new approach could
lead to the same health gains at a cost about £10 billion
lower than by the next-best means.
Public Goods
• The most cost effective interventions in public health are
usually some form of public good.
– (A “public good” is like a Freeview TV programme – once it is
provided to one person, everyone else in the country with a TV
set can access it without cost, and cannot be excluded from it.
Programmes that are pay-per-view are ‘excludable’, so are not
public goods.)
• Legislation
– Smoking ban (big health gains, low cost of initiating, subsequent
cost saving)
• Taxation
– Almost no work done to implement, improves by reducing
drinking/smoking
• Regulation
– Speed cameras, traffic lights on foodstuffs
Public goods (2)
• Can’t often do RCTs in public goods, so in the past they
have been difficult to model for cost effectiveness.
• But recommendations about public goods are telling central
government what to do, and central government feels it has
primacy when it comes to decision-making.
• NICE has to tread carefully in this area and therefore its
recommendations must be oblique.
• Additionally, health gain is not the only consideration. A
recommendation for better health could be to tax beer until
a pint in a pub was £20. But this has a political dimension,
too, and no politician wishing for re-election could approve
such a recommendation.
• By making recs that are not the most cost effective, does
NICE ‘crowd out’ the most cost effective interventions?
Infectious disease modelling
• Time horizon
– Might be longer than a single lifetime
– But models only go for perhaps 20 years
•
•
•
Why? Is it because errors accumulate?
Yet health gains for an individual are usually at the end of life, which
might be more than 20 years away.
So the transmission model stops at 20 years, and we then follow
through the lives of those in the model till death? Is that the way to do
it?
• Models are tricky to calibrate
– A modelling assignment cannot be done in 3 to 6 months unless
modellers have already been working in the area for a couple of
years
– Much intellectual capital invested.
– NICE are required by law to make an executable model available to
consultees if requested. This is at odds with the intellectual property
rights of the model’s developers.
– Maybe there are better ways of ensuring model transparency
Infectious disease modelling (2)
• Model transparency through calibration
– Models are deemed ‘fit’ if they fit the past, and other tests of
robustness.
• Optimising over time in an infectious disease model would
appear to be the finding of an optimal time path out of the
infinity of possible paths.
– This is an optimal control problem. Has it been modelled as
such? Is it optimal to spend a huge amount today in order to
keep expenditure low in future, or to spend less now and rather
more later?
– This is a very tricky problem for governments whose time
horizons might be short. NICE might not be the best vehicle for
advising government on this.
Public health and local government
• Public health is now the responsibility of local
government.
• They have many other objectives, not just health.
• In particular, they spend a lot of money on caring for
the elderly, which means additional expenditure and
incentives for preventing dependency in old age.
• The QALY has far less meaning, and probably the only
common currency for evaluation studies is money.
• They have far tighter budgets than central government,
and thus their time horizons are shorter/their discount
rates are higher.
• The electoral cycle is only 3 years, again raising the
discount rate
Conclusions
• Many rethinks need to be carried out
– A change to a decision-theoretic from a frequentist
paradigm
– Learning more about transmission models and how
they can best be used for determining cost
effectiveness
– A more pragmatic approach to decision-making in line
with local government priorities and capabilities
– An educative role carried out by modellers and health
economists for both local councillors and LG officials
– There’s a tension between the immediacy of treating
the sick and ensuring in the longer term that the well
do not become sick. That is an implied SVJ that will
probably always be with us.
Infectious disease modelling (3)
• In my past life as an experimental economist, I played with
the idea of a vaccination programme that allowed two
optimal societal vaccination schedules.
– At a lower level of vaccination, there was a local optimum that
kept a disease under control
– At a higher level of vaccination, there was a global optimum (that
by definition gave higher benefits to society than the local
optimum) which eradicated the disease.
– In my replications of the model, the global optimum was not
attained, because of free riders.