Transcript Is Glennan’s Theory Complex Enough?
What’s EPB? What’s so good about RCTs?
(And what are we reading?)
Philosophy of Social Science Phil 152 Winter 2011 Week 8
A theory of evidence for use
• • Foundation for a guide by C&H for the use of evidence in evaluating policy effectiveness.
The (C&H) guide should be – – Simultaneously well-grounded and practicable .
– More comprehensive than those currently available – For policy-makers
not expert
in natural and social science with limited amounts of time and resources. 2
The question of effectiveness
Will the proposed policy produce the targeted outcomes were it to be implemented in the targeted setting in the way it would in fact be implemented there?
So, what’s evidence – good evidence – for answers?
3
The rise of
evidence-based policy • • • • • In the UK, USA and increasingly in Europe we see a huge drive to use
evidence
to inform
policy and practice
mandated by executive and legislative branches at international, national and local levels pushed by national and international organisations like the
Campbell
and
Cochrane Collaborations
with accompanying institutions and regulations to ensure evidence is appropriately considered.
4
Some guides for use of evidence to judge policy effectiveness
• • • • • • • • IARC: International Agency for Research on Cancer SIGN: Scottish Intercollegiate Guidelines Network What Works Clearinghouse USEPA: US Environmental Protection Agency CEPA: Canadian Environmental Protection Act Cochrane Collaboration Oxford Centre for Evidence-Based Medicine Daubert decision!! (US Supreme Court) 5
For an evidence-based policy…
We want – Evidence of
high quality
– That
speaks for
the policy.
6
High Quality Evidence
We do not want to build an argument for a policy on shaky premises.
7
High Relevance Evidence
No matter how sturdy this foundation: It won’t support these houses: 8
High Relevance v High Quality Evidence The Weight of Argument When do symmetry arguments provide the best evidence?
When are detailed calculations found more convincing ?
When does modeling appear conclusive ?
For how much does societal acceptance account ?
not
QUALITY
: The
likely truth
of evidence .
but
RELEVANCE
: Evidence that
supports the conclusion.
9
High Quality Evidence
We want to admit as evidence only claims that are true, or highly probable: P(e) is high .
Ranking schemes rank evidence according to the
method
by which it is produced. Top-ranked evidence is produced by methods that make it likely that the result is true: P(e) is high .
10
Evidence-ranking Schemes:
SIGNS (Scottish Intercollegiate Guidelines Network) 11
Levels of evidence
1++ High quality meta analyses 1 , systematic reviews 2 of RCTs 3 , or RCTs with a very low risk of bias 1+ 1 - Well conducted meta analyses, systematic reviews of RCTs, or RCTs with a low risk of bias Meta analyses, systematic reviews of RCTs, or RCTs with a high risk of bias 2++ High quality systematic reviews of case-control or cohort studies High quality case-control or cohort studies with a very low risk of confounding, bias, or chance and a high probability that the relationship is causal 2+ 2 - Well conducted case control or cohort studies with a low risk of confounding, bias, or chance and a moderate probability that the relationship is causal Case control or cohort studies with a high risk of confounding, bias, or chance and a significant risk that the relationship is not causal 3 4 Non-analytic studies, e.g. case reports, case series Expert opinion 12 1 See Glossary 2 See Glossary 3 See Glossary
The RCT
• • RCTs establish causal conclusions: In Tennessee reducing class size caused better reading scores.
In two Zambian hospitals introducing tri-co… increased survival rates in HIV-positive children.
• • If properly done, they guarantee That there was the difference indicated.
That it was
caused
in the way indicated (and not, e.g., by accident or by some other factor introduced at the same time).
The method itself makes it very probable that the resulting conclusion is true .
13
• • •
What’s an RCT for ‘T causes O’?
A Mill’s method-of-difference study: – 2 groups where all causal factors for O are distributed the same except for T and it’s downstream effects.
– T is universally present in the treatment group, universally absent in the control.
If Prob T (O) > Prob C (O), T must be responsible for the difference.
SO: T was a contributing cause towards O in at least some members of study population.
14
What’s special about RCTs?
• • They assumptions for the study are met and Prob T
clinch
causal conclusions: (O) > Prob C (O), it follows If the deductively that T causes O in some study members.
They are
self-validating
: – Blinding (quadruple maybe) – Random assignment – Placebo control aim to make the distribution of other factors the same in treatment and control groups.
15
Other methods…
• • • Can be clinchers but none are self validating: they require substance-specific assumptions. The assumptions of the RCTs are supposedly all based on method.
– Econometric modelling – Process tracing – Derivation from theory Some methods merely suggest a conclusion or vouch for it.
– Causal relations from analogue systems – ‘Observational’ studies’.
Standard advice guides tell you to ignore other clinchers and vouchers.
16
Daft advice
HARD WON KNOWLEDGE 17
The RCT
• The RCT can produce
high quality
claims – claims we have good reason to judge true – that we may adduce as evidence. • But
what are they evidence for
?
• That’s our second requirement. We want high quality claims that
speak for
the policy. Showing that a claim is very likely true goes
no way
to showing that it is
relevant
to the truth of our policy hypotheses.
18
Turn now to Relevance
How do we decide which evidence supports which conclusions?
19
US Dept of Education Website
• • Strong evidence for your policy = two or more high quality RCTs in ‘settings similar to that of your schools/classrooms’.
Later elaboration adds 4 lines – trials on white suburban populations do not constitute strong evidence for large inner city schools serving primarily minority students.
20
A SIGNS Grades of recommendation At least one meta analysis, systematic review, or RCT rated as 1++, and directly applicable to the target population ; or A systematic review of RCTs or a body of evidence consisting principally of studies rated as 1+, directly applicable to the target population, and demonstrating overall consistency of results B A body of evidence including studies rated as 2++, directly applicable to Extrapolated evidence from studies rated as 1++ or 1+ C A body of evidence including studies rated as 2+, directly applicable to the Extrapolated evidence from studies rated as 2++ D Evidence level 3 or 4; or Extrapolated evidence from studies rated as 2+ 21
Practical advice about relevance???
So—look for high quality is ‘ directly applicable population’ !!!!
evidence (1++) that to the target For better advice read Cartwright & Hardie 22
23
RCTs cannot hand evidentiary support directly to effectiveness claims 24