Transcript Where is epidemiology going, Part III

Registration of observational
research
• Should STROBE change some of its
wordings?
• Should STROBE take a stand?
History
• Sept 2009, meeting London, organized by ECETOC
(European Center for Ecotoxicology and Toxicology of
Chemicals) – an organisation of the European chemical
industry.
• Attended by invited epidemiologists (Dickersin, Samet,
Weed, Porta) And editors BMJ (Trish Groves?) and
Lancet (Bill Summerskill)
• After meeting:
– Paper by ECETOC (copyrighted by them, only accessible with
permission), subtle call on STROBE to endorse registration
– Editorials Lancet & BMJ calling for registration.
– Overall: wish to emulate registration as it exists for RCTs
ECETOC, quotes
• There is evidence that a proportion of observational
epidemiology studies are not published, either because
they are not submitted for publication or because they
were rejected by journals. This results in selective
reporting and publication bias, accompanied by a
distorted representation of observational study findings
in the open scientific literature. One solution to this
problem is to create a system for tracking, organising
and disseminating information about observational
epidemiology studies in a similar way to clinical trials.
• The workshop participants did not come to a consensus
on whether registration would be required or voluntary. If
the latter it could be stated that potential authors would
be encouraged to register and comply with the STROBE
guidelines (von Elm, 2007) as part of a quality standard
and quality indicator for journal articles.
Editorial Lancet, quotes
• At one extreme, the value of hypothesisgenerating observations in a small case series
might not be foreseen and its communication
should not be constrained by unnecessary
bureaucracy. At the other, a large, hypothesisdriven cohort study, with a solid protocol that has
taken years to organise and fund, has little
excuse not to register its intent.
• Furthermore, we welcome inclusion of the
protocol when an observational study is
submitted, which we will forward in confidence to
peer-reviewers.
Editorial BMJ, opening
Observational studies, such as cohort and case-control
studies, are an important form of medical research, but they
are also vulnerable to bias and selective reporting. They
often produce large datasets that can be subjected to
multiple analyses. Researchers may then craft a paper that
selectively emphasises certain results, often those that are
statistically significant or provocative. These decisions may
reflect strong financial or academic interests and prior
beliefs. At present, consumers of observational research
cannot easily distinguish hypothesis driven studies from
exploratory, post hoc data analyses. Researchers do not
routinely disclose the number of additional analyses
performed. Nor is there any satisfactory way to know
whether the research questions or methods of statistical
analysis diverged from those initially planned. It has been
observed that there is “little or no penalty” for data dredging
and selective reporting. Rather than attracting censure it can
“get you into the BMJ and the Friday papers”.
Further down BMJ editorial
The STROBE statement has improved reporting of
observational studies by asking authors to spell out in their
papers exactly what they did during their studies. It asks
authors to “explain the scientific background and rationale
for the investigation being reported” and “state specific
objectives, including any prespecified hypotheses.” As
journal editors, we have probably not paid enough attention
to emphasising these points, but we aim to do so from now
on. However, like most reporting statements, STROBE is
aimed at improving the clarity of study reporting and comes
too late to influence study design
“Multiplicity” is main argument
• Good science is equated with: adherence
to a hypothesis that was prespecified, an
analysis that was prespecified
• Next a few words about ‘transparency’ and
‘ethics’
Reaction in epi community, so far
• 6 opinion papers (5 commentaries plus 1 editorial) in
Epidemiology
• Observational research has produced many results that
we all adhere to: from brushing your teeth, to seat belts,
non-smoking, care with ionizing radiation etc
• “Ethical” argument largely dismissed as being very
different from RCTs: secondary analyses of existing
data, no intervention
• Nobody against idea that knowing which studies and
data exist might be a good thing for science (whether
voluntary or not, difference in views)
• Main objection is against the ‘multiplicity reasoning’.
• Several epidemiologists who attending to the ECETOC
meeting have in the mean time dissented from the
ECETOC document: they want more facts and more
insight in pro, con and practicality – e.g. debate ACE
Debate about multiplicity in
pharmacoepidemiology ISPE August 2010
• Participants: Sonia Hernandez-Diaz, Stuart
Pocock, Ken Rothman, Stan Young
• Epi argument: good science needs multiplicity to
be able to understand Nature:
– Study several aspects of a question, several
outcomes of one exposure, and vice versa, to
understand how Nature works
– One learns during data analysis: about the data and
their mutual relation, much like lab scientists learn
from their previous experiment
What is real epi practice?
• In epidemiology, we always tell our PhD
students to have a prespecified question and
analysis plan when attacking a data set
• Reason is:
– Not to make findings more credible
– To avoid getting lost: to be able to trace your
reasoning. Equivalent to lab scientists: if you keep
changing your lab experiment without taking notes in
a lab journal, at certain point you do not know any
more where you are. In addition, you can only detect
something new if you now securely where you were
heading.
Spectre raises
• A ‘punitive’ registry: if you deviate, your
chances of publishing plunge – you are
treated with suspicion, as a “data dredger”.
• Question: whenever you have a new idea
during data-analysis, should you stop
analyzing and amend the registration of
your study?
• If you do that: you are already ‘cheating’;
you have already deviated.
• “Stop Science?” (Sonia Hernandez)
How to understand the difference in
views?
• Recent paper Parascandola about “Epistemic
Risk” 2010: “In drawing an inferential conclusion
or accepting a hypothesis as true, one taks on an
‘epistemic risk’ – the risk of being wrong.”
• Minimization of ER = maximal avoidance of type
I error: prespecified hypothesis, no multiple
testing, no subgroups. Standard safeguard for
RCTs – and rightly so:
– RCTs are set up with a strong background of
knowledge & start with rather high priors
– RCTs are therefore directly meant for decision making
& may have immediate consequences
Drawbacks of minimization of
Epistemic Risk in epidemiology
– Increases type II error: prevents to see new
things
– Calculation of risk of multiplicity are done
under ‘universal null hypothesis’: as if nothing
was ever true
– Full avoidance of epistemic risk is contrary to
aims of science: to advance knowledge by
finding new explanations
Epi community tolerant of ER
• Much more tolerant than RCT community,
because of different outlook: it is not clear
to them which error (type I or II) is always
the worst
• The aims of use epi data sets and studies
are changing all the time.
We are not the only ones
• Sir Peter Medawar (1963): “Is the scientific
paper a fraud?”
– A scientific paper does not describe the real history of
your research: the detours, blind alleys, etc. It is a fixed
IMRAD format to describe your results.
• George Whitesides (2004): Writing a paper
Epidemiologists’ evaluation of a study:
• What was in mind investigator is not
important: readers should make up their
own mind, and may be of a different mind
• Most important: clarity of data description
and analysis, is the analysis defensible,
prior info acceptable?
The main problem: observational
research is also used for regulation
• We want to understand Nature, to change it.
• Observational research may lead to recall of
drugs, occupational safety measures in chemical
industry, bans on smoking etc.
• Things often starts with investigations with low
priors; they are scrutinized and the process of
their scrutiny is visible to all (known as
‘contradictory findings’) – in contrast to RCTs
where the prior has already been scrutinized
The heart of the matter?
• “… academics and commentators… care more about whether
ideas are interesting than whether they are true. Politicians live
by ideas just as much as professional thinkers do, but they can't
afford the luxury of entertaining ideas that are merely interesting.
They have to work with the small number of ideas that happen to
be true and the even smaller number that happen to be
applicable to real life. In academic life, false ideas are merely
false and useless ones can be fun to play with. In political life,
false ideas can ruin the lives of millions and useless ones can
waste precious resources. An intellectual's responsibility for his
ideas is to follow their consequences wherever they may lead. A
politician's responsibility is to master those consequences…"
• Ignatieff, NYT 2007
Process by which observational research
leads to action
• Decision whether a hypothesis is acceptable demands:
– Alternative explanations be ruled out by analyses that try to
remedy the potential biases of the original studies
– Findings supported or plausible from a mechanistic point of view
(preferably experimental lab/animal data)
– Combined evidence sufficiently strong to make a sufficiently
strong bet on ‘causality’
• Action demands more: an assessment of the costs and
benefits of different decisions.
• Prespecification of hypotheses plays no role – reasoned
combination of evidence.
• Examples: 1959 paper about Smoking and Lung Cancer
by Cornfield et al. IARC assessments of carcinogens.
The challenge for epidemiology
• How can we have:
– The benefit of knowing what data exist, what studies exist
– Without the drawbacks of registration of prespecified hypotheses
• My personal answer:
– The idea behing a registration should be non-punitive; no implied
sanction for not adhering to hypothesis – which is also the only
condition under which people will really tell ‘what they did’
– In each instance: editors, reviewers and readers should have to
judge whether the changes in protocol, analysis, hypothesis
were a great thing, or whether they detract.
– Example of the latter: Comparative Effectiveness Research into
effects of therapy. Editors PLoS Med (personal communication)
think that for CER, it would be good to have a situation that is
similar to RCTs – e.g., CER of different anti-hypertensives – ad
hoc ‘nice subgroup’ might indeed be viewed with suspicion.
What could STROBE do (1)?
• Emphasize even more the reporting of the ‘why’ of
the research and/or analysis hypothesis and
analysis plans in intro? Question: how far does one
go? Is it really of interest to know every fleeting
thought that passes through the head of a data
analyst?
• Emphasize that there is nothing against improving
your analysis plans during the experience with the
study, or pursuing a new hypothesis based on
findings that arose during data analysis, but that
this should be clearly stated, preferably in the
introduction or in methods, sometimes even the
abstract, and that the reasons for reporting this
(supporting evidence) should be clearly described
in the discussion.
What could STROBE do (2)?
• Avoid generalities like ‘allowing for multiplicity’ –
the multiplicity may only be one or two ideas –
they should be specified; with examples?
– A study with everything prespecified that
adhered to it for reasons that are also
explained
– A study that started prespecified but
something new was found in data or came up
from other sources during analysis
– A study in which better insight was gained in
data during data analysis
What could STROBE do (3)?
• STROBE is about clarity of reporting to facilitate
the tasks of editors, reviewers and readers.
• It is not primary task of STROBE to call for
registration of prespecified hypotheses. The
enactment of a registration is the duty of others:
editors, epi societies etc.