Transcript Interpreting Clinical Trial Results

Interpreting Clinical Trial Results
Betty Anne Schwarz MSc, BA, RN
OHRI Clinical Research Conference Workshop
October 16, 2012
Interpreting Clinical Trial Results
“I have no conflicts of interest related to this presentation”
Interpreting Clinical Trial Results
Workshop Applicability
 Anyone who is working in clinical trials that is accustomed to reading
published literature (RCTs)
Interpreting Clinical Trial Results
Objectives
 Review what is considered statistically significant when interpreting
clinical trial results
 Illustrate how statistically significant is not the same as clinically
significant
 Interactive workshop will involve participation of the audience
through review of specific trials to determine clinical significance
with the tools provided
Interpreting Clinical Trial Results
Outline of Workshop
I.
Review statistical features that are routinely expected within
published trial results
II.
Focus on the merits of considering MCID (minimally clinical
important difference)
Look at examples of positive vs. negative trials
III. Break into groups – utilize tools provided in reviewing specific
trials to determine clinical significance
IV. Review and Summarize
Interpreting Clinical Trial Results
Statistics & Clinical Trials
 Interpreting trials – depends on level of comprehension and
experience
I. Key Statistical Terms
 Choice of statistical test utilized to analyze data (interval or
categorical, paired vs. unpaired) depends on the data being
analyzed
 Null hypothesis proposes ~ no difference between study groups with
respect to variable's of interest
Ref: (2007) McCluskey & Ghaaliq Lalkhen
Interpreting Clinical Trial Results
I. Key Statistical Terms
 Unpaired vs. Paired Data ~ compare effects of an intervention within
study samples imperative groups are as similar as possible
 Randomization – large enough sample size
 Ensures group differences within the groups cancel out as they may
influence outcome of interest
(weight, age, sex ratio, smoking habit)
 Study contained independent groups – unpaired statistical tests
 i.e., comparison of efficacy for two different drugs – HTN trial
Ref: (2007) McCluskey & Ghaaliq Lalkhen
Interpreting Clinical Trial Results
I. Statistical vs. clinical significance
 One should not be confused by another
Scenario
 Comparison of 2 hypotensive agents: mean arterial pressure after
Rx. with drug A is 2 mm lower < Rx. with drug B.
 If sample size is not large enough even a small difference between
the 2 groups may be statistically significant with a P <0.05.
 The clinical significance of a 2 mm Hg drop in mean arterial
pressure is small and therefore not clinically significant
Ref: (2007) McCluskey & Ghaaliq Lalkhen
Interpreting Clinical Trial Results
I. “Statistical Significance” and P Values
 “Not everything that can be counted counts, and not everything that
counts can be counted” Albert Einstein
 Statistically significant simply means that a result is most likely
caused by something other than chance
 In other words ~ significant does not mean important
 Segue into what’s considered clinically significant
 Interpreting trial results requires a P value (conventional P < 0.05
statistically significant). P ≥ 0.05 not significant, P <0.01 highly
significant
Ref: (2004) libdoc.who.int.,(2007) McCluskey & Ghaaliq Lalkhen,
Interpreting Clinical Trial Results
I. Confidence Intervals (CI)
 Range of sample data measures the probability that a population
parameter will fall between two set values. The confidence interval
can take any number of probabilities, with the most common being
95% or 99%
 CI should always be presented for the relative risk (RR) and odds
ratio (OR)
 A RR or OR of 1 ~ no association between the risk factors and the
disease under study
 If RR > 1, but CI overlaps 1 – the increase in risk is not statistically
significant and can attributed to chance
Ref: (2004) libdoc.who.int
Interpreting Clinical Trial Results
I. Important Variables and Concepts
 Study power ~ Type I and Type II errors
 After data has been analyzed, the null hypothesis is either accepted
or rejected based on the P value
 If the null hypothesis is true and the P value (P < 0.05) is obtained;
incorrect inference is drawn that the data collected from the sample
groups is different : TYPE I STATISTICAL ERROR
 From the data collected conclude null hypothesis is false but the P
value obtained is ≥0.05 ~ conclude sample groups are similar but in
fact missed a real difference: TYPE II STATISTICAL ERROR
Ref: (2004) libdoc.who.int
Interpreting Clinical Trial Results
I. Important Variables and Concepts
 Interpreting diagnostic tests ~ sensitivity and specificity
 Comparing to gold standard or reference standard
 Sensitivity – ability of a test to single out people who have disease.
Low sensitivity ~ many false positives
 Specificity – ability of a test to identify people who do not have a
disease (negative). A low sensitivity ~ many false positives
 Predictive Value (PV) – the frequency in which a positive test
actually signifies disease
 Confound
 Bias
Ref: (2004) libdoc.who.int
Interpreting Clinical Trial Results
II. MCID (minimally clinical important difference)
 Interpretation of RCTs emphasis on statistical significance rather
than clinical relevance
 CONSORT (Consolidated Standards of Reporting Trials) statement
developed to help authors improve their methods in reporting trials
results with the aid of a checklist and flow diagram
 Enable readers to comprehend the trial methodology and assess the
validity of the results
 Recently revised based on new evidence and address criticism
Ref: (2001) Chan et al., (2001) Moher et al.
Interpreting Clinical Trial Results
II. MCID (minimally clinical important difference)
 CONSORT revised – failed to recommend that authors specifically
discuss the clinical importance of their results
 Chan et al., (2001) randomly chose 27 (total 266) published RCTs
from 5 major journals over a 1 year time period that were
independently reviewed by 4 reviewers to discern if factors they
considered as clinically relevant were stated in the study results
Ref: (2001) Chan et al., (2001) Moher et al.
Interpreting Clinical Trial Results
II. MCID (minimally clinical important difference)
 Specifically ~ primary outcome clearly defined, expected difference
between the two groups reported used in the calculation of the
sample size (delta value), whether the results were based on
minimal clinically important difference of the intervention, the
statistical significance of the results, presentation of confidence
intervals pertinent to the study findings and finally the authors overall
interpretation of the clinical relevance of the trial results
Ref: (2001) Chan et al., (2001) Moher et al.
Interpreting Clinical Trial Results
II. MCID (minimally clinical important difference)
 Defined as the smallest treatment effect required that would render
a change in the management of a patient taking into account the
side effects, costs and overall inconveniences
 Key concept to be considered
 Designing RCT Trial ~ calculate the sample size needed to detect
magnitude of difference between the treatment groups that the study
can reliably detect (delta value)
 Key point ~ for a trial to have a reasonable chance of detecting
clinically important effect size, the delta value needs to reflect the
MCIDs for the trial interventions
Ref: (2001) Chan et al., (2001) Moher et al.
Interpreting Clinical Trial Results
II. MCID (minimally clinical important difference)
 Example: individuals without a previous history of MI or CVA; regular
use of ASA will reduce the incidence of MI by 0.2% per year (from a
baseline rate of 0.7%/year to 0.5%/year which is a RR reduction of
about 25%)
 Benefit is offset by a concomitant absolute increase in the chance of
CVA by 0.02% per year (from 0.30%/year – 0.32%/year, which is
now a relative increase of about 10%/year) and GI bleed of about
1% per year (from about 1%/year – 2%/year)
 Thoughts?
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
II. MCID (minimally clinical important difference)
 Weigh the benefits and disadvantages of taking ASA in this clinical
situation; expert panel recommended not to take ASA in the
prevention of MI as there was insufficient evidence to overcome the
increased incidence of CVA, GI bleed in this low risk group.
 Therefore; the efficacy of ASA in this particular scenario was
insufficient to either meet or exceed its MCID
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
II. MCID (minimally clinical important difference)
 Compare the actual study results ~ include point estimates and
accompanying confidence intervals with the MCID values will render
the clinical importance of the study results
 Point estimate: can be either a single number or a range of scores.
 Point estimates are not usually as informative as confidence
intervals. Their importance lies in the fact that many statistical
formulas are based on them
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
II. MCID (minimally clinical important difference)
 If the MCID estimate is less than the value of the lower limit of the
95% CI ~ results are statistically significant and most likely clinically
significant
 If the MCID value is greater than the upper limit of the 95% CI, the
results are very likely to be less clinically significant
 If the results provided for the MCID are somewhere in the middle of
the CI – the clinical importance is less clear
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Methods
 December 1, 1998 – November 30, 1999 of 266 articles randomly
chosen - 27 RCTS were selected from 5 peer reviewed journals
 Utilized standard data collection sheets to evaluate key features
when interpreting the trial results and whether the results were
clinically significant (MCID)
 Disagreements were resolved collaboratively
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Methods (hand-out)
1. Primary Outcome – clearly defined in the methods section and is
also defined within the calculation of a sample size
 No sample size reported – remainder of the article was reviewed for
an explicitly stated primary outcome
 No primary outcome – the outcome that was deemed of greatest
clinical importance was chosen
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Methods
2. Reporting of a Sample Size Delta Value
 Magnitude of difference in outcomes between treatment groups that
the trial was attempting to detect
 Recorded as either the author’s interpretation of a MCID for the
various study intervention, and whether or not the value was in
absolute or relative terms
3. Statistical significance of results for the primary outcome
 P value < 0.05%
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Methods
4. Confidence Intervals
 Primary outcome – whether the 95% CI surrounded the point
estimate for the efficacy of a particular trial intervention
5. Authors overall interpretation of clinical importance found from their
trial results
 If the results were discussed – clinical relevance explicitly noted
whereas; indirect reference to the results was considered implicitly
reported
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Methods
Levels of Justification
 Rating scale used to grade the level of strength that was developed
to justify the reviewers overall interpretation of the clinical
importance from the trial results obtained
 Level 1: the clinical importance of the primary study result in relation
to the work done to determine the MCID
 Level 4: no accompanying justification
Ref: (2001) Chan et al.,
Methodological attributes important in the
interpretation of study results from a clinical
perspective
Section
Attribute
Methods
• Explicit primary outcome
stated
• Expected magnitude of
difference “delta value” stated
• Expected magnitude of
difference identified explicitly
as the MCID
• Delta/MCID value reported as
an absolute value
Ref: (2001) Chan et al.,
Methodological attributes important in the
interpretation of study results from a clinical
perspective
Section
Attribute
Results
• Statistical significance of
primary outcome reported
• Confidence intervals for
primary outcome reported
Discussion
• Clinical importance of primary
outcome discussed
• Discussion explicit or implicit
• Level of justification
• Appropriate clinical
interpretation of results
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Workshop Articles:
1. GISSI-Prevenzione Investigators. Dietary Supplementation with
n-3 polyunsaturated fatty acids and vitamin E after myocardial
infarction: results of the GISSI-Prevenzione trial. Lancet
1999;335 (9177):447-55.
2. Cleare AJ., Heap, E, Malhi GS, Wessely, S, O’Keane V, Miell J.
Low-dose hydrocortisone in chronic fatigue syndrome. A
randomized crossover trial. Lancet 1999;353:455-58.
Ref: (2001) Chan et al., (1999) Cleare et al.,
Interpreting Clinical Trial Results
Workshop Articles:
1. GISSI-Prevenzione Investigators. Dietary Supplementation with n-3
polyunsaturated fats acids and vitamin E after myocardial
infarction: results of the GISSI-Prevenzione trial. Lancet 1999;335
(9177):447-55.
 Chan et al., (2001) found that 26% studies did not report a sample
size or a delta value
 If reported – reported in relative instead of absolute terms
 Likewise the delta value did not appear to reflect the MCID of the
intervention
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Workshop Articles:
1. GISSI-Prevenzione Trial
 Assessed the effect of dietary supplementation with
polyunsaturated fats demonstrated an absolute decrease of 1.3%
(95% CI 0.1%-2.6%) in the primary outcome (combined end point
of death, nonfatal MI and stroke)
 In the sample size calculation; delta value was a 4% absolute
difference between groups over 3.5 year period
 Therefore: the efficacy of the intervention found by the study was
significantly smaller than the sample size delta
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Workshop Articles:
1. GISSI-Prevenzione Trial
 If we assume that this value represents the MCID of the
intervention, the magnitude of the intervention's effect is definitely
not clinically important.
 Authors reported “a clinically important and statistically significant
benefit”, their estimation of the MCID of the intervention is not
reflected in the sample size delta
Ref: (2001) Chan et al., (2002) Hing et al.,
Interpreting Clinical Trial Results
Workshop Articles:
2. Cleare AJ., Heap, E, Malhi GS, Wessely, S, O’Keane V, Miell J.
Low-dose hydrocortisone in chronic fatigue syndrome. A
randomized crossover trial. Lancet 1999;353:455-58.
 Assessed the effect of low-dose hydrocortisone on chronic fatigue
disorder, a 9 point reduction on a fatigue scale was deemed to be
clinically important.
 However; the delta value for the sample size was reported as a 4
point reduction on the same scale
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Workshop Articles:
2. Hydrocortisone in chronic fatigue study
 Therefore; the authors reported that the study result, a statistically
significant 4.5 point reduction on the fatigue scale when comparing
the intervention group with the control group , was not clinically
important.
Ref: (2001) Chan et al.,
Interpreting Clinical Trial Results
Interpreting Study Results
 Study results are regarded as statistically significant (P < 0.5) when
values indicating the lower limit of the 95% confidence interval (CI)
are greater than the null effect
 Judging clinical relevance can be difficult
 Clinical relevance can take 4 forms, pending the relationship of the
MCID on the intervention to the point estimate (the best single value
of the efficacy of the intervention that has been derived from the
study results) and the 95% CI surrounding it
Ref: (2002) Hing et al.,
Interpreting Clinical Trial Results
Interpreting Study Results
 Definite: when the MCID is smaller that the lower limit of the 95% CI
 Probable: when the MCID is greater than the lower limit of the 95%
CI, but smaller that the point estimate of the efficacy of the
intervention
 Possible: when the MCID is less than the upper limit of the 95% CI,
but greater than the point estimate of the efficacy of the intervention
 Definitely not: when the MCID is greater than the upper limit of the
95% CI
Ref: (2002) Hing et al.,
Interpreting Clinical Trial Results
Study Results of Definite Clinical Importance
 Randomized placebo controlled trial that assessed the efficacy of
the addition of spironlactone to the medication regime of patients
with severe heart failure.
 Results demonstrated that the study intervention reduced the
chance of death by an absolute value of 11.4% (95% CI, 6.7% to
16.1%). The delta value calculated for the sample size was an
absolute 6.5% reduction in mortality.
 If the delta value is assumed to be the MCID, the results of this
study can be interpreted as both statistically significant and clinically
important
Ref: (2002) Hing et al.,
Ref: (2002) Hing et al.,
Interpreting Clinical Trial Results
Limitations of MCID
 Methodological limitation: uncertainty in the choice of MCID
 MCID values will vary from person to person depending on
values/perspectives
 Without widespread used of sample size deltas that accurately
reflect the MCID is it appropriate to use the sample size deltas as
benchmarks for determining the clinical relevance?
 However………..
 Compare the relationship of possible MCID values to the point
estimate of the efficacy of the intervention and it’s CI – this will allow
determining the level of clinical significance
Ref: (2002) Hing et al.,
Interpreting Clinical Trial Results
Summary
 Formal statistical methods of analyzing and reporting clinical trial
results are routinely performed and expected within the literature
 Interpreting and reporting trial results from the perspective of clinical
relevance is not always done with the same emphasis
 Creates imbalance as clinically significant trials are often considered
clinically significant and those that are statistically insignificant are
also clinically unimportant
 Encourage evaluate both when interpreting clinical trial results with
the tools reviewed today (MCID)
References
1. Chan, K. B. J. Y., Man-Son-Hung, M., Molnar, F. J. & Laupacis, A. (2001) 'How well is the clinical
importance of study results reported? As assessment of randomized controlled trials', CMAJ, 165, (9),
pp. 1197-1202.
2. Man-Son-Hing, M., Laupacis, A., O’Rourke, K., Molnar, F. J., Mahon, J., Chan, K. B. Y. & Wells, G.
(2002) ‘Determination of the Clinical Importance of Study Results’, JGIM, (17) pp. 469-476.
3. McCluskey, A. & Ghaaliq Lalkhen, A. (2007) ‘Statistical IV: Interpreting the results of statistical
tests’, Continuing Education in Anesthesia Care & Pain, [Online]. DOI:
10.1093/biaceaccp/mkm042ical, Volume 7, (6) Accessed: September 16, 2012
4. Moher, D., Schulz, K. F. & Altman, D. G. (2001) ‘The CONSORT statement revised
recommendations for improving the quality of reports of parallel group randomized trials’, BMC
Medical Research Methodology, [Online]. Available from:
http://www.biomedcentral.com/content/pdf/1471-2288-1-2.pdf
Accessed: September 17, 2012.
5. Moher, D., Dulberg,, C. S. & Wells, G. (1994) ‘Statistical Power, Sample Size, and Their Reporting
in Randomized Clinical Trials’, JAMA, 272 (2) pp. 122-124.
6. Library Doctor WHO International (1994) ‘Chapter 9 Interpreting research results’, Available from:
http://whqlibdoc.who.int/emro/2004/9290213639_chap9.pdf (Accessed: September 21, 2012).
Thank you!