The impact of the STARD checklist on the reporting of studies of diagnostic accuracy in obstetrics and gynecology literature Class 1: Ana Costa, André.
Download ReportTranscript The impact of the STARD checklist on the reporting of studies of diagnostic accuracy in obstetrics and gynecology literature Class 1: Ana Costa, André.
Slide 1
The impact of the STARD checklist
on the reporting of studies of diagnostic
accuracy in obstetrics and gynecology
literature
Class 1: Ana Costa, André Sena, Duarte Pinto, Inês Brandão, João Silva, Lídia Sousa, Luís Barros,
Márcia Marinho, Maria Rocha, Pedro Falcão, Rafael Silva, Sara Tapa, Susana Franco.
Supervisors: Altamiro da Costa Pereira, MD, PhD; Cristina Santos, MD, PhD
Introduction to Medicine; Porto Faculty of Medicine 2006
Slide 2
Index
1) Introduction
2) Objective
3) Methods
4) Acknowledgements
Slide 3
Introduction
• Obstetrics and Gynecology are branches of
medicine.(*)
• These areas depend on diagnostic tests.(*)
• Studies of the new diagnostic tests, and their
development, are reported in scientific
articles. (**)
(*) - Cunningham, F.Gary. Williams obstetrics, 26ª edição (2005)
- Berek, Jonathan S. Novak’s gynecology 12ª edição (1996)
(**) - Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studios of diagnostic accuracy: the
STARD initiative.Annuals of internal Medicine. 2003; 138:40-44
Slide 4
Introduction
• The reports not always are well made and this
may lead to unwanted consequences. (*)
• STARD was created to guideline the reporting
of studies of diagnostic accuracy. (*)
(*) - Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studios of diagnostic accuracy: the
STARD initiative.Annuals of internal Medicine. 2003; 138:40-44
Slide 5
Objective
To evaluate the impact of the introduction of the
STARD checklist on the reporting of diagnostic
accuracy, in obstetrics and gynecology literature.
Slide 6
Methods
Articles’ search
Articles’ selection
Selected Articles’ analysis
Statistical analyze of the extracted data
Slide 7
Methods
Articles’ search
Query used in MedLine:
(((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR
"specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All
Fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All
Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive
values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR
"reference values/standards"[All Fields]) OR ((((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc
analysis"[All Fields]) OR "roc and"[All Fields]) OR "roc area"[All Fields]) OR "roc auc"[All Fields]) OR "roc
characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All
Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields]) OR ("likelihood ratio"[All Fields]))
AND
(hum reprod update[Journal] OR Obstet Gynecol[Journal] OR Menopause [Journal] OR Hum
Reprod[Journal] OR Obstet Gynecol Surv[Journal] OR Fertil Steril[Journal] OR Am J Obstet
Gynecol[Journal])
• This query was based on a search strategy in PubMed (MEDLINE) for publications about the evaluation of diagnostic
accuracy, suggested by a research article:
Devillé, W. L. et. al., Conducting systematic reviews of diagnostic studies: didactic guidelines
Slide 8
Methods
Articles’ search
Slide 9
Methods
Articles’ selection
• Creation of 2 groups (with 2 elements each*):
- one group read and analyse the abstract of each
article published in 1998
- the other do the same for the those published in 2005
*Note: each element of the group works independently and in the end, if occurs a
disagreement about de inclusion of exclusion of an article the entire article is read. Even so,
if no agreement is achieved, an expert e called to make the final decision
Slide 10
Methods
Articles’ selection
Inclusion criteria:
• The article is a report of a study of diagnostic accuracy
• The diagnostic test is used in obstetrics and/or genecology
• Was published in 1998 or in 2005
Slide 11
Methods
Articles’ selection
Exclusion criteria:
• The article could not be found in the Internet, the facilities of the
School of Medicine or the local libraries
• The article is written in languages other than English
• The article is a clinical trial, a letter, an editorial, a practice
guideline, a meta-analyse or a review
Slide 12
Methods
Selected Articles’ analysis
STARD checklist
Section and topic
Item
TITLE/ABSTRACT/
KEYWORDS
1
The article as a study on diagnostic accuracy (recommend MeSH heading 'sensitivity and specificity')
1
2
The research question(s), such as estimating diagnostic accuracy or comparing accuracy between tests or across
participant groups
1
INTRODUCTION
Describe
METHODS
Participants
Reference standard
Test methods
Statistical methods
Value
1
3
The study population: the inclusion and exclusion criteria, setting(s) and location(s) where the data were collected
1
4
Participant recruitment: was this based on presenting symptoms, results from previous tests, or the fact that the
participants had received the index test(s) or the reference standard?
1
5
Participant sampling: was this a consecutive series of patients defined by selection criteria in (3) and (4)? If not specify
how patients were further selected.
1
6
Data collection: were the participants identified and data collected before the index test(s) and reference standards were
performed (prospective study) or after (retrospective study)?
1
7
The reference standard and its rationale
1
8
Technical specification of material and methods involved including how and when measurements were taken, and/or cite
references for index test(s) and reference standard
0,5
0,5
9
Definition and rationale for the units, cutoffs and/or categories of the results of the index test(s) and the reference
standard
0,5
0,5
10
The number, training and expertise of the persons (a) executing and (b) reading the index test(s) and the reference
standard
0,5
0,5
11
Whether or not the reader(s) of the index test(s) and reference standard were blind (masked) to the results of the other
test(s) and describe any information available to them
0,5
0,5
12
Methods for calculating measures of diagnostic accuracy or making comparisons, and the statistical methods used to
quantify uncertainty (e.g. 95% confidence intervals)
1
13
Methods for calculating test reproducibility, if done
0,5
0,5
Article
score
Slide 13
Methods
Selected Articles’ analysis
Section and topic
STARD checklist
Item
Describe
Value
Article
score
RESULTS
Participants
Reference standard
Test results
Estimation
DISCUSSION
14
When study was done, including beginning and ending dates of recruitment
1
15
Clinical and demographic characteristics (e.g. age, sex, spectrum of presenting symptom(s), comorbidity,
current treatment(s), recruitment center)
1
16
How many participants satisfying the criteria for inclusion did or did not undergo the index test and/or the
reference standard; describe why participants failed to receive either test (a flow diagram is strongly
recommended)
1
17
Time interval and any treatment administered between index and reference standard
1
18
Distribution of severity of disease (define criteria) in those with the target condition; describe other
diagnoses in participants without the target condition
1
19
A cross tabulation of the results of the index test(s) by the results of the reference standard; for continuous
results, the distribution of the test results by the results of the reference standard
1
20
Indeterminate results, missing responses and outliers of index test(s) stratified by reference standard result
and how they were handled
1
21
Adverse events of index test(s) and reference standard
1
22
Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals)
1
23
Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if
done
1
24
Measures of test reproducibility, if done
25
The clinical applicability of the study findings
0,5
0,5
1
Bossuyt, P. M. et. al., Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative, Annals
of Internal Medicine Vol 138 – nº1, 7 January 2003
Slide 14
Methods
Selected Articles’ analysis
• 3 groups ():
- G1: Title, Abstract, Keywords, Introduction
- G2: Methods
- G3: Results, Discussion
• 1 group (α):
- Makes the final decision when occur disagreements
within groups in the classification of the article
- In the end, analyses 1/3 of the articles, randomly selected
Slide 15
Methods
Statistical analyze of the extracted data
Points to analyse:
• Final score of each article.
• Mean score of the articles for each year (1998 and 2005)
• Comparison of the 2 means
• Interobserver variability
• The status of each STARD item in each year
Slide 16
Methods
Statistical analyze of the extracted data
How ?
• Creation of a data base, in SPSS
• Determination of the scores of each article
• Execution of an Inter-class correlation coefficient and the limits of
Bland and Altaman
• Determination of mean, range, and standard deviation of each
item of the STARD, for each year
• Determination of mean score of the articles, for each year
• Execution of an unpaired two-tailed t-test for independent samples
(significance level 0,05)
Slide 17
Results (Articles’ search)
Slide 18
Planning
Slide 19
Acknowledgments
• Professor Altamiro da Costa Pereira
• Professor Cristina Santos
The impact of the STARD checklist
on the reporting of studies of diagnostic
accuracy in obstetrics and gynecology
literature
Class 1: Ana Costa, André Sena, Duarte Pinto, Inês Brandão, João Silva, Lídia Sousa, Luís Barros,
Márcia Marinho, Maria Rocha, Pedro Falcão, Rafael Silva, Sara Tapa, Susana Franco.
Supervisors: Altamiro da Costa Pereira, MD, PhD; Cristina Santos, MD, PhD
Introduction to Medicine; Porto Faculty of Medicine 2006
Slide 2
Index
1) Introduction
2) Objective
3) Methods
4) Acknowledgements
Slide 3
Introduction
• Obstetrics and Gynecology are branches of
medicine.(*)
• These areas depend on diagnostic tests.(*)
• Studies of the new diagnostic tests, and their
development, are reported in scientific
articles. (**)
(*) - Cunningham, F.Gary. Williams obstetrics, 26ª edição (2005)
- Berek, Jonathan S. Novak’s gynecology 12ª edição (1996)
(**) - Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studios of diagnostic accuracy: the
STARD initiative.Annuals of internal Medicine. 2003; 138:40-44
Slide 4
Introduction
• The reports not always are well made and this
may lead to unwanted consequences. (*)
• STARD was created to guideline the reporting
of studies of diagnostic accuracy. (*)
(*) - Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studios of diagnostic accuracy: the
STARD initiative.Annuals of internal Medicine. 2003; 138:40-44
Slide 5
Objective
To evaluate the impact of the introduction of the
STARD checklist on the reporting of diagnostic
accuracy, in obstetrics and gynecology literature.
Slide 6
Methods
Articles’ search
Articles’ selection
Selected Articles’ analysis
Statistical analyze of the extracted data
Slide 7
Methods
Articles’ search
Query used in MedLine:
(((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR
"specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All
Fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All
Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive
values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR
"reference values/standards"[All Fields]) OR ((((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc
analysis"[All Fields]) OR "roc and"[All Fields]) OR "roc area"[All Fields]) OR "roc auc"[All Fields]) OR "roc
characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All
Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields]) OR ("likelihood ratio"[All Fields]))
AND
(hum reprod update[Journal] OR Obstet Gynecol[Journal] OR Menopause [Journal] OR Hum
Reprod[Journal] OR Obstet Gynecol Surv[Journal] OR Fertil Steril[Journal] OR Am J Obstet
Gynecol[Journal])
• This query was based on a search strategy in PubMed (MEDLINE) for publications about the evaluation of diagnostic
accuracy, suggested by a research article:
Devillé, W. L. et. al., Conducting systematic reviews of diagnostic studies: didactic guidelines
Slide 8
Methods
Articles’ search
Slide 9
Methods
Articles’ selection
• Creation of 2 groups (with 2 elements each*):
- one group read and analyse the abstract of each
article published in 1998
- the other do the same for the those published in 2005
*Note: each element of the group works independently and in the end, if occurs a
disagreement about de inclusion of exclusion of an article the entire article is read. Even so,
if no agreement is achieved, an expert e called to make the final decision
Slide 10
Methods
Articles’ selection
Inclusion criteria:
• The article is a report of a study of diagnostic accuracy
• The diagnostic test is used in obstetrics and/or genecology
• Was published in 1998 or in 2005
Slide 11
Methods
Articles’ selection
Exclusion criteria:
• The article could not be found in the Internet, the facilities of the
School of Medicine or the local libraries
• The article is written in languages other than English
• The article is a clinical trial, a letter, an editorial, a practice
guideline, a meta-analyse or a review
Slide 12
Methods
Selected Articles’ analysis
STARD checklist
Section and topic
Item
TITLE/ABSTRACT/
KEYWORDS
1
The article as a study on diagnostic accuracy (recommend MeSH heading 'sensitivity and specificity')
1
2
The research question(s), such as estimating diagnostic accuracy or comparing accuracy between tests or across
participant groups
1
INTRODUCTION
Describe
METHODS
Participants
Reference standard
Test methods
Statistical methods
Value
1
3
The study population: the inclusion and exclusion criteria, setting(s) and location(s) where the data were collected
1
4
Participant recruitment: was this based on presenting symptoms, results from previous tests, or the fact that the
participants had received the index test(s) or the reference standard?
1
5
Participant sampling: was this a consecutive series of patients defined by selection criteria in (3) and (4)? If not specify
how patients were further selected.
1
6
Data collection: were the participants identified and data collected before the index test(s) and reference standards were
performed (prospective study) or after (retrospective study)?
1
7
The reference standard and its rationale
1
8
Technical specification of material and methods involved including how and when measurements were taken, and/or cite
references for index test(s) and reference standard
0,5
0,5
9
Definition and rationale for the units, cutoffs and/or categories of the results of the index test(s) and the reference
standard
0,5
0,5
10
The number, training and expertise of the persons (a) executing and (b) reading the index test(s) and the reference
standard
0,5
0,5
11
Whether or not the reader(s) of the index test(s) and reference standard were blind (masked) to the results of the other
test(s) and describe any information available to them
0,5
0,5
12
Methods for calculating measures of diagnostic accuracy or making comparisons, and the statistical methods used to
quantify uncertainty (e.g. 95% confidence intervals)
1
13
Methods for calculating test reproducibility, if done
0,5
0,5
Article
score
Slide 13
Methods
Selected Articles’ analysis
Section and topic
STARD checklist
Item
Describe
Value
Article
score
RESULTS
Participants
Reference standard
Test results
Estimation
DISCUSSION
14
When study was done, including beginning and ending dates of recruitment
1
15
Clinical and demographic characteristics (e.g. age, sex, spectrum of presenting symptom(s), comorbidity,
current treatment(s), recruitment center)
1
16
How many participants satisfying the criteria for inclusion did or did not undergo the index test and/or the
reference standard; describe why participants failed to receive either test (a flow diagram is strongly
recommended)
1
17
Time interval and any treatment administered between index and reference standard
1
18
Distribution of severity of disease (define criteria) in those with the target condition; describe other
diagnoses in participants without the target condition
1
19
A cross tabulation of the results of the index test(s) by the results of the reference standard; for continuous
results, the distribution of the test results by the results of the reference standard
1
20
Indeterminate results, missing responses and outliers of index test(s) stratified by reference standard result
and how they were handled
1
21
Adverse events of index test(s) and reference standard
1
22
Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals)
1
23
Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if
done
1
24
Measures of test reproducibility, if done
25
The clinical applicability of the study findings
0,5
0,5
1
Bossuyt, P. M. et. al., Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative, Annals
of Internal Medicine Vol 138 – nº1, 7 January 2003
Slide 14
Methods
Selected Articles’ analysis
• 3 groups ():
- G1: Title, Abstract, Keywords, Introduction
- G2: Methods
- G3: Results, Discussion
• 1 group (α):
- Makes the final decision when occur disagreements
within groups in the classification of the article
- In the end, analyses 1/3 of the articles, randomly selected
Slide 15
Methods
Statistical analyze of the extracted data
Points to analyse:
• Final score of each article.
• Mean score of the articles for each year (1998 and 2005)
• Comparison of the 2 means
• Interobserver variability
• The status of each STARD item in each year
Slide 16
Methods
Statistical analyze of the extracted data
How ?
• Creation of a data base, in SPSS
• Determination of the scores of each article
• Execution of an Inter-class correlation coefficient and the limits of
Bland and Altaman
• Determination of mean, range, and standard deviation of each
item of the STARD, for each year
• Determination of mean score of the articles, for each year
• Execution of an unpaired two-tailed t-test for independent samples
(significance level 0,05)
Slide 17
Results (Articles’ search)
Slide 18
Planning
Slide 19
Acknowledgments
• Professor Altamiro da Costa Pereira
• Professor Cristina Santos