Transcript Slide 1
The science of peer review
David C. Cone, MD Associate Professor of Emergency Medicine Yale University School of Medicine Editor-in-Chief
Academic Emergency Medicine
Donald M. Yealy, MD Professor and Chair of Emergency Medicine University of Pittsburgh Deputy Editor
Annals of Emergency Medicine
Disclosures
D. Cone’s department receives a stipend for his services as Editor-in Chief of
Academic Emergency Medicine
D. Yealy receives a stipend for
Annals of Emergency Medicine
service.
Neither has any financial interests in peer review No commercial interests or off-label uses to declare
Outline
1. What do we know about the science of the traditional peer review system?
2. What changes are being proposed?
3. How might these affect our careers?
Two facets of engaging the broader scientific community
“Front end” = peer review To determine what is “worth” publishing in the literature To optimize the message being presented “Back end” = post-publication discourse To allow others to comment and discuss after a paper makes it into the literature Occasionally, correct (“erratum”) or retract part or whole publication
The traditional peer review model
A decision editor determines if he/she needs more input to assess a paper. If so: The editor recruits a small number (2-5) of additional “expert” reviewers to aid The reviewers provide comments The editor uses collective wisdom to determine the disposition of the paper
The traditional post publication discourse model
Letters to the editor, with replies Most journals limit to one “round” of letters and author replies Local journal clubs Discussion in future papers on the topic Retraction, erratum, or other post publication piece if malfeasance, error, or another important concern arises that could alter the paper’s impact/message.
What do we know about peer review?
Remarkably little!
Not much is encouraging.
Much of the work on the science of peer review has been done by Mike Callaham, Editor-in-Chief of
Annals of Emergency Medicine
Mike Callaham’s evil twin?
Grunting for worms: seismic vibrations cause Diplocardia earthworms to emerge from the soil.
Mitra O, Callaham M , Smith ML, Yack JE. Biol Lett. 2009 Feb 23;5(1):16-9.
We can’t train reviewers!
“…formal, 4-hour, highly interactive workshop on peer review” Scores on reviews for next 2 yrs: Control: 0.11 (95%CI -0.25 to 0.48) Attendees: 0.10 (95%CI -0.20 to 0.39) Active recruitment: ○ “Efforts to aggressively recruit average reviewers to a second workshop were time consuming, had low success rates, and showed a similar lack of effect on ratings” Callaham ML, Schriger DL.
Ann Emerg Med
2002;40:323-8.
We can’t improve the reviewers we already have!
Written feedback from editor vs usual (receiving other reviews & decision letter) Mean individual reviewer rating change: Study 1: lower-quality reviewers ○ Control: 0.16 (95%CI -0.26 to 0.58) ○ Intervention: -0.13 (-0.49 to 0.23) Study 2: average reviewers, more feedback ○ Control: 0.12 (-0.20 to 0.26) ○ Intervention: 0.06 (-0.19 to 0.31).
Callaham ML , Knopp RK , Gallagher EJ .
JAMA
2002;287(21):2781
Peer reviewers miss stuff!
Fake manuscript describing a double blind, placebo-controlled study showing that IV propranolol reduced the pain of acute migraine headache 10 major and 13 minor errors inserted Sent to all 262
Annals
reviewers 203 (78%) reviews were returned
Reviewers miss stuff!
15 Accept: 17.3% major, 11.8% minor 117 Reject: 39.1 % major, 25.2% minor 67 Revision: 29.6% major, 22.0% minor 68% did not realize that the conclusions were not supported by the results.
“Peer reviewers in this study failed to identify two thirds of the major errors…” (or is that really “failed to report…”?) Baxt WG et al.
Ann Emerg Med
1998 Sep;32(3 Pt 1):310-7.
It’s not just EM!
Sent a “positive” and a “no-difference” version of the same RCT manuscript to 238 reviewers at 2 orthopedics journals.
Papers differed only in the direction of the finding of the principal study end point.
Identical “Methods” sections 210 returned reviews. Emerson GB et al.
Arch Int Med
2010;170(21):1934-9
Positive outcome bias
More likely to recommend the positive version for publication 97.3% vs 80.0%, P < 0.001
Detected more errors in the no difference version 0.85 vs 0.41, P < 0.001
Awarded higher methods scores to the positive manuscript 8.24 vs 7.53, P = 0.005
Who are the good reviewers?
306
Annals
reviewers surveyed for training and experience Correlated to scores of 2856 reviews “Multivariable analysis revealed that most variables, including academic rank, formal training in critical appraisal or statistics, or status as principal investigator of a grant, failed to predict performance of higher quality reviews.”
Who are the good reviewers?
“The only significant predictors of quality were working in a university-operated hospital versus other teaching environment and relative youth (under ten years of experience after finishing training).” Callaham ML, Tercier J. The relationship of previous training and experience of journal peer reviewers to subsequent review quality.
PLoS Med
2007;4:e40.
Peer review performance drops off just a bit with age
15K reviews by 1500 reviewers in 14 yrs -0.8% change in scores per year “This could be due to deteriorating performance (caused by either cognitive changes or competing priorities ) or, to a partial degree, escalating expectations… Callaham & McCulloch. Longitudinal trends in the performance of scientific peer reviewers.
Ann Emerg Med
2011;57:141-8
Cochrane Analysis: 28 studies
“We could not identify any methodologically convincing studies assessing the core effects of peer review.” “At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research.” Jefferson T, et al. Cochrane Database of Systematic Reviews. (2):MR000016, 2007.
One Alternative: “Scrap Peer Review”
Peer review “is slow, expensive, largely a lottery, poor at detecting errors and fraud, anti-innovatory, biased, and prone to abuse.” “The time has come to move from a world of ‘filter then publish’ to one of ‘publish then filter’—and it’s happening.” Richard Smith blogs.bmj.com/bmj/2010/03/22/richard-smith-scrap peer-review-and-beware-of-%E2%80%9Ctop journals%E2%80%9D/
New York Times, 23 August 2010
“Scholars Test Web Alternative to Peer Review” Test by
Shakespeare Quarterly
Posted four essays to web site – not yet accepted for publication Recruited “core experts” to review Opened review process to up to anybody 41 “reviewers” with 350 comments, plus replies from the authors Essays revised Accepted by editors www.nytimes.com/2010/08/24/art s/24peer.html?_r=1&emc=eta1
Open Peer Review: Supporters
The first question that Alan Galey, a junior faculty member at the University of Toronto, asked when deciding to participate in The Shakespeare Quarterly’s experiment was whether his essay would ultimately count toward tenure . “I went straight to the dean with it,” Mr. Galey said. (It would.) Although initially cautious, Mr. Galey said he is now “entirely won over by the open peer review model.” The comments were more extensive and more insightful, he said, than he otherwise would have received on his essay.
NYTimes article
Open Peer Review: Skeptics
“Knowledge is not democratic,” said Michèle Lamont, a Harvard sociologist who analyzes peer review in her 2009 book, “How Professors Think: Inside the Curious World of Academic Judgment.” Evaluating originality and intellectual significance, she said, can be done only by those who are expert in a field.
NYTimes article
Open Peer Review: Skeptics
“…faculty have generally been reluctant to participate in open access schemes because they do not see the advantages: it takes too much effort and they have so much at stake in the ways that the current systems of peer review and publication preserve trust in the authenticity of academic work and reliably allocate credit for that work .” Walters D.
JEP
2008:11(1)
Electronic Publishing
Using electronic media tools to publish in new and different ways Some of these can be considered post publication peer review Reading articles versus citing/searching – variation between e-article and hard copy use.
A lot of this is already happening…
Podcasts and other unidirectional exchanges
Allows expert(s) to summarize message in a user-friendly, conversational tone (? impact) Some pro/con possible (interviewer, expert, author) Trendy, growing; easy to do “bare bones” but polished is more resource intensive ? Real use
Moderated Discussion Boards
Live Journal Clubs
Internet interaction: Journal club participants (e.g. residents, faculty, medical students) Authors Statisticians Editors
Wiki Pages
eScholarship: University of California
“eScholarship provides a suite of open access, scholarly publishing services and research tools that enable departments, research units, publishing programs, and individual scholars associated with the University of California to have
direct control over the creation and dissemination of the full range of their scholarship
.” www.escholarship.org
Post-Publication Peer Review
WebMedCentral Guaranteed publication within 48 hrs No pre-publication peer review “Peer review takes place post publication in an open and transparent manner” No cost (yet) to authors or readers Author retains copyright, and can publish elsewhere
EM article on WebMedCentral
“Integrated clinical decision support in emergency medicine: transforming the electronic health record in order to reduce risk and improve medical decision making” Submitted 03 Sep 2010, 19:51:34 GMT Published 04 Sep 2010, 01:01:23 GMT As of 24 March 2011: 451 views 0 comments 0 reviews
WebMedCentral.com
“We are a biomedical publishing house, owned and managed by a group of medical and management professionals. We are an independent group with no links to the pharmaceutical, traditional publishing or any other industry. “
Liquid Publications
“The current approach encourages authors to write many (possibly incremental) papers to get more “tokens of credit”, generating often unnecessary dissemination overhead for themselves and for the community of reviewers. Furthermore, it does not encourage or support reuse and evolution of publications: whenever a (possibly small) progress is made on a certain subject, a new paper is written, reviewed, and published, often after several months. The situation is analogous if not worse for textbooks.” www.liquidpub.org
Liquid Journals
Dynamic, not static Encourage early posting of findings Link material from disparate sources Harness the filtering power of the entire community, not just a couple of “experts” E-publishing 2.0?
Or maybe it’s really 3.0?
“Interesting-ness”
Judge the quality of papers by how interesting they are to the audience “Post-posting metrics” (liquidpub.org) Citations, downloads Number of comments entered ○ “reputation” of those commenting Bookmarks Incorporation into a reader’s library
Rejecta Mathematica
Only accepts papers rejected from peer-reviewed journals Author submits, with an open letter about the value of the paper, the rejection, and changes made as a result Editors select based on how interesting the paper would be to researchers No “peer-review” math.rejecta.org