The science of peer review

David C. Cone, MD Associate Professor of Emergency Medicine Yale University School of Medicine Editor-in-Chief

Academic Emergency Medicine

Donald M. Yealy, MD Professor and Chair of Emergency Medicine University of Pittsburgh Deputy Editor

Annals of Emergency Medicine

Disclosures

 D. Cone’s department receives a stipend for his services as Editor-in Chief of

Academic Emergency Medicine

 D. Yealy receives a stipend for

Annals of Emergency Medicine

service.

 Neither has any financial interests in peer review  No commercial interests or off-label uses to declare

Outline

 1. What do we know about the science of the traditional peer review system?

 2. What changes are being proposed?

 3. How might these affect our careers?

Two facets of engaging the broader scientific community

 “Front end” = peer review  To determine what is “worth” publishing in the literature  To optimize the message being presented  “Back end” = post-publication discourse   To allow others to comment and discuss after a paper makes it into the literature Occasionally, correct (“erratum”) or retract part or whole publication

The traditional peer review model

 A decision editor determines if he/she needs more input to assess a paper. If so:  The editor recruits a small number (2-5) of additional “expert” reviewers to aid  The reviewers provide comments  The editor uses collective wisdom to determine the disposition of the paper

The traditional post publication discourse model

 Letters to the editor, with replies  Most journals limit to one “round” of letters and author replies  Local journal clubs  Discussion in future papers on the topic  Retraction, erratum, or other post publication piece if malfeasance, error, or another important concern arises that could alter the paper’s impact/message.

What do we know about peer review?

 Remarkably little!

 Not much is encouraging.

 Much of the work on the science of peer review has been done by Mike Callaham, Editor-in-Chief of

Annals of Emergency Medicine

Mike Callaham’s evil twin?

 Grunting for worms: seismic vibrations cause Diplocardia earthworms to emerge from the soil.

 Mitra O, Callaham M , Smith ML, Yack JE. Biol Lett. 2009 Feb 23;5(1):16-9.

We can’t train reviewers!

 “…formal, 4-hour, highly interactive workshop on peer review”  Scores on reviews for next 2 yrs:  Control: 0.11 (95%CI -0.25 to 0.48)  Attendees: 0.10 (95%CI -0.20 to 0.39)  Active recruitment: ○ “Efforts to aggressively recruit average reviewers to a second workshop were time consuming, had low success rates, and showed a similar lack of effect on ratings” Callaham ML, Schriger DL.

Ann Emerg Med

2002;40:323-8.

We can’t improve the reviewers we already have!

 Written feedback from editor vs usual (receiving other reviews & decision letter)  Mean individual reviewer rating change:  Study 1: lower-quality reviewers ○ Control: 0.16 (95%CI -0.26 to 0.58) ○ Intervention: -0.13 (-0.49 to 0.23)  Study 2: average reviewers, more feedback ○ Control: 0.12 (-0.20 to 0.26) ○ Intervention: 0.06 (-0.19 to 0.31).

Callaham ML , Knopp RK , Gallagher EJ .

JAMA

2002;287(21):2781

Peer reviewers miss stuff!

 Fake manuscript describing a double blind, placebo-controlled study showing that IV propranolol reduced the pain of acute migraine headache  10 major and 13 minor errors inserted  Sent to all 262

Annals

reviewers  203 (78%) reviews were returned

Reviewers miss stuff!

 15 Accept: 17.3% major, 11.8% minor  117 Reject: 39.1 % major, 25.2% minor  67 Revision: 29.6% major, 22.0% minor  68% did not realize that the conclusions were not supported by the results.

 “Peer reviewers in this study failed to identify two thirds of the major errors…” (or is that really “failed to report…”?) Baxt WG et al.

Ann Emerg Med

1998 Sep;32(3 Pt 1):310-7.

It’s not just EM!

 Sent a “positive” and a “no-difference” version of the same RCT manuscript to 238 reviewers at 2 orthopedics journals.

 Papers differed only in the direction of the finding of the principal study end point.

 Identical “Methods” sections  210 returned reviews. Emerson GB et al.

Arch Int Med

2010;170(21):1934-9

Positive outcome bias

 More likely to recommend the positive version for publication  97.3% vs 80.0%, P < 0.001

 Detected more errors in the no difference version  0.85 vs 0.41, P < 0.001

 Awarded higher methods scores to the positive manuscript  8.24 vs 7.53, P = 0.005

Who are the good reviewers?

 306

Annals

reviewers surveyed for training and experience  Correlated to scores of 2856 reviews  “Multivariable analysis revealed that most variables, including academic rank, formal training in critical appraisal or statistics, or status as principal investigator of a grant, failed to predict performance of higher quality reviews.”

Who are the good reviewers?

 “The only significant predictors of quality were working in a university-operated hospital versus other teaching environment and relative youth (under ten years of experience after finishing training).” Callaham ML, Tercier J. The relationship of previous training and experience of journal peer reviewers to subsequent review quality.

PLoS Med

2007;4:e40.

Peer review performance drops off just a bit with age

 15K reviews by 1500 reviewers in 14 yrs  -0.8% change in scores per year  “This could be due to deteriorating performance (caused by either cognitive changes or competing priorities ) or, to a partial degree, escalating expectations… Callaham & McCulloch. Longitudinal trends in the performance of scientific peer reviewers.

Ann Emerg Med

2011;57:141-8

Cochrane Analysis: 28 studies

  “We could not identify any methodologically convincing studies assessing the core effects of peer review.” “At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research.” Jefferson T, et al. Cochrane Database of Systematic Reviews. (2):MR000016, 2007.

One Alternative: “Scrap Peer Review”

 Peer review “is slow, expensive, largely a lottery, poor at detecting errors and fraud, anti-innovatory, biased, and prone to abuse.”  “The time has come to move from a world of ‘filter then publish’ to one of ‘publish then filter’—and it’s happening.”  Richard Smith  blogs.bmj.com/bmj/2010/03/22/richard-smith-scrap peer-review-and-beware-of-%E2%80%9Ctop journals%E2%80%9D/

New York Times, 23 August 2010

 “Scholars Test Web Alternative to Peer Review”  Test by

Shakespeare Quarterly

 Posted four essays to web site – not yet accepted for publication  Recruited “core experts” to review   Opened review process to up to anybody 41 “reviewers” with 350 comments, plus replies from the authors   Essays revised Accepted by editors www.nytimes.com/2010/08/24/art s/24peer.html?_r=1&emc=eta1

Open Peer Review: Supporters

 The first question that Alan Galey, a junior faculty member at the University of Toronto, asked when deciding to participate in The Shakespeare Quarterly’s experiment was whether his essay would ultimately count toward tenure . “I went straight to the dean with it,” Mr. Galey said. (It would.) Although initially cautious, Mr. Galey said he is now “entirely won over by the open peer review model.” The comments were more extensive and more insightful, he said, than he otherwise would have received on his essay.

 NYTimes article

Open Peer Review: Skeptics

 “Knowledge is not democratic,” said Michèle Lamont, a Harvard sociologist who analyzes peer review in her 2009 book, “How Professors Think: Inside the Curious World of Academic Judgment.” Evaluating originality and intellectual significance, she said, can be done only by those who are expert in a field.

 NYTimes article

Open Peer Review: Skeptics

 “…faculty have generally been reluctant to participate in open access schemes because they do not see the advantages: it takes too much effort and they have so much at stake in the ways that the current systems of peer review and publication preserve trust in the authenticity of academic work and reliably allocate credit for that work .”  Walters D.

JEP

2008:11(1)

Electronic Publishing

 Using electronic media tools to publish in new and different ways  Some of these can be considered post publication peer review  Reading articles versus citing/searching – variation between e-article and hard copy use.

 A lot of this is already happening…

Podcasts and other unidirectional exchanges

 Allows expert(s) to summarize message in a user-friendly, conversational tone (? impact)  Some pro/con possible (interviewer, expert, author)  Trendy, growing; easy to do “bare bones” but polished is more resource intensive  ? Real use

Moderated Discussion Boards

Live Journal Clubs

 Internet interaction:  Journal club participants (e.g. residents, faculty, medical students)  Authors  Statisticians  Editors

Wiki Pages

eScholarship: University of California

 “eScholarship provides a suite of open access, scholarly publishing services and research tools that enable departments, research units, publishing programs, and individual scholars associated with the University of California to have

direct control over the creation and dissemination of the full range of their scholarship

.” www.escholarship.org

Post-Publication Peer Review

 WebMedCentral  Guaranteed publication within 48 hrs  No pre-publication peer review  “Peer review takes place post publication in an open and transparent manner”  No cost (yet) to authors or readers  Author retains copyright, and can publish elsewhere

EM article on WebMedCentral

 “Integrated clinical decision support in emergency medicine: transforming the electronic health record in order to reduce risk and improve medical decision making”  Submitted 03 Sep 2010, 19:51:34 GMT  Published 04 Sep 2010, 01:01:23 GMT  As of 24 March 2011:  451 views  0 comments  0 reviews

WebMedCentral.com

 “We are a biomedical publishing house, owned and managed by a group of medical and management professionals. We are an independent group with no links to the pharmaceutical, traditional publishing or any other industry. “

Liquid Publications

 “The current approach encourages authors to write many (possibly incremental) papers to get more “tokens of credit”, generating often unnecessary dissemination overhead for themselves and for the community of reviewers. Furthermore, it does not encourage or support reuse and evolution of publications: whenever a (possibly small) progress is made on a certain subject, a new paper is written, reviewed, and published, often after several months. The situation is analogous if not worse for textbooks.” www.liquidpub.org

Liquid Journals

 Dynamic, not static  Encourage early posting of findings  Link material from disparate sources  Harness the filtering power of the entire community, not just a couple of “experts”  E-publishing 2.0?

 Or maybe it’s really 3.0?

“Interesting-ness”

  Judge the quality of papers by how interesting they are to the audience “Post-posting metrics” (liquidpub.org)  Citations, downloads  Number of comments entered ○ “reputation” of those commenting   Bookmarks Incorporation into a reader’s library

Rejecta Mathematica

 Only accepts papers rejected from peer-reviewed journals  Author submits, with an open letter about the value of the paper, the rejection, and changes made as a result  Editors select based on how interesting the paper would be to researchers  No “peer-review” math.rejecta.org