Transcript AWHILES Study Day 13 May 2011
AWHILES Study Day 13 May 2011
Julie Browne External Relations Manager, Wales Deanery, PGMDE University of Cardiff
“Published research: how do you know if it’s any good?”
…or
“The role of the editorial office in safeguarding the integrity and academic rigour of the scholarly archive, with particular reference to the medical education literature”
What I’d like to cover today • The role of journals in safeguarding quality • Medical education journals • Peer review – what is it?
• Peer review – how is it done?
• Peer review – why do it?
• Publication ethics
Meet Jamie …
“How do you know if a published paper is likely to be reliable?”
Jamie again …
“What are the characteristics of a high quality journal?”
Quality in journals
• Metrics • People • Content • Systems
Leg 1: Metrics • Citations (including self-citation) • Indexes • Rejection rates • Speed of peer review/acceptance to publication • Impact factor/immediacy
ISI Web of Knowledge: Journal Citation Reports
2.5
3 Trend spotting: Impact factor 2 1.5
1 0.5
0 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
Impact Factors in Medical Education Journals by Year
2007 2008 2009 Acad Med AHSE* from 2003 Med Educ Med Teacher Teach Learn Med* from 1998 J Cont Educ Health* from 2007
Leg 2: People
– Editorial board, editor, editorial staff – Publisher – Learned society – Peer reviewers – Authors
Leg 3: Content
• High quality papers (goes without saying – we’ll discuss this further in a minute!) • Letters and commentaries sections (post hoc peer review) • Editorials and guidance on scope and mission • Retractions, corrections, errata • Participation in journalology research
Leg 4: Editorial Systems • Transparent processes, clearly announced – how we do things, what you can expect from us • Published policies: e.g. ethics, peer review, handling of complaints, handling of editors’ papers • Membership of COPE or other editors’ group • An ethics committee • A statement regarding editorial independence
(e.g. “All content in
Academic Medicine
reflects the views of the authors and does not reflect the official policy of the AAMC unless clearly specified.” “The BMA grants editorial freedom to the editor of the
BMJ
. The views expressed in the journal are those of the authors and may not necessarily comply with BMA policy.”)
But, above all… A rigorous, effective and fair peer review system
“The purpose of peer review is to keep egg off authors’ faces” Drummond Rennie (JAMA)
The review process
Completion of research Writing and preparation Submission Internal review External review Rejection Decision Acceptance Publication Revision Resubmission Re-review Rejection
Common reasons for rejection
• Did not address an important scientific issue • Research question not clearly stated • Has been done before • Results are unsurprising/does not add enough new information to be interesting • Inappropriate study design/methodological weaknesses/methods not described clearly enough • Sample size too small/response rate too low • Incorrect or unjustified conclusions • Statistical analysis incorrect or inappropriate • Wrong journal/readership; would not interest our readers • Too badly written to understand
But sometimes something murky is lurking under the surface….
Murky things: misconduct
• Fabrication: invention of data or cases • Falsification: wilful distortion of data including failure to report missing or negative data • Plagiarism: copying of ideas, data or words without attribution • Failing to get consent from an ethics committee for research and/or conducting research in humans without informed consent • Misreporting authorship contributions • Not revealing conflicts of interest
Peer review to the rescue?!
More from Jamie …
“How widespread is research and publication misconduct?” “How much does the peer review system catch?”
Things inevitably slip through …
Some big cases (1)
Jon Sudbø In January 2006 it was revealed that a paper by Norwegian researcher Jon Sudbø published in
The Lancet
in October 2005 was based upon fraudulent patient data. The article had suggested that some types of NSAIDs diminish the risk of oral cancer in smokers. However, it turned out that he had made up all the patient data; a quarter of all the subjects in
The Lancet
study had the same birthday!
Some big cases (2)
Andrew Wakefield
1996
A paper by Wakefield and colleagues published in
The Lancet
suggested that there was a connection between the MMR vaccine and autism and colitis.
1998
No subsequent research was able to reproduce his findings. The paper was retracted.
2001
Immunisation rates dropped from 92% to 73%, almost certainly leading to a rise in mortality.
2010
Wakefield was struck off by GMC for over 30 charges of misconduct including four counts of dishonesty. Wakefield continues to defend his reputation.
Some big cases (3)
Malcolm Pearce 1994: Obstetrician Malcolm Pearce claimed in
BJOG
to have successfully relocated an ectopic pregnancy, with a healthy subsequent birth. A junior doctor at St George’s blew the whistle. The patient had never existed.
“OK, so these are big cases.
But does the little stuff really matter that much …?”
The journalology literature
• • • • •
56% of newly appointed consultants reported witnessing research fraud
Geggie (2001)
J Med Ethics
27:344 –6
On average 2% of scientists admit to have falsified research at least once and up to 34% admit other questionable research practices
Fanelli (2009). PLoS ONE 4(5): e5738
31% of submissions to a Chinese journal were plagiarized.
Zhang. (2010) Nature 467:153
30% of submitted manuscripts included plagiarism from a previous publication of the senior author. 16% of submitted manuscripts included plagiarism from another investigator’s work and/or website.
McFarlin et al (2010) IJ Exercise Sci (3):3
9.5% of authors in the BMJ were ‘guest authors’: i.e. they did not make any significant contribution to the paper
Bates et al (2004) JAMA. 292(1):86-8.
How dangerous is it?
• erodes public trust in the scientific community • is extremely costly to investigate and police • may create a flawed knowledge base (systematic reviews) • may negatively influence funders and politicians • may divert funding and effort away from reputable research • may influence the political process, esp in controversial areas e.g. stem cell research (Hwang Woo Suk) • can create false hope among patients
Corrections, Retractions and "Expressions of Concern" “The retraction or expression of concern, so labelled, should appear on a numbered page in a prominent section of the print journal as well as in the online version, be listed in the Table of Contents page, and include in its heading the title of the original article. The text of the retraction should explain why the article is being retracted and include a complete citation reference to that article.”
Uniform Guidelines for manuscripts submitted to biomedical journals
(The Vancouver Guidelines) http://www.icmje.org
But retracted papers live on …
“Examination of one
Nature
paper by former Bell Labs physicist Jan Hendrik Schön, published in 2000 and retracted in 2003, revealed that it's been noted in research papers 17 times since, although the drop-off after retraction was steep: prior to being pulled, the paper was cited 153 times.” Unger K. Couzin J. Even Retracted Papers Endure.
Science
2006;312(5770):40-41 “The difference between citation levels of flawed originals and corrected republications does not approach statistical significance until eight to twelve years post republication.” Peterson GM. The effectiveness of the practice of correction and republication in the biomedical literature.
J Med Libr Assoc.
2010; 98(2): 135 –139.
http://www.publicationethics.org/
Further information
Brice, J. and J. Bligh.
Author misconduct: not just the editors' responsibility.
Medical Education, 2005.
39
(1): p. 83-89 Smith, R.
The trouble with medical journals.
London: RSM Books 2006 Hames, I.
Peer review and manuscript management in scientific journals: Guidelines for good practice.
2007, London: John Wiley and Sons.
Wager, E., F. Godlee, and T. Jefferson.
How to survive peer review.
2002, London: BMJ Books.
Bligh, J. and J. Brice,
New Year reflections on professionalism.
Education, 2005.
39
(1): p. 2-3.
Medical Organisations: ICMJE: http://www.icmje.org/ World Association of Medical Editors: http://www.wame.org/ Committee on Publication Ethics: http://www.publicationethics.org/