Transcript Promoting Research Integrity Historical Background & Current Trends Nicholas H. Steneck, PhD Subtitle …from a Misconduct- to an Integrity-centered Universe Workshop on Best Practices for.

Promoting Research Integrity
Historical Background & Current Trends
Nicholas H. Steneck, PhD
Subtitle
…from a Misconduct- to an Integrity-centered Universe
Workshop on Best Practices for Ensuring Scientific Integrity & Preventing
Misconduct
22-23 February, 2007
Mita Conference Hall, Tokyo, Japan
Research misconduct is not new
 Galileo fabricated & falsified data
 Piltdown hox and other scandals
 Some things do not change over time
Major changes between 1600 and 2000
Science ca. 1600
Science today
•Privately supported
•Publicly supported
•Limited public benefit
•Essential to technological
growth & public welfare
http://en.wikipedia.org/wiki/Image:Galileo.arp.300pix.jp
http://www.talkorigins.org/faqs/piltdown/pskull.gif
g
Historical Perspective
By 1980, when first cases of misconduct emerge, science
(research) is a public activity
Over time, scientists (researchers) became public servants!
2
Major growth since 1950
Financial support
 1930s, under 1% GDP
 1950 ca. 1% GDP
 Currently 2-3%
http://www.nsf.gov/statistics/databrf/sdb99357.pdf
3
 1 of 100 public dollars in
US spent on health research
 Research is a public
activity
Science linked with unpopular events and problems:
 Cold War links science with “military-industrial complex
 Clock, Bulletin of the Atomic Scientist (1947)
 War in Vietnam (1960s)
 Environmental impact of nuclear power
 Concerns about human & animal experimentation
 US, Tuskegee experiments
 Declaration of Helsinki (1964)
 Global energy crisis (1970s)
 New worldviews compete with scientific worldview
By 1980 (first major US misconduct cases) public is taking
a closer look at the way research is conducted
4
http://www.thebulletin.org/minutes-to-midnight/
Some doubts along with the growth
Response to research misconduct, 1980 ff.
Events have been driven by major cases & media
 Story breaks in the news
 Local institution responds
 Pressure for an official/government response
Official response
 Gather information ~ committees, hearings, reports
 Try to resolve the immediate problem ~ the major cases
 Develop policies and procedures to avoid similar problems in the
future
Timing
5
 US begins in early 1980s, policy development still in process
 Europe, early 1990s, policy development in process
 Asia, late 1990s, policy development in process
The misconduct-centered universe
First priority ~ major cases
 Define misconduct
 Assign authority
 Develop procedures for investigation
Definitions focus on deliberate
reporting false data & information
 Careful to not confuse with scientific disagreement
 Not the same as waste and sloppiness
Policies protect researchers from improper charges
Working assumption: pursuing individual cases of misconduct
is the best way to protect the integrity of publicly supported
research
6
Misconduct-based universe rested on 5 assumptions
 Serious misconduct in research is rare
 Self-regulation keeps improper behavior in check
 Research misconduct is difficult to detect
 Research misconduct cannot be prevented
 Apart from misconduct, standards for integrity in research
are high
Assumptions were based on common perceptions, not
empirical evidence
All five can be questioned!
7
 Research misconduct is rare?
Martinson, Nature (June 2005)
 Goal: factors that influence research behavior
 Method:
 Developed peer-based list of major offenses
 Survey to 6,000+ researchers (3,000+ response)
 Major question: “have you done … in last three years?”
Results
 Major offenses, ca. 0.3%
 Questionable Research Practices (QRP) ca. 5-15% or higher
8
Data from other recent studies
JM Ranstam, Control Clin Trials (2000)
 Survey, 442 biostatisticians, 37% response
 51% knew about fraud in medical research
 26% involved FF
 31% directly involved in projects with misconduct
 Estimates of rate, .69% –> .80% (.25% standard)
Geggie, J Med Ethics (2001)
 Survey, 305 new medical consultants, 64% response




9
55.7% observed misconduct (FF lower)
5.7% committed misconduct in the past
18% would commit in future
17% had research ethics training
Studies continued
Gardner, Contemporary Clinical Trials (2005)
 Authors pharmaceutical clinical trials (64% response)
 1% reported target article misrepresented the research
 5% reported fabrication in a study they had participated in over the last
10 years
 17% knew personally of fabrication in a study over the last 10 years
Rossner, Journal of Cell Biology
 11 in 1,100 papers had serious improper digital image manipulation
10
Rough approximation:
 Evidence ~ 1/1,000+
 Assume ~ 1/10,000
Cases predicted




US ~ 1,500
EU ~ 1,000
Japan ~ 600
Other OECD ~ 400
Cases seen
 US ~ 20/year
 EU ~ 10/year
Lesson #1 ~ policy makers have always under-estimated the
amount of misconduct in research
11
Science & Engineering Indicators 2006, Figure 3-36
Realistic estimates:
Other assumptions about misconduct?
 Self-regulation keeps improper behavior in check?
 Researchers do not report suspected misconduct (20-40%)
 Journals often do not report misconduct
 Research misconduct is difficult to detect?
 Hwang could not have completed work in the time reported
 Sudbø, trial not started, patients did not exists, data repeated
 Research misconduct cannot be prevented?
 Schön’s co-author/mentor did not check experiments or data
 Pohlmann’s MD co-author did not oversee clinical results
Lesson #2: Policy makers did not understand the
strengths & weaknesses of self-regulation in research
 If they understood, did not honestly and accurately report
12
 Integrity in research otherwise high?
Martinson study, self-reported misbehaviors:
15.5% Changing the design, methodology or results of a study in response to
pressure from a funding source
12.5% Overlooking others' use of flawed data or questionable interpretation
7.6% Circumventing certain minor aspects of human-subject requirements
6.0% Failing to present data that contradict one’s own previous research
1.7% Unauthorized use of confidential information
1.4% Using another’s ideas without obtaining permission or giving due credit
1.4% Relationships with students, research subjects or clients that may be
interpreted as questionable
0.3% Not properly disclosing involvement in firms whose products are based on
one‘s own research
0.3% Ignoring major aspects of human-subject requirements
0.3% Falsifying or ‘cooking’ research data
13
Al-Marsouki, Contemp Clin Trials 26(2005)
Practices felt likely to occur and adversely impact research
83%
80%
76%
75%
68%
68%
64%
64%
60%
60%
59%
56%
54%
14
Over-interpretation of “significant" findings in small trials
Selective reporting based on p-values
Selective reporting of outcomes in the abstract
Subgroup analyses done without interaction tests
Negative or detrimental studies not published
Putting undue stress on results from subgroup analysis
Inappropriate subgroup analyses
Selective reporting of (i) subgroups (ii) outcomes (iii) time points
Selective reporting of positive results/omission of adverse events data
Failure to report results or long delay in reporting
Post-hoc analysis not admitted
Giving incomplete information about analyses with non significant results
Analysis conducted by the sponsor of the trial
How do researchers behave?
Lesson #3. Significant gap between ideal (high standards)
and actual standards for integrity in research
Research behavior
Misconduct ~ 0.1% <–> 1%
15
QRP ~ 10% <–> 50%
High standards ??
How should researchers behave?
Integrity-centered
Misconduct-centered
16
Slow change to an integrity-centered universe
More emphasis on prevention and improving integrity
First major change ~ improve education/training
US, efforts to require training/education




1989 Institute of Medicine Report called for training
1990/92, National Institutes of Health required for trainees
1997, National Science Foundation required for trainees
2000, Public Health Service proposed general requirement
 Strong objections raised by research community
 Requirement has been suspended
Global initiatives
 Finland, national requirement for graduate students
 Elsewhere, growing number of courses and resources
17
Training covers more than misconduct
Areas developed over time:
 1994 Training Grant Requirement





Conflict of Interest
Responsible authorship
Policies for handling misconduct
Data management
Human & animal subjects
 2000 HHS RCR requirement
 9 areas
Other areas could be added:
Nine Core Areas
1. Data acquisition,
management, sharing, and
ownership
2. Mentor/trainee responsibilities
3. Publication practices and
responsible authorship
4. Peer review
5. Collaborative science
6. Human subjects
7. Research involving animals
8. Research misconduct
9. Conflict of interest and
commitment
 Grant and financial management
 Workplace rules
 Laboratory safety & special rules (biological materials…)
18
Current status of “RCR” training
 Abundant resources for teaching
 Textbooks
 Web pages
 Train-the-trainer programs
 No standards for content or approach
 Coverage depends on the instructor
 Training not integrated with research
 Minimal testing or follow-up
 Little assessment of effectiveness
 Major challenge:
 Research is global; research teams and laboratories are international
 RCR training is local, inconsistent, and for the most part inadequate
19
Second change ~ Better understanding of behavior
Misconduct-based universe ~ cause of misconduct?
 Too few cases/too much variation to draw “scientific” conclusions
 Suggested some areas for further attention:
 Quality of mentoring, supervision, peer review
Integrity-centered universe ~ many issues to study
 Social processes
 Authorship - who is listed and why?
 Peer review - weaknesses and how to correct?
 Data -how do researchers collect and record data?
 Institutional role and influences
 Policies - how institutions develop and promote policies
 Good management - how administrators and committees work
 Conflict of interest - how institutions manage their own conflicts
Much better prepared to implement effective policies
20
Four crucial challenges
 Establish clear objectives
 “Misconduct” policies can have many objectives:
A.
B.
C.
D.
Establish procedures for responding to misconduct in research
Detect and eliminate/correct fraudulent information
Protect public from consequences of flawed research findings
Maintain/restore public confidence in research
 US Policy (OSTP, 2000) has three general objectives
 Protect “reliability of the research record”
 Maintain public “confidence in the research record”
 Achieve policy unity
 Reason for objectives
 Clear understand of goals to be achieved
 Standard for measuring success and improving policy
21
 Fair, effective procedures for responding
A. Establishing national policies is a starting point
 Comprehensive definitions that cover all serious misbehavior
 Assure reporting and accountability
 Protect informants and insulate from bias and conflicts of interest
 Global harmonization and communication are next step
 Research is no longer local or national
 Laws of nature and scientific methods are not local
 Standards for reporting, investigating, and judging misconduct should
be global

Some accommodation for differences in law and government as long as
good research practices are not compromised
 The globalization of research requires the globalization of
policies and best research practices
22
 Take steps to make policies more effective
Detect and eliminate/correct fraudulent information??
 Investigating 1 of every 100 cases has little impact on the reliability of
the research record
 Some FFP has is trivial
 Current policies have minimum impact on the research record
Steps that would eliminate/correct current system





Improved education, emphasizing professional responsibility
Clearer rules for data management, mentoring, and peer review
Random audits of publications and supporting data
Institutional climate surveys to assess reliability of self-regulation
Extend responsibility for reporting to journals
Investigating major cases is essential but has little impact on
the overall reliability of the research record
23
 Take a serious look at QRPs
 Viewed from a public perspective, questionable practices are
more significant than research misconduct
 Occur more frequently ~ 10x or more
 Have serious impacts



Poor literature reviews have led to harm of research subjects
Bias reporting/duplicate publication impact health-care decisions
Ineffective public decisions due to improper statistical analysis
 How would you explain the following to the public?
 25% of researchers reported recording results in loose-leave notebooks
 40% of abstracts misrepresent findings reported in the article
 Funding makes researcher 3-5 times more likely to report results
favorable to the funding source
 Protecting the integrity of the research record requires more
than simply responding to cases of misconduct
24
25