Transcript Document

National Workshop on Human Subjects,
July 28-30, 2003, University of Georgia:
“Issues in Social & Behavioral Research”
http://www.ovpr.uga.edu/hso/
John Mueller ([email protected])
Division of Applied Psychology
University of Calgary
“WHY IS COMPLIANCE MORE
IMPORTANT THAN EFFECTIVENESS?”
http://mueller.educ.ucalgary.ca/Georgia2003
7/17/2015
NWHS-Georgia-2003
1
Acknowledgements
Thanks to my colleagues at the Society for
Academic Freedom and Scholarship (SAFS,
http://www.safs.ca) who encourage my interests
in exploring the ways in which research ethics
reviews and grant agencies constrain scholarly
inquiry:
 John Furedy (U. Toronto)
 Clive Seligman (U. Western Ontario)
 Steve Lupker (U. Western Ontario)
7/17/2015
NWHS-Georgia-2003
2
Background





35+ years, human experimental psychology
Traditional academic, retro even (Tolman: “have fun”)
Research is about decoding Mother Nature, not dictating to her
Chair department research reviews, now “just” a researcher
"History of Psychology" instructor:
» How sciences advanced/evolved from philosophy
» Socio-political influences on inquiry:
 zeitgeist, paradigms, … politics, censors, etc.
What I describe may sound like a “worst case” scenario, but it is
disturbing how practices seem to converge on these seemingly
ineffective practices.
Individual viewpoints, not endorsed by my employer, etc.
( though the world would be a better place! )
7/17/2015
NWHS-Georgia-2003
3
It’s a Bureaucracy:
You will be Assimilated
From the “Charge of the Light Brigade”:
Their’s not to reason why,
Their’s but to do and die.
Well, what a choice!
But this implies a third option, that someone else can, or
must, reason why, so I will volunteer for that!
Just remember, IRB is just one letter removed from IRS,
another relentless bureaucracy we all know and love!
7/17/2015
NWHS-Georgia-2003
4
Compliance vs. Discernment
Two cultures ("solitudes"):
1. Bureaucrats comply, that's a culture of control
2.Researchers ask “why,” that's a culture of inquiry
Two key characteristics of bureaucracies:
1. Bureaucrats don't get fired for following the rules, but
they do for not following the rules.
2. Success in a bureaucracy is not about solving a problem,
instead success is about not being blamed for a problem.
Ethics reviews today have compliance as the end goal, not inquiry,
not balance. In fact, coexistence may not be workable -- our
way or the highway, first and second-class citizens, not really
"rational" differences (i.e., shortfall of information).
7/17/2015
NWHS-Georgia-2003
5
Communication Networks
In the technology innovation area (e.g., Rogers), interaction styles
in a business are thought to help or hinder diffusion of
innovation:
» Homophilius: communicate within same workplace role
(techies talk to techies, worker bees talk to workers,
marketers talk to purchasers, suits talk to suits)
» Heterophilius: communicate across workplace roles and/or
adopter category
Researchers talk to researchers, ethicists talk to ethicists, with
essentially no meaningful dialogue across roles even on campus,
much less nationally? And then there is the government, with
stone tablets …
7/17/2015
NWHS-Georgia-2003
6
The Good Old Days




(circa 1970)
Voluntary participation
Informed consent
Confidentiality, or anonymity
Not more than “Everyday Risk”
» “harm” beyond routine (not “zero”)
 Departmental review (expertise)
 Only federally-funded projects
» And only AFTER funding was approved (quaint, but efficient)
 Classroom belonged to instructor and students
 Mainly medical research (“safety” a true issue)
7/17/2015
NWHS-Georgia-2003
7
It’s Not Broken,
But Let’s Fix It Anyhow
Several parallel developments:
» Expanding range of control
» Redefining focus of reviews
» Medical research concerns become the prototype
for social sciences, directly or indirectly
7/17/2015
NWHS-Georgia-2003
8
The World Is Our Oyster
 ALL research, not just federally funded, including classroom
demonstrations and “interviewing mother.”
 Forms-R-Us™ -- create that paper trail
 Adversarial climate, in effect demonizing researchers as
untrustworthy and selfish
» “We make the rules, comply”
» “We don’t make the rules, comply”
» Napoleonic Code -- presumed guilty, prove yourself innocent.
 Isn't that “unethical”? It sure is offensive.
The IRB preoccupation with ink color, check boxes, and such seems just a variant of what
Zimbardo's prison guards did to establish authority. -- make the prisoners do petty and trivial
stuff to establish the pecking order? Consider THAT as a selection factor for defining future
faculty characteristics -- compliance. The meek shall inherit the universities?
7/17/2015
NWHS-Georgia-2003
9
Redefining Reviews
The Demotion Of Expertise:
» Move to campus-level reviews
» Lawyers, lay people, “other voices”
» Volunteers (we know they're not "normal")
Single-issue activists (the missionaries)
Professional “expert-in-all-things”
People with "control needs"
Career builders, self-anointed "ethicists,” etc.
Why not at least include true “risk experts,” such as insurers? Is it
because we no longer care about “risk” and “safety”, instead
something far more nebulous? Good drivers pay less for
insurance, bad doctors pay more, many such precedents.
7/17/2015
NWHS-Georgia-2003
10
If Not Experts, What?
 Non-experts can’t judge risk, safety (nor
methodology), so what is reviewed must be
changed.
 Thus a need to “indoctrinate” (not really
"educate") to know just what to review, but
» Why not Probability 101? Insurance 101?
» Why not Ethics 101? Philosophy 101? Logic 101?
» Why not Research 101? Statistics 101?
» Why not subject-area expertise?
» Why just “The regulations say …” ?
7/17/2015
NWHS-Georgia-2003
11
Prime Directive: Abandoned
Public Safety replaced by “ETHICS”:
» Undefined - idiosyncratic, ideological
Moralistic: my God is better than yours (my
politics too)
Not “legal,” but “right/wrong” -- "VALUES"
» Untestable - is that deliberate? Closed
system, theological discussions are futile.
Cassell & Jacobs (1987):
"My beliefs differ from yours, therefore you are unethical.”
Is the war in Iraq ethical? (The person sitting next to you disagrees …)
7/17/2015
NWHS-Georgia-2003
12
Reason: Abandoned
Zero tolerance -- can’t “play favorites”:
»
»
»
»
»
Proportionate review … no more
Expedited review … no more
Exempt research … no more
Everyday risk … no more
One person, one veto (everyone do it my way - or
else … if one person objects to a Christmas tree
it comes down -- zero tolerance disorder)
» No longer recruiting subjects, or participants, we
are trolling for VICTIMS
You have to have expertise to exercise good judgment,
“regulations” alone can’t do it for you.
7/17/2015
NWHS-Georgia-2003
13
Federal-local disconnect
 Federal authorities like to claim these distortions
have local origins, and that they have no authority.
 Local reviewers can claim this <distorted rule> is a
federal requirement .
 Institutions in a panic, overkill
 Unaccountable, at all levels
If this system worked, it would be a surprise;
it wouldn’t survive submission to a peer-review journal !
7/17/2015
NWHS-Georgia-2003
14
Fuzzifying Reviews
Is it “worthwhile” research (not is it safe)
» Is it socially desirable? Useful? (medical legacy?)
» Is it a “waste of time”? (“frivolous” vs. profound)
» Is it profitable to the institution?
At best, these are a recipe for “popular research,” -- “everybody
knows” (even experts!) has a long history of being dead wrong, as
in flat earth, heavier than air can’t fly, and so forth. Good luck
getting "breakthroughs" this way, nothing controversial. Basic
research seldom seems useful, in the short-term, so again we
sacrifice long-term benefits for short-term "gains".
At worst it is Lysenko and Galileo all over again, ideology-driven
research (whether implicitly or explicitly).
7/17/2015
NWHS-Georgia-2003
15
When you have a hammer?
 Canadian ethicist:
"In too many cases, researchers, members of behavioural
science REBs, and directors of research have blandly claimed
that the research under review poses no real risks to anyone
and that there is no need for quality assurance,
accreditation, or enforcement. In one case, a research
director told me that the absence of need was shown
because, in 10 years, his institution had not been sued by
any research participants. This lack of concern over the
adequacy of REB review in the social sciences and humanities
has disturbing implications for advancing interdisciplinary
health research."
 JM wonders:
Maybe it is not "lack of concern," maybe it's just good
reality contact, corroboration of a tempest in a teapot,
maybe there isn't an unethical researcher under every bed?
7/17/2015
NWHS-Georgia-2003
16
Compliance vs. Effectiveness
What evidence is there that any of these
changes have improved anything? None?
To answer we need to know:
»What problem was to be solved?
 Ill-defined (arguably no problem even)
 Not collecting baseline data on “incidents”
»How do we know it was solved?
 No data on incident change (pre-post)
Researchers expect effectiveness, bureaucrats do not
-- can’t publish null results.
7/17/2015
NWHS-Georgia-2003
17
We measure everything else
“Universities as businesses” at least try to measure:
» Key performance indicators
» Course evaluations
» Promotions and tenure
» Annual reports
We may often think these are bogus, but the efforts do acknowledge
that effectiveness would be nice, and that there should be some
accountability somewhere. But not for research ethics boards?
7/17/2015
NWHS-Georgia-2003
18
Pre-post is everywhere
 Diets, exercise programs, medicines
 Traffic changes (seat belt, cell phones, air bags)
 Business investments, stock portfolio
 Global warming, etc.
This is Research Methods 101, not to mention common sense!
Why is the research ethics industry apparently exempt?
Why is it allowed self-monitoring?
7/17/2015
NWHS-Georgia-2003
19
Is it working? Evidence?
 If reviews are working, show us some data -- not
feelings, not presumptions, not “surely,” not
“obviously.” (Could it be that ethicists don't care
about pre-post, they just "know" they're taking
us to a better place - missionaries?)
 If benefits are so obvious, then there must be
scads of good data ….
 Show me some -- show me any. REAL data.
Bertrand Russell:
“Assumptions have all the advantages of theft over honest toil."
7/17/2015
NWHS-Georgia-2003
20
Sample questions ….
 Was research in unfunded studies more hazardous than in federally
funded studies?
 Did research become less hazardous when the ethics review was
taken out of the department?
 Is classroom research safer now than before IRBs chimed in?
 Are things safer now that we review ALL proposals BEFORE they're
sent to the federal agency?
 What baseline will we use to document the benefits of accreditation?
 What baseline will we use to determine the effectiveness of
mandatory education/indoctrination courses?
Most of us would be embarrassed to be found doing something inefficient
and ineffective! Is it because these ethics reviews are done in a group,
behind closed doors, that we/they don't get embarrassed ?
7/17/2015
NWHS-Georgia-2003
21
Is it a waste? Whose cost?
If it is not working, then whose loss is it?
» Yours, mine, your university -- it’s local resources.
» Local resources diverted from other activities.
 Fewer classes, larger classes, …
 Resources diverted from screening research proposals
that actually are risky. Reviewers feel overworked because
they are wasting time treating everything as a major risk
(like airport security?). Need to do it right, not do it more.
If there never were real problems to begin with, then there will still be
none, in which case obfuscating the futility of these efforts makes
sense, of a sort. If there are real problems, and real benefits, why the
reluctance to really document them?
7/17/2015
NWHS-Georgia-2003
22
Problem Finding
 “Find a problem” (see how clever/moral I am) is not
the same as “Is there a problem” (risk assessment).
 “Problems” are not diagnostic of negative outcomes,
nor lives saved, nor a better world.
 “Revision requested” is not a “negative outcome
avoided” -- missed a checkbox, wrong color ink, etc.
» but these are ways to establish arbitrary authority, ala
Zimbardo et al.
The expectation of “zero risk” indicates a lack of reality contact, and we
should remind ourselves now and again of just where that famous road
paved with “good intentions" too often ends. Setting a goal of “zero
risk” assures job security in a way that makes tenure look lame!
7/17/2015
NWHS-Georgia-2003
23
Staying on-task
The review is not about:
» how "clever" is the reviewer (problem finding)
» how "sensitive" is the reviewer (who has the biggest halo)
» being more "religious" (ethical) than others
» hypothetical problems
» whether research “of this type” is safe in general
» a reviewer's preference in methodology, etc.
» whether “this type of research” is desirable, or profound, or
profitable or litigious (anything and everything is litigious!)
The mandate is whether this specific research will likely be UNUSUALLY
hazardous. Stay focused, stay on-task, have the ethics bull sessions
some other time. Need a strong leader or manager in the review
committee.
7/17/2015
NWHS-Georgia-2003
24
“Two aspirins” -- solves a lot
 Do not trivialize “harm” or “safety”
 Medical side-effects, some quite serious, are
readily accepted (by subjects/patients)
 Psychological side-effects are also a fact of life,
and they too “heal” quite nicely
 Pathologizing unlikely, minor, or rare outcomes is
worse than just silly, it is harmful
 You can’t eliminate “human error”
 You can't control "unforeseen" developments
 You can’t get “zero” risk
7/17/2015
NWHS-Georgia-2003
25
It’s not just a waste
 Waste would be a more or less neutral outcome,
regrettable but otherwise not uncommon or harmful.
 However, there are incidents that reveal a very
harmful side of the ethics industry:
» harassment of researchers.
And there is no sign of willingness to deal with this problem no accountability again. Perhaps this is a symptom that
researchers are tacitly viewed as just a "nuisance" to be
controlled? Or is it just that there is no regulation covering
it, can't do anything without a policy? (Topaz)
7/17/2015
NWHS-Georgia-2003
26
Censorship and more
 A senior scholar (E. Loftus) was harassed
through her University’s ethics board because
her research findings on false memories of
childhood abuse were “not worthwhile”
according to some (Tavris, 2002).
 Was the University sanctioned? Was the
ethics committee? Were you born yesterday?
7/17/2015
NWHS-Georgia-2003
27
Just a single case - not.
 Others have been harassed for going public
with findings that embarrass the authorities
(Pagliaro)
 Others have been harassed to the point of
suicide in personal vendettas (Sergent)
 Others have been harassed by the “blue-ink
syndrome” (Birnbaum)
"Only happening here" -- No, distortion is more general
"Couldn't happen here" -- Yeah, sure.
“It’s a local problem” -- speechless
7/17/2015
NWHS-Georgia-2003
28
Tip of the iceberg
 We know of these because senior faculty were
involved.
 Junior faculty, and students, dare not publicize
such workplace bullying (and that is what it is).
 The “best” result is that young scholars learn
deference: avoid controversy, novelty. The only
agenda for which this is desirable is "control,"
censorship, it certainly doesn't serve discovery.
 Others find other niches for "scholarship"?
7/17/2015
NWHS-Georgia-2003
29
What does the future hold?
 Continuing this ideological engineering, how
implausible is it that soon researchers will be
required to submit their results back to the ethics
committee before submitting to a journal?
» Drug companies expect it now, why not everyone again?
» This would be quite consistent with the control
(censorship) agenda, but like other changes would have
absolutely no benefit to public safety .
 Then there is accreditation looming, again without
any evidence of need nor any assessment plan:
A bureaucracy's main goal is growth.
7/17/2015
NWHS-Georgia-2003
30
It’s a duck
 A primary criterion as to whether a project has to
be reviewed is whether it is intended for publication
("contribution to generalized knowledge").
 That is, it is not really whether the activity per se is
"ethical," or safe !
Looks like, sounds like, censorship to me.
Censors always claim a good reason, even the Nazis.
7/17/2015
NWHS-Georgia-2003
31
Politicizing research
Censors to the left of me, censors to the right?
» Grant funds for this and not for that (vote buying)
» Commercial interests (help the university with funding)
» Research criticized as “irrelevant,” frivolous, elitist,
controversial, or undesirable (outcome), etc.
» Journals and politicians (e.g., Rind/Lilienfeld at APA, Child
Development 2003, etc.)
» Morton Hunt (1999) - "The new know-nothings."
Are disciplines “better” than 50 years ago? Is life gentler and
kinder? Where’s the payoff (to disciplinary databases) for all the
control we have conceded in order to get federal grants to
campus?
7/17/2015
NWHS-Georgia-2003
32
"I like Ike"
Dwight D. Eisenhower, "Farewell Address," January
17, 1961:
"The prospect of domination of the nation's
scholars by Federal employment, project
allocations, and the power of money is ever
present -- and is gravely to be regarded."
"The kept university,"
http://www.theatlantic.com/issues/2000/03/press.htm
7/17/2015
NWHS-Georgia-2003
33
Research: NOT just a
"privilege" ("honor")
 It’s a job requirement (faculty)
 It’s a degree requirement (students)
 It’s a form of learning, and if you must ask
someone else’s permission for that, in a university,
we are in big trouble.
 Isn’t limiting communication between people
actually an infringement of the “freedom of
association”?
7/17/2015
NWHS-Georgia-2003
34
Am I less ethical than
Geraldo? Jason Blair?
 Amazing double standard between what a
trained and experienced academic can do and
what the rest of the world can do, e.g.,
journalists, pollsters -- BBC repeats Zimbardo
 Consider how robust people really are:
» Everyday risk (SARS, West Nile, photo radar, etc.)
» Everyday ethics (Mr. "did not inhale" Clinton, et al.)
» Everyday deception (Baghdad Bob, Enron et al.)
People just don't break all that easily!
7/17/2015
NWHS-Georgia-2003
35
Concluding - I
The mandate: the Nuremberg trials for Nazi
war crimes.
» Does the desire to stamp out mad scientists
justify it all? Any mad scientists here? Reviewed
any ethics applications from one lately?
» None of the modern research ethics rigmarole
would have prevented them (even in medical
research reviews).
Those atrocities can’t be used to justify the tactics and minutiae
of modern IRBs in regard Social Science research. Ditto for
Milgram, Zimbardo, et al.
7/17/2015
NWHS-Georgia-2003
36
Concluding - II
 However, the Nuremberg trials did establish
something else, namely that “just following orders”
is not an acceptable defense.
» So, again: why is compliance with research ethics regulations more
important than knowing “why,” more important than knowing about
their effectiveness?
 Further, it is interesting to speculate just what the
Nuremberg judges might have said about the
practice of "protecting the institution from risk"
(Nature, 2001).
H.L. Mencken:
"The urge to save humanity is almost always only a falseface for the urge to rule it."
7/17/2015
NWHS-Georgia-2003
37
Concluding - III
 Transparency and accountability should be minimal expectations
for "ethics" reviews, not
» Secretive
» Self-policing
» Self-serving, Self-indulgent, Self-perpetuating, …
 Benefits to public safety must be documented for any rules.
 Expunge the “warm fuzzies” and the “velvet totalitarianism”
(Furedy) - that’s just a platform for censorship.
 Give up the “moral panic” (Fekete) strategy: turn down the volume
of rhetoric from the ethicists and activists.
 Honor proportionate, expedited, exempt, and "everyday risk."
 Lose the medical model (for social sciences) altogether.
 The word “protocol” is for Consumers Reports.
 Review boards are a service to research, not vice versa.
7/17/2015
NWHS-Georgia-2003
38
References - I
Berry, C. (2001). Risk, science, and society.
< http://www.spiked-online.com/Articles/00000002D29C.htm>
Editorial (2001). Time to cut regulations that protect only regulators. Nature, 22
(November) 2001, v. 414, p. 379.
Fekete, J. (1994). Moral panic: Biopolitics rising. Montreal: Robert Davies, 1994.
Furedy, J.J. (1997). Velvet totalitarianism on Canadian campuses: Subverting
effects on the teaching of, and research in, the discipline of psychology. Canadian
Psychology, 38, 204-211.
Hunt, M. (1999). The new know-nothings. Transaction Publishers: New Brunswick,
NJ.
Kalbfleisch, J. (1994-1997). Articles in the Montreal Gazette (re Justine Sergent
suicide). <http://mueller.educ.ucalgary.ca/sergent.txt>
7/17/2015
NWHS-Georgia-2003
39
References - II
Mueller, J. H., & Furedy, J (2001). Reviewing for risk: What's the evidence it is
working? Observer, 14 (September), 1, 26-28.
<http://www.psychologicalscience.org/observer/0901/irb_reviewing.html>
Mueller, J. H., & Furedy, J (2001). The IRB review system: How do we know it
works? Observer, 14 (October),19-20.
<http://www.psychologicalscience.org/observer/1001/irbsystem.html>
Mueller, J. H., Furedy, J. & Seligman, C. (2002) ABCs of IRBs. Observer, 15
(October), 7. <http://mueller.educ.ucalgary.ca/Observer2002-ABCs.html>
Mueller, J. H., Furedy, J. & Seligman, C. (2003) IRBs for Dummies. Observer, 16
(February), 7. <http://mueller.educ.ucalgary.ca/ObserverFeb2003-Dummies.html>
Mueller, J. H. (2003). The origins of the riotous elephant barricade (REB). Presented
at the Society for Academic Freedom and Scholarship, London, Ontario, May 3,
2003. <http://mueller.educ.ucalgary.ca/SAFS2003/>
7/17/2015
NWHS-Georgia-2003
40
References - III
Mueller, J. H., Furedy, J., & Seligman, C. (2003). Re: 'Accreditation helps.'
Observer, 14 (July), 6. <http://mueller.educ.ucalgary.ca/ObserverJuly2003.html>
Ravitch, D. (2003). The language police. NY: A.A. Knopf.
<http://www.languagepolice.com/>
Shea, C. (2000). Don't talk to the humans. Lingua Franca, Volume 10, No. 6
(September), 2000. <http://mailer.fsu.edu/~njumonvi/irb-article.htm>
Tavris, C. (2002). The high cost of skepticism. Skeptical Inquirer, 26(4), 41-44.
http://www.csicop.org/si/2002-07/high-cost.html
7/17/2015
NWHS-Georgia-2003
41
Why is the sky blue?
 Asking “why” is not childish.
 Asking for evidence is not childish.
» “The feds make us do it,” a local response, is not
an acceptable answer.
» “Because I say so,” the federal response, is not
an acceptable answer. Nor is “Who cares,” nor is
“Trust us."
Actually, aren’t these answers childish?
7/17/2015
NWHS-Georgia-2003
42
Serenity Prayer, a better
mandate than assimilation?
“Give us serenity to accept what cannot be changed, courage
to change what should be changed, and wisdom to know the
one from the other.”
There is some latitude with regard to local regulations, don’t
squander it. If nothing else, effectiveness can be assessed
locally. Who could oppose that? It should be as noncontroversial as, well, public safety.
7/17/2015
NWHS-Georgia-2003
43
Why the deceptions?
"You can fool some of the people all the
time, and all of the people some of the
time, but you cannot fool all of the people
all the time." (Abraham Lincoln)
"You can fool too many of the people too
much of the time." (James Thurber)
7/17/2015
NWHS-Georgia-2003
44
Disciplinary autonomy gone?
 Including lay people, lawyers, and others, puts a
discipline’s identity into the hands of novices, people
who have no stake in moving forward the intellectual
integrity of the discipline.
 These “voices” are incapable of judging methodology
and other substantive matters.
 The epistemology of a discipline is a matter for its
members, those with requisite expertise, those who
stand to lose if the disciplinary database is
compromised by other agendas.
And this is happening without a shed of evidence that safety is
improved at all.
7/17/2015
NWHS-Georgia-2003
45
Why the annual review?
Just as she was celebrating her 80th birthday, our friend
received a jury-duty notice. She called to remind the people
at the clerk's office that she was exempt because of her
age.
"You need to come in and fill out the exemption forms," they
said.
"I've already done that," she replied. "I did it last year."
"You have to do it every year," she was told.
"Why?" came the response. "Do you think I'm going to get
younger?"
7/17/2015
NWHS-Georgia-2003
46
"Protocols" atypical?
Much of the medical "research" that attracts so much attention
isn't really research, it is just product testing. It is like
Consumer Reports: is this toaster (drug) better than that one?
In such cases there is indeed the PROTOCOL, there is an
established right way to do it, innovation is not appropriate.
This protocol mindset is not only a barrier for anything new, it convinces
some reviewers they know how to and should be allowed to evaluate
"methodology."
However, in BASIC research the "right" method is not typically
clear in advance, not does basic research always have a clear
"social contribution" in the short-term. Theory testing and
theory development (basic research) is not product testing.
“Outsiders” can’t contribute meaningfully, even insider experts
can get it wrong: that's why you do the research!
7/17/2015
NWHS-Georgia-2003
47