Transcript Biases in Environmental Decision Analysis

Can We Avoid Biases in
Environmental
Decision Analysis ?
Raimo P. Hämäläinen
Helsinki University of Technology
Systems Analysis Laboratory
[email protected]
www.paijanne.hut.fi
S ystems
Analysis Laboratory
Helsinki University of Technology
Structure of the presentation
•
•
•
•
•
•
•
Background & decision analysis interviews
Goals of the study
Case: Regulation of Lake Päijänne
Splitting bias & swapping of levels
Description of the experiment
Results of the experiment
Conclusions ?
S ystems
Analysis Laboratory
Helsinki University of Technology
Environmental decision analysis
• Parliamentary nuclear power decision
(Hämäläinen et. al)
• Decision analysis interviews
(Marttunen & Hämäläinen)
• Spontaneous decision conferencing in
nuclear emergency management
(Hämäläinen & Sinkko)
S ystems
Analysis Laboratory
Helsinki University of Technology
Cognitive biases
• Splitting bias
– attribute receives more weight if it is split
– origins: subjects give rank information only
(Pöyhönen & Hämäläinen)
– Not observable in hierarchical weighting
S ystems
Analysis Laboratory
Helsinki University of Technology
Decision analysis interviews
• Opinions of large groups of people
traditionally collected through questionnaires
• Decision analysis interviews may provide a
more reliable way to collect these opinions
• Idea:
– one value tree for all = common terminology
– emphasis on finding the viewpoints of different
stakeholder groups
– interactive, computer supported
S ystems
Analysis Laboratory
Helsinki University of Technology
Research interest
• Existence of biases in a real case
• Can biases can be avoided through
training and proper instructing ?
• Identify what can go wrong in the Lake
Päijänne case
• Compare the well trained university
students’ and spontaneous stakeholders’
responses
S ystems
Analysis Laboratory
Helsinki University of Technology
The Lake Päijänne case
• Regulation started 1964
• Main aims were to improve hydroelectricity
production and to reduce damages caused by
flooding
• Environmental values & increase in free time
• need for an improved regulation policy
S ystems
Analysis Laboratory
Helsinki University of Technology
Splitting bias
• When an attribute is split, the weight it
receives increases
0.4
Attribute 1
0.4
0.3
Attribute 2
Attribute 3
0.3
Attribute 3
Analysis Laboratory
Helsinki University of Technology
0.1
subattribute 1b
0.3
Attribute 1
Attribute 2
S ystems
Subattribute 1a
0.3
0.3
Swapping of levels
• Does the order of the levels affect the
resulting weights?
• Important question in environmental
decision analysis:
– stakeholder groups may vary regionally
• Not studied before
S ystems
Analysis Laboratory
Helsinki University of Technology
Example of swapping of levels
Attribute 1
Lake Päijänne
Attribute 1
Lake Päijänne
Attribute 2
River Kymijoki
Attribute 3
Lake Päijänne
Attribute 2
River Kymijoki
Attribute 1
River Kymijoki
Attribute 2
Lake Päijänne
Attribute 3
Attribute 3
S ystems
Analysis Laboratory
Helsinki University of Technology
River Kymijoki
Earlier experiments on biases
• Structure of the decision model affects the
results
• Previous experiments typically:
– subjects: university students
– problems: artificial
– results: taken from group averages
• Lake Päijänne-case: a real problem with real
stakeholders
S ystems
Analysis Laboratory
Helsinki University of Technology
Important new features
• Realistic case
• Decision analysis interviews instead of
passive decision support or survey
• Interactive computer support (resulting
weights shown immediately)
• Instructions and training before the
weighting
S ystems
Analysis Laboratory
Helsinki University of Technology
Subjects:
• University students attending a course on
decision analysis (N = 30)
– held during a tutorial session, not mandatory
• Habitants of Asikkala (N = 40)
– 3 groups of students
– 1 group of adults (volunteers)
• 3 experts from the Finnish Environment
Institute & 2 summer residence owners
S ystems
Analysis Laboratory
Helsinki University of Technology
Experimental setting
• Weighting done with the SWING method
using a tailored Excel interface
• Subjects entered the numbers themselves,
two assistants were present to help
• Resulting weights shown as bars
• Order of value trees partly randomized
S ystems
Analysis Laboratory
Helsinki University of Technology
Sessions
• A short introduction to:
– Lake Päijänne case
– value trees & weighting
– different structures of the value tree
• In HUT the avoidance of biases was
emphasized more
• Duration: 60 - 90 minutes
S ystems
Analysis Laboratory
Helsinki University of Technology
SWING method
• Easy to use
• Attribute ranges clearly presented
• Idea:
– choose the attribute you would first like to
move to its best level
– assign it 100 points
– assign other attributes points less than 100 in
respect to the first attribute
S ystems
Analysis Laboratory
Helsinki University of Technology
Flat-weighting
Rantojen käytettävyys
Virkistys
Virkistyskalastus
Kalojen lisääntyminen
Ympäristö
Luonto
Lahtien
umpeenkasvu
Rantakasvillisuus
Vesivoima
Vesivoima
Tulvat, maatalous ja
teollisuus
Muu talous ???
Talous
Tulvat, loma-asutus
Muu talous
Vesiliikenne
Ammattikalastus
S ystems
Analysis Laboratory
Helsinki University of Technology
Upper level weights:
Rantojen käytettävyys
Virkistys
Virkistyskalastus
Kalojen lisääntyminen
Ympäristö
Luonto
Lahtien
umpeenkasvu
Rantakasvillisuus
Vesivoima
Vesivoima
Tulvat, maatalous ja
teollisuus
Muu talous ???
Talous
Tulvat, loma-asutus
Muu talous
Vesiliikenne
Ammattikalastus
S ystems
Analysis Laboratory
Helsinki University of Technology
ENV5-tree:
Rantojen käytettävyys
Virkistys
Virkistyskalastus
Kalojen lisääntyminen
Ympäristö
Lahtien
umpeenkasvu
Luonto
Rantakasvillisuus
Talous
S ystems
Analysis Laboratory
Helsinki University of Technology
ENV2-tree:
Virkistys
Ympäristö
Luonto
Talous
S ystems
Analysis Laboratory
Helsinki University of Technology
EC5-tree:
Ympäristö
Vesivoima
Vesivoima
Tulvat, maatalous ja
teollisuus
Muu talous ???
Talous
Tulvat, loma-asutus
Muu talous
Vesiliikenne
Ammattikalastus
S ystems
Analysis Laboratory
Helsinki University of Technology
EC2-tree:
Ympäristö
Vesivoima
Muu talous ???
Talous
Muu talous
S ystems
Analysis Laboratory
Helsinki University of Technology
Swapping of levels:
Päijänne
Tulvavahingot
Tulvavahingot
Päijänne
Muu talous ???
Muu talous ???
Kymijoki ja muut
Rantakasvillisuus
Päijänne
Rantakasvillisuus
Kymijoki ja muut
Kymijoki ja muut
S ystems
Analysis Laboratory
Helsinki University of Technology
Tulvavahingot
Rantakasvillisuus
Flat weights vs.
upper level weights
• Both in group averages and in results of
individuals the total weights for the
environment and economy were similar with
both methods
• One explanation: symmetric value tree
S ystems
Analysis Laboratory
Helsinki University of Technology
Splitting bias
Means of Wenv
0.5
0.0
0.3
Wenv
0.8
1.0
TREE
EC2
EC5
ENV2
ENV5
ASIKKALA
GROUP
S ystems
Analysis Laboratory
Helsinki University of Technology
HUT
A typical resident in Asikkala
1
ENVIRONMENT
ECONOMY
0.8
0.6
0.4
0.2
0
5 1
S ystems
Analysis Laboratory
Helsinki University of Technology
5
2 1
1
5
1
1 1 5
2
Example from HUT
(one of the best ones)
1
ENVIRONMENT
ECONOMY
5
5
0.8
0.6
0.4
0.2
0
S ystems
Analysis Laboratory
Helsinki University of Technology
1
5 2
1 1
1 1
1 5
2
Why even weights ?
• Some students: none of the attributes seemed
to be important
• Asikkala: all of the attributes were important
even weights for all attributes
S ystems
Analysis Laboratory
Helsinki University of Technology
What caused the bias ?
• Similar points for
all attributes in one branch
regardless of the structure
of the value tree
100
100
80
100
S ystems
Analysis Laboratory
Helsinki University of Technology
80
70
70
80
70
70
50
Effect of instructions
• Students had good instructions
–
only some had bias in their results
• In the spontaneous stakeholders’
sessions the information load was too
high and thus the instructions were not
adopted as well
–
nearly all had systematically consistent bias
S ystems
Analysis Laboratory
Helsinki University of Technology
Adjusted / not adjusted weights
STUDENTS
STAKEHOLDERS
w env
w env
w econ
w econ
n env
n econ
S ystems
Analysis Laboratory
Helsinki University of Technology
n env
n econ
Examples
STUDENTS
w env
w econ
STAKEHOLDERS
w env 7
w econ 6
7
6
5
4
3
2
1
0
5
4
3
2
1
0
0
1
2
3
4
5
n env
n econ
S ystems
Analysis Laboratory
Helsinki University of Technology
0
1
2
3
4
5
n env
n econ
Observation
• The students and the experts from FEI could
nearly avoid the splitting bias
– good background education + instructions did
reduce the bias
• What did the students think? - Arithmetics or
real avoidance of biases
S ystems
Analysis Laboratory
Helsinki University of Technology
Avoiding the splitting bias ?
• Good instruction can eliminate it
• When the economical attributes were split,
the magnitude of the bias was slightly larger
• Graphical feedback did not eliminate
• Hierarchical weighting
S ystems
Analysis Laboratory
Helsinki University of Technology
Swapping of attribute levels
If the order of the levels would not affect the
weigts, the pairs of weights should be equal
(as in the first picture)
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
S ystems
Analysis Laboratory
Helsinki University of Technology
Conclusions about
swapping of levels ?
• Only a few had clearly differing weights
with the two trees
• No systematic pattern was found
• Less differences residents of Asikkala and
students than with the splitting bias
• A simple scale lead to similar weights with
both trees (100, 70 for example)
• Neither tree gained clear support
S ystems
Analysis Laboratory
Helsinki University of Technology
Solutions to reduce biases ?
• Hierarchical weighting
• Models should be tested on real decision
makers
• Interactiveness of weighting (= possibility to
return to change the points given earlier )
• Well balanced trees
S ystems
Analysis Laboratory
Helsinki University of Technology
Other observations in Asikkala
• Concept of weight seemed to be difficult for
most subjects in Asikkala
• Information load was high
• Facilitators role becomes important when the
DM’s are uncertain
S ystems
Analysis Laboratory
Helsinki University of Technology
Problems related to the Lake
Päijänne case
• Current regulation policy cannot be
improved very significantly
– no big differences between the alternatives
– unrealistic hopes and false information are
probably larger problems than the regulation
itself
• ‘money is not money’
– strong feelings against the power companies
and regulation (shape of value function ?)
S ystems
Analysis Laboratory
Helsinki University of Technology
Suggestions for future research
• Hierarchical weighting
• Encouragement to reconsider and readjust
the statements iterate
• Decision Analyst must supervise!
S ystems
Analysis Laboratory
Helsinki University of Technology
References
R.P. Hämäläinen, E. Kettunen, M. Marttunen and H. Ehtamo: Evaluating a framework for
multi-stakeholder decision support in water resources management, Group Decision and
Negotiation, 2001. (to appear)
M. Pöyhönen, Hans C.J. Vrolijk and R.P. Hämäläinen: Behavioral and procedural consequences
of structural variation in value trees. European Journal of Operational Research, 2001. (to appear)
M. Pöyhönen and R.P. Hämäläinen: There is hope in attribute weighting, Journal of Information
Systems and Operational Research (INFOR), vol. 38, no. 3, Aug. 2000, pp. 272-282. Abstract
R.P. Hämäläinen, M. Lindstedt and K. Sinkko: Multi-attribute risk analysis in nuclear emergency
management, Risk Analysis, Vol. 20, No 4, 2000, pp. 455-467.
M. Pöyhönen and R.P. Hämäläinen: Notes on the weighting biases in value trees, Journal of
Behavioral Decision Making, Vol. 11, 1998, pp. 139-150.
Susanna Alaja: Structuring effects in environmental decision models, Helsinki University of
Technology, Systems Analysis Laboratory, Theses, 1998.
S ystems
Analysis Laboratory
Helsinki University of Technology
M. Pöyhönen, R.P. Hämäläinen and A. A. Salo: An experiment on the numerical modeling of
verbal ratio statements, Journal of Multi-Criteria Decision Analysis, Vol. 6, 1997, pp. 1-10.
R.P. Hämäläinen and M. Pöyhönen: On-line group decision support by preference programming in
traffic planning, Group Decision and Negotiation, Vol. 5, 1996, pp.485-50.
M. Marttunen and R.P. Hämäläinen: Decision analysis interviews in environmental impact assessment,
European Journal of Operational Research, Vol. 87, No. 3, 1995, pp. 551-563.
R.P. Hämäläinen, A.A. Salo and K. Pöysti: Observations about consensus seeking in a multiple criteria
environment, in: Proceedings of the Twenty-Fifth Hawaii International Conference on System
Sciences, Vol. IV, 1991, IEEE Computer Society Press, Hawaii, pp. 190-198.
R.P. Hämäläinen: Computer assisted energy policy analysis in the parliament of Finland, Interfaces,
Vol. 18, No. 4, 1988, pp. 12-23.
Also in: Case and Readings in Management Science, 2nd edition, M. Render, R.M. Stair Jr. and
I. Greenberg (eds.), Allyn & Bacon, Massachusetts 1990 pp. 278-288.
S ystems
Analysis Laboratory
Helsinki University of Technology