Transcript Decision Making Under Deep Uncertainty Robert Lempert Director RAND Pardee Center for Longer Range Global Policy and the Future Human Condition July 8, 2009

Decision Making Under Deep Uncertainty
Robert Lempert
Director
RAND Pardee Center for Longer Range Global Policy
and the Future Human Condition
July 8, 2009
Traditional Planning Methods Can
Illuminate Trees Rather Than Forest
Traditional analytic methods characterize uncertainties as a prelude to
assessing alternative decisions
Predict
Act
Climate change confronts decisionmakers with deep uncertainty,
where
– They do not know, and/or key parties to the decision do not agree on, the system
model, prior probabilities, and/or “cost” function
Decisions can go awry if decisionmakers assume risks are wellcharacterized when they are not
– Uncertainties are underestimated
– Competing analyses can contribute to gridlock
– Misplaced concreteness can blind decision-makers to surprise
2
Forecasting the Unpredictable
Can Contribute to Bad Decisions
Gross national product (trillions of 1958 dollars)
• In the early 1970s
forecasters made
projections of U.S
energy use based on
a century of data
2.2
2.0
1.8
1.6
1.4
1.2
1.0
.8
.6
.4
.2
0
1975 Scenarios
1973
1890
1900
1973
1970
1910
0
20
Historical
trend
continued
40
1960
1950
1940
1920 1929
60
80
100
120
140
160 180
Energy use (1015 Btu per year)
3
Forecasting the Unpredictable
Can Contribute to Bad Decisions
Gross national product (trillions of 1958 dollars)
• In the early 1970s
forecasters made
projections of U.S
energy use based on
a century of data
… they all were wrong
2.2
2.0
1.8
1.6
1.4
1.2
1.0
.8
.6
.4
.2
0
1975 Scenarios
2000
2000Actual
Actual
1990
1990
1890
1900
1973
1970
1910
0
20
Historical
trend
continued
1980
1980
1977
1977
1973
40
1960
1950
1940
1920 1929
60
80
100
120
140
160 180
Energy use (1015 Btu per year)
4
Outline
• Robust Decision Making (RDM)
• Climate vulnerability and response option analysis
for Inland Empire Utilities Agency (IEUA)
• Observations
5
New Technology Allows Computer to Serve As
“Prosthesis for the Imagination”
• Robust Decision Making (RDM) is a quantitative decision analytic
approach that
– Characterizes uncertainty with multiple, rather than single, views of the future
– Evaluates alternative decision options with a robustness, rather than optimality,
criterion
– Iteratively identifies vulnerabilities of plans and evaluates potential responses
Candidate
strategy
Identify
vulnerabilities
Assess alternatives
for ameliorating
vulnerabilities
• RDM combines key advantages of scenario planning and quantitative
decision analysis in ways that
– Decision makers find credible
– Contribute usefully to contentious debates
6
RDM Has Effectively Addressed Many Types
of Decisions Under Deep Uncertainty
Energy,
Environment,
and Climate
Change
• Long-Range Natural Resource Management
National
Security
• Terrorism Insurance
• Renewable portfolios standards
• Center on climate change decision making
• Force procurement and deployment
• Pre-conflict shaping strategies
CommercialSector
Applications
• Electric utilities’ strategies under deregulation
• Product and technology planning in the auto
industry
7
Compare Alternative Approaches to Managing
Catastrophic Event with Unknown Probability
• Consider town on shore of pristine lake
– Lake can switch abruptly to undesirable and potentially
irreversible eutrophic state at unknown pollution concentration
• Citizens must decide how much pollution to emit
– Gain small utility from emitting pollution to lake and lose
significant utility if lake goes eutrophic
– Deeply uncertain about location of concentration threshold
• Alternative decision approaches include:
– Optimum expected utility
– Precautionary principle
– Robust decision making
Robert J. Lempert and Myles T. Collins., 2007: “Managing the Risk of Uncertain Threshold Response:
Comparison of Robust, Optimum, and Precautionary Approaches” Risk Analysis 27 (4), 1009–1026
8
Use Simple Simulation Model of Lake System to
Assess Consequences of Town’s Decisions
•Three “policy levers” describe town’s citizens’ adaptive strategy
– Initial pollution emissions (L0)
– Maximum yearly increase in emissions (DL)
– Safety margin (S) – buffer between pollution emissions and estimate of critical
threshold (Xcrit)
•Over time, citizens learn true value of critical threshold
L  f L ,D L, S 
– Observations increasingly accurate
as level of pollution approaches unknown
threshold
t
0

Natural Emissions
Nutrients in Lake
Anthropogenic
Emissions
Lt = f(L0, DL,S)
Recycling when
eutrophic
Nutrient sink
Learning
Lempert and Collins (2007)
9
Well Characterized Uncertainty
Suggests An Optimal Strategy
Optimal
Strategy w
Uncertainty
0.45
0.4
0.35
Probability distribution
for critical threshold
Probability Density
0.3
0.25
0.2
0.15
Linit
0.37
Safety
Margin
3.0
DL
0.11
Mean PVU
13.
0.1
0.05
0
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Xcrit
Xcrit
Lempert and Collins (2007)
10
Robust Strategies Trade Some Optimal
Performance for Less Sensitivity to Assumptions
• Represent uncertainty about critical threshold with set of
multiple, plausible distributions  i  X 
• Define expected regret of strategy s contingent on
distribution i
R s, i   R s  x  i  x dx
x
where strategy s regret is R x  Max pv U x  pv U x
 s'  
s 
s 
s'

• Compared
to optimal strategy, a robust strategy has small
weighted average ofbest and worst expected regret
V s  zR s,best  1  z R s,worst
0  z 1

Lempert and Collins (2007)
11
Find Vulnerabilities of Optimal Strategy
Regret of Strategy A over different values of Xcrit
307
73
30
Regret of Strategy A
25
20
Present value utility of optimum strategy
15
10
5
0
0.3
0.4
Vulnerabilities
0.5
0.6
0.7
0.8
0.9
Xcrit
Lempert and Collins (2007)
12
Compare Regret Over Range of Futures
Expected Regret over Initial Priors (z=1)
9
8
G
7
F
6
E
5
D
4
Optimal Strategy
C
3
B
2
A
1
0
0
20
40
60
80
100
120
140
Expected Regret over Strategy A's Vulnerable Futures (z=0)
Lempert and Collins (2007)
13
Town’s Citizens Have More Robust Options
Than Strategy A
20
18
A
16
C
Optimal Strategy
B
Expected Regret
14
12
Potential
Robust Strategy
10
D
G
8
F
6
E
4
2
0
0.01
0.1
1
10
100
Odds of Strategy A Priors
Lempert and Collins (2007)
14
Outline
• Robust Decision Making (RDM)
• Climate vulnerability and response option analysis
for Inland Empire Utilities Agency (IEUA)
– What impacts may climate change have on IEUA’s current
plans?
– What should IEUA do in response?
• Observations
15
Climate Change Poses Significant
Planning Challenge for Water Managers
• Climate change will likely have large but uncertain
impacts on supply and demand for water
• “Stationarity is dead”
– Most agencies already include climate (often implicitly) in many
decisions
– Amidst all the uncertainty one thing we do know for sure -tomorrow’s climate will not be like the past’s
• Relaxing this assumption poses key challenges
– How do you adjust plans based on uncertain climate projections?
– How do you communicate these plans, especially when uncertain
long-term benefits require near-term costs?
16
Conducted Vulnerability and Options Analysis
for Inland Empire Utilities Agency (IEUA)
– IEUA currently serves 800,000
people
• May add 300,000 by 2025
– Water presents a significant
challenge
David G. Groves, Debra Knopman, Robert J. Lempert, Sandra H. Berry, and Lynne Wainfan, Presenting
Uncertainty About Climate Change to Water Resource Managers, RAND TR-505-NSF, 2007.
17
Conducted Vulnerability and Options Analysis
for Inland Empire Utilities Agency (IEUA)
– IEUA currently serves 800,000
people
• May add 300,000 by 2025
– Water presents a significant
challenge
– Current water sources include:
• Groundwater
Groves et. al. (2007)
56%
• Imports
32%
• Recycled
1%
• Surface
8%
• Desalter
2%
18
Conducted Vulnerability and Options Analysis
for Inland Empire Utilities Agency (IEUA)
– IEUA currently serves 800,000
people
• May add 300,000 by 2025
– Water presents a significant
challenge
– Current water sources include:
• Groundwater 56%
• Imports
32%
• Recycled
1%
• Surface
8%
• Desalter
2%
Focus of IEUA’s 20 year plan
Groves et. al. (2007)
19
We Built a Model to Assess Performance of
IEUA Plans in Different Future States of World
IEUA
Plans
System data &
climate forecasts
Model
Performance
of plans
– Model projects future water
supply and demand for IEUA
service area
•
Consistent with IEUA
management plans and
assumptions
•
Reflect plausible trends of
climate change
Groves et. al. (2007)
Based on WEAP software tool
20
GCMs Project Plausible Temperature and
Precipitation Ranges for Southern California
5
.4
Temperature
6
4
3
.3
8
.2
2
1
.1
9
0
Probability Density
7
-1
.03
2
7
8
1
9
Precipitation
-30
Groves et. al. (2007)
5
0
(Tebaldi et al.)
3
6
3
.02
Probability Density
– Each forecast weighted
by ability to reproduce
past climate and level of
agreement with other
forecasts
4
.01
– Derived from forecasts
from 21 GCMs with A1B
emissions scenario
0
1
2
Change in summer temperature (deg C) from 2000 - 2030
-20
-10
0
10
Percent change in winter precipitation from 2000 - 2030
20
21
Generate Future Weather Sequences by
Resampling Historic Local Climate Records
100 120
Historical
0
20 40 60 80
Drier
1980
2000
2020
Year
2040
Groves et. al. (2007)
2060
28
Temperature
Hotter
26
Warmer
Neutral
Historical
20
Temperature (dec C)
– Temperature and
precipitation trends that
match climate model
forecasts
(Yates et al.)
Neutral
IEUA
24
– Daily and monthly variability
that matches historic Chino
climate
Wetter
22
Precipitation (cm)
KNN method produces
hundreds of local weather
sequences
Precipitation
IEUA
1980
2000
2020
Year
2040
2060
22
Model Assess Performance of IEUA Plans
in Many Different Scenarios
IEUA Plans
Performance
of plans
Model
System data & climate forecasts
Scenario B
Plan suffers shortages in
adverse future climate
400
400
350
350
Surplus
300
Imports
250
200
Local Supplies
150
Groundwater
100
50
0
2005
2015
2020
Year
2025
2030
Shortage
Dry-year yield
Surplus
300
Imports
250
200
Local Supplies
150
Groundwater
100
50
Recycled
2010
Annual supply (taf)
Annual supply (taf)
Scenario A
Plan generates surpluses in
benign future climate
0
2005
Recycled
2010
2015
2020
2025
2030
Year
Temp: +0.7oC Precip: +3%
Temp: +1.6oC Precip: -10%
Groves et. al. (2007)
23
Many Uncertain Factors Could Impact
the Performance of Current IEUA Plan
Natural
Processes
• Future temperatures
• Future precipitation
• Changes in groundwater processes
Performance of
Management
Strategies
Costs of Future
Supplies and
Management
Activities
• Development of aggressive waste-water
recycling program
• Implementation of groundwater
replenishment
• Imported supplies
• Water use efficiency
Groves et. al. (2007)
24
Planners in S. California, for Instance, Face a
Range of Possible Future Climate Conditions
Summer-time temperature change
(2000- 2030)
No change
Hotter
Likely range
0
+.1C
+2.1C
Winter-time precipitation change
(2000 - 2030)
Much drier
Wetter
Likely range
-19%
0
+8%
Results based on statistical summary of 21 of the world’s best
Global Climate Models
Groves et. al. (2007)
25
Many Uncertain Factors Could Impact
the Performance of Current IEUA Plan
Natural
Processes
• Future temperatures
• Future precipitation
• Changes in groundwater processes
Performance of
Management
Strategies
Costs of Future
Supplies and
Management
Activities
• Development of aggressive waste-water
recycling program
• Implementation of groundwater
replenishment
• Imported supplies
• Water use efficiency
Groves et. al. (2007)
26
“Scenario Maps” Help Decision Makers
Visualize How Plans Evolve Over Many Futures
Current IEUA 2005 Urban Water Management Plan
4.0
PV supply 3.5
cost
($ billions)
Scenario B
Scenario A
3.0
•
Benign climate
•
$3.3 billion in supply
cost
•
$0 in shortage cost
•
Adverse climate
•
$3.4 billion in supply
cost
•
$1.9 billion in
shortage cost
2.5
0
1.0
2.0
3.0
4.0
PV shortage cost ($ billions)
David G. Groves, Robert J. Lempert, Debra Knopman, Sandra H. Berry: Preparing for an Uncertain
Climate Future: Identifying Robust Water Management Strategies, RAND DB-550-NSF, 2008.
27
“Scenario Maps” Help Decision Makers
Visualize How Plans Evolve Over Many Futures
Current IEUA Plan
4.0
PV supply 3.5
cost
($ billions)
3.0
(200 Scenarios)
2.5
0
1.0
2.0
3.0
4.0
PV shortage cost ($ billions)
Groves et. al. (2008)
28
“Scenario Maps” Help Decision Makers
Visualize How Plans Evolve Over Many Futures
Current IEUA Plan
4.0
Current plan
generates high costs
in 120 of 200
Scenarios
PV supply 3.5
cost
($ billions)
3.0
$3.75 billion
cost threshold
2.5
0
1.0
2.0
3.0
4.0
PV shortage cost ($ billions)
Groves et. al. (2008)
29
Discover Key Scenarios
in Ensembles of Many Model Runs
1.
Ran the model 200 times under different combinations of uncertain factors
(e.g. temperature and precipitation trends and others)
2.
Used statistical algorithms to identify conditions that lead to 2005 UWMP to
perform poorly
3.
These factors become key driving forces for “policy-relevant” scenarios
Number
60 70
10 20 30 40of50runs
Statistical analysis
suggests factors that
contribute most to
these undesirable
outcomes
High Cost
(120 runs)
0
Number of futures
80
UWMP Forever
2.5
3
3.5
4
4.5
5
5.5
NPV total costs ($ billions)
6
6.5
Groves et. al. (2007)
30
Statistical Analysis Suggests Key Factors
That Create Vulnerabilities for Existing Plan
•
Natural
Processes
Future temperatures
• Future precipitation
• Changes in groundwater
processes
Current IEUA Plan
PV supply cost ($ billions)
4.0
• Development of aggressive
waste-water recycling program
Performance of
Management
Strategies
Costs of Future
Supplies and
Management
Activities
3.5
•
Implementation of groundwater
replenishment
•
Imported supplies
•
Water use efficiency
These three factors
explain 70% of
vulnerabilities of
IEUA’s current plans
3.0
2.5
0
1.0
2.0
3.0
PV shortage cost ($ billions)
4.0
Groves et. al. (2008)
31
Response Options May Help IEUA
Address These Vulnerabilities
Groves et. al. (2008)
32
Can Quantify Some, But Not All, Of These Costs
Average Cost
Shortages
Desalted Groundwater
Imported (Tier 2)
Imported Replenishment*
Imported (Tier 1)
Recycled Replenishment*
Groundwater
Stormwater Replenishment*
Recycled
Saved through efficiency
0
200
* includes the cost of spreading
400
600
800
1000
1200
Cost in 2005 ($/AF)
Costs increase over time
Groves et. al. (2008)
33
Should IEUA Act Now or Later to
Reduce Potential Climate Vulnerabilities?
In 2015, 2020, 2025, ….
NO
Monitor, and take
additional action
if supplies drop
too low
Act now to
augment
In 2015, 2020, 2025, ….
2005 Plan?
YES
Implement
additional
efficiency,
recycling, and
replenishment
Monitor, and take
additional action
if supplies drop
too low
Groves et. al. (2008)
34
Compare Nine Strategies Over
200 Scenarios Reflecting Key Uncertainties
Current Plan forever
Current Plan + DYY and recycling
Current Plan + replenishment
Current Plan with updates
Current Plan + DYY and recycling with updates
Current Plan + replenishment with updates
Current Plan + efficiency
Current Plan + efficiency with updates
Static options
Update options
Current Plan + all enhancements
0
20
40
60
80
100
120
Number of Scenarios (PV Costs > $3.75 billion)
Groves et. al. (2008)
35
Just Allowing IEUA’s Current Plan to Update
Reduces Vulnerability Substantially
Current Plan forever
Current Plan + DYY and recycling
From 120
Down to 30
Current Plan + replenishment
Current Plan with updates
Current Plan + DYY and recycling with updates
Current Plan + replenishment with updates
Current Plan + efficiency
Current Plan + efficiency with updates
Static options
Update options
Current Plan + all enhancements
0
20
40
60
80
100
120
Number of Scenarios (PV Costs > $3.75 billion)
Groves et. al. (2008)
36
Acting Now
Reduces Future Vulnerabilities Even More
Current Plan with updates
Current Plan + DYY and recycling with updates
Current Plan + replenishment with updates
Current Plan + efficiency
Static options
Update options
Current Plan + efficiency with updates
Current Plan + all enhancements
0
10
20
30
40
Number of Scenarios
(PV Costs > $3.75 billion)
Groves et. al. (2008)
37
Acting Now
Reduces Future Vulnerabilities Even More
Current Plan with updates
Current Plan + DYY and recycling with updates
Current Plan + replenishment with updates
Current Plan + efficiency
Static options
Implementation
becomes
more
challenging
Update options
Current Plan + efficiency with updates
Current Plan + all enhancements
0
10
20
30
40
Number of Scenarios
(PV Costs > $3.75 billion)
This analysis helped IEUA decide to make more near-term
efficiency investments, and to monitor performance and adapt
as needed down the road
Groves et. al. (2008)
38
Outline
• Robust Decision Making (RDM)
• Climate vulnerability and response option analysis
for Inland Empire Utilities Agency (IEUA)
• Observations
39
Conducted Elicitations Among IEUA’s
Planners and Community to Estimate
Likelihood of Achieving Goals
Probability of meeting UWMP goals
.04
Goal
Miss
goal
Goal
.01
.01
.02
Density
.02
.03
Miss
goal
0
0
Density
Replenishment
.03
Recycling
40
50
60
Recycling
70
80
80
90
100
110
120
130
GW
Meet Goals
Miss Goals
Groves et. al. (2007)
40
Many Uncertain Factors Could Impact
the Performance of Current IEUA Plan
Natural
Processes
• Future temperatures
• Future precipitation
• Changes in groundwater processes
Performance of
Management
Strategies
Costs of Future
Supplies and
Management
Activities
• Development of aggressive waste-water
recycling program
• Implementation of groundwater
replenishment
• Imported supplies
• Water use efficiency
Groves et. al. (2007)
41
Analysis Suggests Factors That Cause
Severe Shortages for IEUA’s 20 Year Plan
Climate-related uncertainties facing IEUA
Meet recycling goal
Miss
Meet
Exceed
Miss
Meet
Exceed
Meet replenishment goal
Future climate
Drier
Wetter
-5%
+20%
-20%
0%
Weak
Strong
New conservation
Percolation decrease
Climate on imports
Explains 70% of high cost cases
Groves et. al. (2007)
42
RDM Enables Effective Planning
Based on Multiple Views of Future
• Use many scenarios to imagine the future
– Not a single forecast
• Seek robust strategies that do well across many scenarios
assessed according to several values
– Not optimal strategies
• Employ strategies that evolve over time in response to
changing conditions
– Not "fixed" strategies
• Use computer as “prosthesis for the imagination”
– Not a calculator
43
More Information
David G. Groves, Robert J. Lempert, Debra Knopman, Sandra H. Berry: Preparing for
an Uncertain Climate Future: Identifying Robust Water Management Strategies,
RAND DB-550-NSF, 2008.
David G. Groves, Debra Knopman, Robert J. Lempert, Sandra H. Berry, and Lynne
Wainfan, Presenting Uncertainty About Climate Change to Water Resource
Managers, RAND TR-505-NSF, 2007.
Groves, David G, David Yates, Claudia Tebaldi, 2008: “Developing and Applying
Uncertain Global Climate Change Projections for Regional Water Management
Planning,” Water Resources Research, 44(12): W12413
Robert J. Lempert and Myles T. Collins., 2007: “Managing the Risk of Uncertain
Threshold Response: Comparison of Robust, Optimum, and Precautionary
Approaches” Risk Analysis 27 (4), 1009–1026
David G. Groves and Robert J. Lempert, 2007: A new analytic method for finding
policy-relevant scenarios, Global Environmental Change 17, 73-85.
Robert J. Lempert, Steven W. Popper, Steven C. Bankes, 2003: Shaping the Next
One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis,
RAND MR-1626-RPC, Aug.
www.rand.org/ise/projects/improvingdecisions/
44
Thank you!
45
We Also Evaluated How This Analysis
Affected Policy-Makers’ Views
• Four IEUA workshops presented modeling results
to participants including:
– Agency professional managers and technical staff
– Local elected officials
– Community stakeholders
• “Real-time” surveys measured participants’
– Understanding of concepts
– Willingness to adjust policy choices based on information
presented
– Views on RDM
Groves et. al. (2007)
46
First Three Workshops Compared
Alternative Approaches to Uncertainty
Compared three approaches
• Traditional qualitative scenarios
• Probabilistic forecasts
• RDM with Scenario discovery
Workshop design approximates on-going laboratory experiments
Groves et. al. (2007)
47
RDM Scenarios More Useful,
But More Difficult to Understand
Questionnaire item
Traditional
Scenarios
Scenario
Discovery
Provides results that can
be used in planning
Agree
somewhat
Agree
strongly
Provides information on
how to improve plan
Agree
somewhat
Agree
somewhat
Is easy to explain to
decisionmakers
Agree
somewhat
Disagree
strongly
• Traditional scenarios
– Gave IEUA much of the information they needed
– Emphasized the importance of achieving goals in IEUA’s plan
• Scenario Discovery
– Provided more useful information
– Sparked discussion of adaptive strategies
Groves et. al. (2007)
48
RDM Scenarios More Useful,
But More Difficult to Understand
Questionnaire item
Traditional
Scenarios
Scenario
Discovery
Provides results that can
be used in planning
Agree
somewhat
Agree
strongly
Provides information on
how to improve plan
Agree
somewhat
Agree
somewhat
Is easy to explain to
decisionmakers
Agree
somewhat
Disagree
strongly
• Traditional scenarios
– Gave IEUA much of the information they needed
– Emphasized the importance of achieving goals in IEUA’s plan
• Scenario Discovery
– Provided more useful information
– Sparked discussion of adaptive strategies
49
Fourth (Adaptive Strategy) Workshop
Compared Different Presentations of RDM
Participants reported:
– RDM helped support comparison of climate-related risks and choice among
plans
– Preference for scatter plot over histogram scenario displays
After the workshop:
– 35% said consequences of bad climate change now appeared “more
serious” than before
– 40% thought the likelihood of of bad climate change outcomes for the IEUA
was “greater” than before
– 75% though the ability of IEUA planners to plan for and manage effects was
“greater” than before
Overall, analysis increased:
– Perceived likelihood of serious climate impacts
– Confidence that IEUA could take effective actions to reduce its vulnerability
to climate change
– Support for near-term efficiency enhancements to current IEUA plan
Groves et. al. (2008)
50
Observations
• Analysis suggests IEUA’s current long-range plans:
– Vulnerable to climate change
– Can be made more resilient by near-term conservation,
attention to storm intensity, and effective future monitoring
and updating
• Measurements suggest
– RDM analysis effectively shifted views on seriousness of
climate challenges and appropriate responses, but requires
more work to be easily understood by policy-makers
– Importance of linking effective response options with
presentation of climate uncertainty
• Currently using this approach to help several major water
agencies include climate in their long-range plans
51