Transcript Module Continue Modellen (tb231)

Exploratory Modelling and Analysis
an approach for model-based foresight under deep uncertainty
Jan Kwakkel
Erik Pruyt
1
Exploratory Modeling and Analysis (EMA)
“Exploratory modeling is using computational experiments to assist in
reasoning about systems where there is significant (or deep)
uncertainty” (Bankes, 1993)”*
•
•
•
EMA was developed at the RAND Corporation
EMA represents a new way of thinking about the use of computer
models to support policy making
Traditional modeling consists of consolidating known facts about a
system that are then used as a surrogate for the system  when
confronted with uncertainties about details or mechanisms, modelers
use educated guesses (resulting in best estimate predictive models).
2
*BANKES, S. 1993. Exploratory Modeling for Policy Analysis, Operations Research, 43 (3), p. 435-449.
Predictive Modeling vs. Exploratory Modeling
Exploratory Modeling
• Model is used as a hypothesis
generator (“what if . . .”)
• Take into account external
(scenario) uncertainty, structural
(model) uncertainty, and uncertainty
about valuation of outcomes
• The objective is to reason about
system behavior: under which
circumstances would a policy
succeed or fail?
• Uses rapid assessment models,
because the uncertainties may swamp
model results
Predictive Modeling
•
Model is used to predict
•
Take into account (external)
uncertainty; deal with internal
uncertainty using educated guesses
The objective is to predict system
behavior and whether a policy will
succeed or fail
•
•
Aims at detailed models that capture
the state of the art
3
EMA Approach
• Specify policy problem
• Analyze the uncertainties associated with the problem
• Develop one or more fast and simple models consistent with the
available information and knowledge that allow for exploring the
specified uncertainties
• Explore the behavior of these models across the ranges of the
uncertainties
• Assess the implications of the exploration for policy
• Iteratively modify plan in light of revealed weaknesses until a
satisfying plan emerges
4
Explore the parameter space
• Exploration versus directed search
• Exploration (sampling techniques)
• Factorial methods
• Monte Carlo sampling
• Latin Hypercube sampling
• Directed search (optimization techniques)
• Conjugant Gradient Optimization
• Genetic Algorithms
• Simulated Annealing
• Etc.
5
Mineral Scarcity Problem
• Many crucial high-volume minerals are expected to become
exhausted in the coming decades
• The disparity between the expected exponential growth of metal
demand and the expected limited growth of metal supply may result
in temporary and/or chronic scarcity; and
• Strategic and/or speculative behavior of countries that have a quasimonopoly on the extraction of (rare earth) metals may seriously
hinder the transition of modern societies towards more sustainable
ones.
• The asynchronous dynamics of supply and demand, aggravated by
reinforcing behaviors and knock-on effects, is a breeding ground for
acute and/or chronic crises
• What kinds of dynamics can happen?
6
System Dynamics Model
7
PRUYT, E. 2010. Scarcity of Minerals and Metals: A Generic Exploratory System Dynamics Model. In: MOON, T. H. (ed.)
The 28th International Conference of the System Dynamics Society. Seoul, Korea.
Uncertainties to be explored
• Parametric variations
• Lifetime of mines
• Lifetime of recycling facilities
• Initial values for most stock variables
• Price elasticity and desired profit margins
• Order of time delays
• Building time of mines, recycling capacity
• Non linear relations captured in table functions
• Learning effect
• Impact shortages on price
• Substitution behaviour
• In total, 27 uncertainties are jointly explored
8
Results for 100 LHS runs
9
Results
Number of
runs
1000
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
Number of behavior
patters
371
1214
2042
2742
3386
3894
4547
4976
5511
5972
6404
• Behavioral clustering of time
series
• Each run is specified as a
concatenation of atomic behavior
patters
10
Airport Planning Problem
• Schiphol Airport is a
environmentally constrained
airport
• It loses demand to other airport
• Low cost and charter to
regional airports
• Long haul and transfer to
other hubs in Europe
• How can the airport invest to
remain competitive, despite a
wide variety of uncertainties?
11
A Fast and Simple Model for Calculating
Airport Performance
• A variety of tools are readily available for aspects of airport
performance (e.g. noise, emissions, capacity)
• Uncertainty about the airport system itself is low
• The Fast and Simple Component integrates
– FAA capacity tool (FCM)
– FAA emissions tool (EDMS)
– FAA noise tool (AEM)
– NATS external safety methodology
• Outcomes:
– Ratio capacity to demand, latent demand, size of noise contour,
average casualty expectancy, emissions
Uncertainties to be explored
• The main uncertainties faced by airports come from external forces
• We developed generators for key external forces:
– Engine technology (exponential and logistic performance
increase)
– Air Traffic Management technology (exponential and logistic
performance increase)
– Population (logistic growth, logistic growth followed by logistic
decline)
– Aviation Demand (exponential, logistic, decline)
– Composition of fleet (logistic change, linear change)
– Weather (parametric uncertainty)
• Together, these uncertainties result in 48 structurally different
scenario generators, each of which can generate an infinite range of
quantitatively different scenarios
Results
• Basic plan
• Try to limit movements to 510.000 in 2020
• From 2015 move up to 70.000 movements to regional airport
• In 2020, a new runway should become operational
• What is the bandwidth of outcomes for this plan given the
uncertainties?
• Approach:
• Conjugant gradient optimization across all the uncertainties
• Multiple different initializations for each optimization to handle
local vs. global optima
• Time required: roughly a week of computer simulations on a
normal desktop PC
14
Performance Bandwidth of Basic Plan
Basic plan
Noise
Emissions
13 – 64 km2
•
•
2,1 – 19,6 ton CO
External
Safety
0,9 – 2,7 ACE
Ratio
capacity
versus
demand
0,3 – 2,5
•
The basic plan has a very wide
bandwidth
Plan unsuccessful in guiding the future
development of the airport
• Overinvestment in runways
• Unnecessary moving of operations
to regional airports
How to improve the plan?
• Introduce flexibility
• Specify the conditions under which
pre-specified actions are taken
• E.g. build runway only if there is a
certain level of demand or certain
deterioration in capacity do to
wind
15
Performance Bandwidth of Adaptive Plan
Initial Static Plan
Adaptive Plan
13 – 64 km2
10 – 47 km2
2,1 – 19,6 ton CO
1,9 – 10,3 ton CO
External Safety
0,9 – 2,7 ACE
1,1 – 2,3 ACE
Ratio capacity
versus demand
0,3 – 2,5
0, 9 – 1,1
Noise
Emissions
• The adaptive plan significantly reduces the bandwidth of outcomes
on the shown indicators across the same uncertainties
• How big is the difference in performance between the two plans?
• Are there regions were the initial static plan is still better?
16
Results
17
Results
18
Static plan
performs better
Results
19
Modify the adaptive plan to deal
with these regions
Concluding remarks
• Deep uncertainty has to be addressed explicitly in any long-term
decision making problem
• EMA offers a useful technique that allows the utilization of models to
explore the implications of the uncertainties
• EMA can be used to develop dynamic adaptive strategies capable of
coping with the multiplicity of plausible futures
• Research is needed
• Visualizing and analyzing results of exploration
• Communication of results to clients
• Efficient techniques for both directed searches and open
exploration
20
21
DEEP UNCERTAINTY
22
What is uncertainty?
•
•
•
•
“Any departure from the unachievable ideal of complete determinism.
Uncertainty is not simply a lack of knowledge, since an increase in
knowledge might lead to an increase of knowledge about things we
don’t know and thus increase uncertainty (Walker et al. 2003)”*
Sources of uncertainty can be specified according to their nature,
location and level
Nature: character of the uncertainty
• Limited knowledge, inherent variability, ambiguity
Location: which aspect of the system or model are we uncertain
about
• Inputs, model structure, outputs, valuation of outputs
Level: degree of uncertainty
• Ranges from complete certainty to absolute ignorance
23
*WALKER, W. E., HARREMOËS, J., ROTMANS, J. P., VAN DER SLUIJS, J. P., VAN ASSELT, M. B. A., JANSSEN, P. H. M. &
KRAYER VON KRAUSS, M. P. 2003. Defining Uncertainty: A Conceptual Basis for Uncertainty Management in ModelBased Decision Support. Integrated Assessment, 4, p. 5-17.
Deep Uncertainty
• Situation where the relevant actors do not know or cannot agree on
• (Aspects of) how the system works
• How likely or plausible various (paths to) future states are
• How to value the various outcomes of interest
• In almost all long-term decision making problems, deep uncertainty
is encountered
• e.g. the climate change debate
• Few techniques are readily available for offering decision support
• Decision making should be based on robustness instead of optimality
• Robustness is to be achieved in part through adaptiveness
• Only take near term actions that overall have desirable
consequences
• Prepare actions to be taken in light of how the future enfolds
24
Treating Uncertainty in Model-Based Decision
Support
• A wide variety of methods and techniques are available for dealing
with low levels of uncertainty in the context of model-based decision
support
• Sensitivity analysis, Monte-Carlo simulation, Multi-Criteria
Decision Analysis, etc.
• Deep uncertainty is more problematic in model-based decision
support
• e.g. what about disagreement between experts about a functional
relationship in a model?
• Conclusion: Limited capabilities for dealing with deep uncertainty in
the context of model-based decision support
25
EXPLORATION DETAILS
26
Exploration
• Generic approach
• Specify the ranges for the parameters
• Choose a sampling strategy
• Specify the number of samples
• Useful for
• Open exploration of what kind of outcomes are possible
• Open exploration of what kinds of behavior can occur
• Most frequently employed strategy in EMA
• Easy to execute, but big risk of information overload
27
Choosing a distribution
28
Choosing a distribution
29
DIRECTED SEARCH
30
Directed search
• Generic approach:
• Conclusions are derived from a model for a specific set of
parameter values
• For each parameter a range of possible values is specified
• Identify under which combinations of parameter values the
conclusions are invalidated
• Directed search is useful for
• Identifying the worst possible performance of a policy option
• Identifying the maximum difference in performance between
several policy options
• Identifying the conditions under which model behavior changes
(so called tipping points)
31
IDENTIFICATION OF PLAUSIBLE
TRANSITION PATHWAYS FOR THE FUTURE
DUTCH ELECTRICITY GENERATION
SYSTEM
32
Case details
• Problem
• Many electricity companies have to replace a large part of their
generation capacity in the coming 20 years
• Will this enable a transition towards sustainable generation?
• ElecTrans Model
• Agent based model of the Dutch system
• Covers generator companies, network companies, and users
• Uncertainties
• Operational costs of options
• Investment costs
• Planning horizon
• Desired Return on investment
• Various demand developments
33
Results
34
EMA AND SA
35
Sensitivity Analysis vs. EMA
Sensitivity analysis (SA) is the study of how the variation in the output
of a mathematical model can be apportioned, qualitatively or
quantitatively, to different sources of variation in the input of the
model.*
• EMA is interested in exploring the behavior of a modeled system
across a wide variety of uncertainties to
• determine modes of behavior
• support the development of adaptive robust strategies
• provide insight into the combinatorial effects of the uncertainties
• EMA and SA have a different purpose
• EMA not only interested in model inputs, but also structural or model
paradigmatic variations
• EMA is directly related to supporting policy development
36
*Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D. Saisana, M., and Tarantola, S., 2008, Global
Sensitivity Analysis. The Primer, John Wiley & Sons.