Transcript Getting started

Getting started with GEM-SA
This talk
Starting GEM-SA program
Creating input and output files
Explanation of the menus, toolbars, etc.
Description of the project window
GEM-SA course - session 3
2
Starting GEM-SA
Double-click the GEM-SA icon to start
The main window appears, with
Menu
Toolbar
Main results area with three tabs
Sensitivity Analysis, Main Effects and Results Summary
Initially all empty
Log window
GEM-SA course - session 3
3
The main GEM-SA window
menu
toolbar
Sensitivity analysis
output grid
Log window
GEM-SA course - session 3
4
Toolbar icons
New project
Open project
Save project
Print output report
Edit project
Generate input design
points
Rescale an input
Standardise design
Copy input design to
clipboard
Convert input to integer
Run the analysis
Help
GEM-SA course - session 3
5
Output tabs
When an emulator has been fitted, the contents of
these tabs will provide the main results
Sensitivity Analysis. This will report the SA
variance decompositions
One line for each input parameter
One line for each pair of inputs, if joint effects are
selected
Main effects. This will plot the main effects of the
various inputs
Results Summary. This will present numerical
summaries of emulator fit and uncertainty analysis
GEM-SA course - session 3
6
Log Window output
Tells us
Which training data are being loaded/saved
Transformations applied to the data
Fitted Gaussian process parameters
Summary of cross-validation analysis
Summary of the uncertainty analysis
GEM-SA course - session 3
7
Creating a GEM project
To build the emulator we first need 3 files:
Data file of code inputs
Data file of code outputs
GEM-SA project file
GEM-SA course - session 3
8
Restrictions on input/output data
Single output
Multiple outputs must be treated individually
GEM can read multiple outputs file, but a single
column is specified within a project
Max 30 input parameters
Max 400 training points
The data files are plain text files
One row for each point
Rows can be space or tab delimited
GEM-SA course - session 3
9
Generating a new input design
Designs can be generated using the toolbar icon
or the menu: Input  Generate…
The design dialog appears
GEM-SA course - session 3
10
Generating a new input design
Click OK and fill in the required range for each
input
Click OK again
GEM-SA course - session 3
11
Editing input designs
If you select a column, you can rescale values of that
input
or round values to be integers
Designs can be loaded into or saved from this window
using the Inputs menu. Use
to copy the points to the
clipboard for use in other programs
GEM-SA course - session 3
12
Types of design
GEM-SA can generate 2 types of design
LP-
Maximin Latin Hypercube designs
Both have good space-filling properties
Ensure all regions of the input space are well
represented
GEM-SA course - session 3
13
LP- design
Very quick to generate
Deterministic set of uniform points
Increasing the sample size just adds points to
the smaller design
Making it useful for sequential analysis
Only have to generate the extra runs
GEM-SA course - session 3
14
Maximin Latin hypercube design
Maximin Latin Hypercube designs
Maximise the minimum distance amongst all pairs of
points
Can take a long time to generate
Projections also generally space-filling
Lower dimensional projections are also latin
hypercubes
Good when only a few inputs are active
GEM-SA course - session 3
15
Creating output points
Each row from the input design must be used to
generate outputs by running the computer code
One run for each row
Various methods to automate this:
Spreadsheet
Simple, but requires functional form
Script
Only need executable code
Loop through inputs, modify code input file
Modify code to loop through the points
Messy, need source code
GEM-SA course - session 3
16
Example: using a spreadsheet
Copy the input design to
the clipboard using
Open Excel and paste
inputs
Create formula in final
column
Copy formula for all rows
of the design
Cut and paste special
(values) in a new sheet
Save as text file
GEM-SA course - session 3
17
Example: using a script
Read simulator’s base input file
Read training inputs file
Loop through training file lines
Replace target inputs using training line
Write new base input file
Run code
Calculate output(s) and add to training output file
GEM-SA course - session 3
18
my $pftchangeline = 21; # change line 21 within the input file for each run
my @pftchangecols = (11,14,23,19); # columns within pftchangeline to modify
my @pftinlh = (0,1,2,3); # ordering of these parameters within training inputs
open(BASEINFILE, "input.dat"); # getinitial (fixed) input file used by sdgvmd
my @lines = <BASEINFILE>; # and store the input lines in @lines
close BASEINFILE;
open(LHFILE, "training_inputs.txt");
my $newpftline = $lines[$pftchangeline];
my @newpftpoints = split(" ", $newpftline);
while (<LHFILE>){
# assigns each line in turn to $_
chomp;
split;
my @lhpoints = @_;
open(INFILE, "> inputfile.dat");
@newpftpoints[@pftchangecols] = @lhpoints[@pftinlh] # modify lines
$lines[$pftchangeline] = join(' ', @newpftpoints)."\n";
print INFILE @lines;
close INFILE;
`sdgvm0 input.dat`; # run sdgvm0 with modified input
# now do something with the output files....
...
}
GEM-SA course - session 3
19
The project window
Appears whenever you
Load a project
Edit a project
Create a new project
This window also has 3 tabs
Files
Options
Simulations
GEM-SA course - session 3
20
Names for
the input
files
Names for
the output
files
GEM-SA course - session 3
21
How many
inputs?
What are
the input
names?
Which
column
from
output file?
GEM-SA course - session 3
22
What
should be
calculated,
and how?
Which joint
effects
should be
calculated?
GEM-SA course - session 3
23
What prior
mean for
the output?
How are
the inputs
uncertain?
GEM-SA course - session 3
24
What kind of
prediction?
What kind of cross
validation?
GEM-SA course - session 3
25
MCMC
control
parameters
How many realisations
of predictions, main
and joint effects to
generate
How many points
used to calculate
main effects, joint
effects
GEM-SA course - session 3
26
The options tab
Input parameter names
This window appears if you press the Names…
button
Giving names is optional, but useful later when
looking at GEM-SA output
Ordering can be changed using the arrows
GEM-SA course - session 3
28
Selecting joint effects
Select calculate joint effects if in sensitivity analysis you
want to see the joint effects (interactions) of pairs of
inputs as well as their individual effects
Use Inputs to include in joint effects panel to select which ones
Default All inputs computes joint effects for all pairs
Can take a lot of computation
To compute only the joint effects between selected inputs,
deselect All inputs and select the two or more inputs
whose joint effects are of interest
GEM-SA course - session 3
29
Other checkboxes
Sum effects
There are two ways to plot the joint effect of two
inputs:
A combined effect in which the value plotted is the mean
output value at that combination of input values
A pure interaction, in which with the main effects of those
inputs are subtracted from the combined effect
Select sum effects if you want to see combined
effects, and deselect it to see interactions
This selection is ignored if you don’t ask for joint
effects to be computed
GEM-SA course - session 3
30
Other checkboxes
Code has numerical error
We generally assume that the model output is computed exactly
every time
So the meta-model passes exactly through all the training points
There are two situations in which this assumption is not right
Your code has numerical errors which you want to smooth out
Your code is stochastic and the output values have random noise
Selecting code has numerical error turns the assumption off
The variance of the error will be estimated as part of the fitting
process
The meta-model will smooth out the training points to a degree
depending on the estimated error variance
Can make the fitting process quite unstable, so beware!
GEM-SA course - session 3
31
Other checkboxes
Use MCMC for emulator parameters
By default, GEM-SA estimates the underlying
smoothness parameters and then pretends that the
estimates are exact
Selecting use MCMC for emulator parameters takes
into account uncertainty in the fitting of the emulator
Slows down the computation substantially, often with
minimal effect on the results
Auto-tune Metropolis algorithm
Use only with MCMC
If not selected, you must supply a tuning file
GEM-SA course - session 3
32
Input uncertainty options
These options are for specifying what kind of
distribution each uncertain input has
There are a limited range of options
All unknown, product normal/uniform
Inputs are independent, with either normal or uniform
distributions
All known
No uncertainty analysis required
Some known, rest product normal/uniform
Some input values will be fixed (in the dialog window or in a
prediction file)
Others will be given independent distributions, either normal
or uniform
GEM-SA course - session 3
33
Input uniform ranges
If you say that some or all have uniform distributions, a
window appears (when you click OK) to specify ranges
Option to use ranges in input data file
All uniform
Some fixed, rest uniform
GEM-SA course - session 3
34
Input normal parameters
If you say that some or all have normal distributions, a
window appears (when you click OK) to specify the
mean and variance of each distribution
Option to use ranges in input data file
All normal
Some fixed, rest normal
GEM-SA course - session 3
35
Prior mean options
The emulator will fit better if it knows roughly
how the output is expected to respond to the
inputs
You have just two choices
If you expect to see a trend in the output in response
to changes in its inputs, select linear term for each
input
Otherwise, selecting constant mean results in no
overall trends being expected or fitted
GEM-SA course - session 3
36
Selecting prediction type
Having fitted the Gaussian process emulator, GEM-SA
can predict what the output would be if the computer
code were run at new input sets
These are specified in a prediction file
If there is no prediction file, selecting the prediction type has no
effect
Predictions can be
Simulated realisations of outputs at the prediction inputs
Similar to main effect outputs
Takes account of correlation between predictions
Marginal means and variances of outputs at the prediction inputs
Faster to compute, especially with many prediction points
Easy to interpret
GEM-SA course - session 3
37
Selecting cross validation type
Cross-validation is a way of checking the validity of the
predictions made by GEM-SA
The idea is to fit the emulator leaving out some of the training
data points, then predict the missing points and see how well the
predictions do
Choice of none, leave-one-out or leave final 20% out
Leave-one-out
Hyper-parameters use all data and are then fixed when
prediction is carried out for each omitted point
Leave final 20% out
Hyper-parameters are estimated using the reduced (80%) data
subset
GEM-SA course - session 3
38
The files and simulations tabs
GEM-SA files
You always have to specify an Inputs File and an
Outputs File
You only need to specify a Prediction Inputs File if you want to
generate predictions
You only need to specify a Metropolis-Hastings Tuning File if you
select MCMC for computation and deselect auto-tuning
The Main effects file will always be created when you do
sensitivity analysis
The Joint Effects file will be created if you ask for joint effects to
be computed
The Predictions File will be created if you ask for predictions (by
specifying a Prediction Inputs File)
It will contain simulated predictions or prediction means
The Predictions Variance File is created if you ask for predictions
and specify prediction means and variances
GEM-SA course - session 3
40
Simulations
The first three of these settings apply only if you
select MCMC computation
For expert users only!
You could choose the number of simulations that
are computed for each main effect and
interaction
But the default is generally plenty
You might want to increase the number of points
on each main effect
To get more detail in the plots
But at the cost of longer computations
GEM-SA course - session 3
41