Transcript No Slide Title

Realistic Uncertainty Bounds
for Complex Dynamic Models
Andrew Packard, Michael Frenklach
CTS-0113985
January 2004
Our research focuses on the benefits of
treating models/data pairs as assertions,
that can be shared and reasoned with
using automated algorithms.
Message: Use collaboration through
model/data sharing and automated
reasoning to extract the totality of
information in the community data sets.
Reasoning on Collections of assertions
• test for consistency
• inconsistency falsifies at least one
• sensitivity of consistency to data
• which are likely false?
• infer additional implications from assertions
•sensitivity of inferred conclusions to data
•which assertions have the most impact?
We have developed a formalism involving
assertions expressed as polynomial
inequalities on a parameter space. We
use global optimization methods,
developed in control systems analysis,
with origins in algebraic geometry. The
GRI-Mech DataSet fits in this framework.
We have carried out extensive novel
analysis described here.
Michael Frenklach, Andrew Packard, Pete Seiler and Ryan Feeley, “Collaborative data processing in developing predictive
models of complex reaction systems,” International Journal of Chemical Kinetics, vol. 36, issue 1, pp. 57-66, 2004.
Michael Frenklach, Andy Packard and Pete Seiler, “Prediction uncertainty from models and data,” 2002 American Control
Conference, pp. 4135-4140, Anchorage, Alaska, May 8-10, 2002.
Pete Seiler, Michael Frenklach, Andrew Packard and Ryan Feeley, “Numerical approaches for developing predictive
models,” submitted to Engineering Optimization, Kluwer December 2003.
Ryan Feeley, Pete Seiler, Andy Packard and Michael Frenklach, “Consistency of a reaction data set,” in preparation.
Copyright 2004, Packard, Frenklach. This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this license, visit
http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.
GRI DataSet
The GRIMech (www.me.berkeley.edu/gri_mech) DataSet is collection of 77 experimental reports, consisting
of models and ``raw'' measurement data, compiled/arranged towards obtaining a complete mechanism for
CH4 + 2O2 → 2H2O + CO2 capable of accurately predicting pollutant formation. The DataSet consists of:
• Reaction model: 53 chemical species, 325 reactions.
• Processes (Pi): 77 widely trusted, high-quality laboratory experiments, all involving methane combustion, but under different
physical manifestations, and different conditions.
• Measured Data (di,ui) data and measurement uncertainty from 77 peer-reviewed papers reporting above experiments.
• Unknown parameters (r): 102 normalized parameters, typically derived from rate constants.
• Prior Information: Each normalized parameter is known to lie between -1 and 1.
• Process Models: 77 1-d and 2-d numerical PDE models, coupled with the reaction model.
• Surrogate (reduced) Models (Si): 77 polynomials in 102 variables.
d1  u1
Chemistry(r)
Transport 1
S1(r)
Chemistry(r)
Transport 2
S2(r)
Transport 77
S77(r)
Process P1
CH4 + 2O2
⇅
2H20 + C02
Process
P2
d2  u2
Process
P77
300+ Reactions,
53 Species, and
102 “Active”
Parameters
d77  u77
The prior information, models and
measured data are assertions about
possible parameter values.
⋮
Chemistry(r)
•k’th assertion associated with prior info: rk 1  0
2
•Assertions associated with i’th dataset unit:
Si (r)  u i  di  0,
d i  u i  Si (r)  0
Consistency of GRI-Mech DataSet
Is there a value of r which satisfies all of the assertions? This
question constitutes a “consistency” analysis.
For simplicity and in light of insufficient records of experimental
uncertainties (ui) even for such a well-documented case as GRIMech, artificial but realistic uniform levels of experimental
uncertainties were used.
•At ui=0.1, the GRI-Mech Data Set is consistent, in that local
optimization readily generates 102-dimensional r satisfying all
assertions. In fact, this can be accomplished for u i as low as 0.087.
•At ui=0.083, the data set is inconsistent, in that no r exists to satisfy
all assertions! How/why? Using semidefinite programming, we find
nonnegative scalars i, i, and k such that the polynomial function
77
77
102
i 1
i 1
k 1

Q( r ) : 1   i di  ui  Si ( r )   i Si ( r )  ui  di     k 1  r k2

is the negative of the sum of squares of other polynomials. Hence,
the quantities inside the bracketed terms (which are asserted to be
nonnegative) cannot all be made nonnegative by any one value of r.
This certifies that for every value of r, at least one assertion is
violated.
What is the sensitivity of the above conclusions to the numeric values within the
assertions (e.g., measured data and prior bounds on parameters)?
Sensitivity of DataSet Consistency to Assertions
The top two panels of the accompanying
figure show the sensitivity of the consistency
measure to the bounds on the 102 active
parameters. The numbers indicate the
consistency of the dataset is almost
unaffected by the prior assumptions on the
unknown parameters (i.e., the -1 and 1
values for the bounds). Note units (and
compare below).
The consistency measure is most sensitive
to two specific experimental reports (57 and
58). Panel lower-left shows the sensitivity of
the consistency measure to reported lower
bound of measurement value. Panel lowerright shows the sensitivity of the consistency
measure to reported upper bound of
measurement value. Note the units -nonzero bars in the lower panels are 2
orders of magnitude larger than those in top
panels.
3
*
( )
x 10
-3
3
2
*
( )
1
0
0
40
60
80
100
20
Parameter number
*
(l)
0.4
0.3
0.2
0.3
(u)
* 0.2
0.1
0.1
0
0
20
40
60
40
60
80
100
Parameter number
0.4
Dataset unit number
-3
2
1
20
x 10
20
40
60
Dataset unit number
Large values are suggestive that some data
should be reexamined…
Upon notification (but no other details) that our automatic tools raised flags, the scientists involved in (57)
and (58) rechecked calculations, and concluded that reporting errors had been made. Both reports were
updated -- one measurement value increased, one decreased -- exactly what the consistency analysis had
suggested.
High price of low cost, noncollaborative data processing
Computational exercise: assess capability of 76 assertions in
predicting the outcome of the 77th model. Quantify
information “lost” by doing more conservative analysis without
the benefit of collaboration at the raw data level.
How much information is lost when one resorts to method C
instead of A? Define the ``loss in using method C'' as
Select one of the 77 models, “treat” it as a process whose
outcome is to be predicted, using the other 76
model/measured data pairs (assertions) as information. The
prediction is an interval -- the min and max values that the
predicted process's outcome could take on, inferred from the
76 assertions.
No loss (LC=0) occurs if prediction by C is as tight as that
achieved by A. Complete loss (LC=1) occurs if prediction by
C is no better than method P (only using prior info). In such
case, the experimental results are effectively wasted. The
statistics of the loss variables (over 77 cases) are
Method P: Use only the prior information on parameters;
gives an interval whose length is denoted P (meaning “prior”).
Method C: Community “pools” prior information and 76
assertions, but for simplicity, chooses only to reduce
parameter uncertainty, maintaining a cube description (i.e.,
intervals for each parameter -- easy to publish, share and
think about). That computation requires a one-time
collaboration with the assertions. The new, improved (i.e.,
smaller) parameter cube will allow future predictions to be
more constrained. The length of the prediction interval on the
77th model using the new improved cube is denoted C
(community).
Method A: The prediction directly uses the raw model/data
pairs from all 76 experiments, as well as the prior information.
The length of the prediction interval (exactly what we want to
compute) is denoted A.
Repeat computation 77 times, interchanging which GRI-Mech
model plays the role of the ``to-be-predicted-process'' while
the remaining 76 assertions are used as the information.
LC 
1
CA
PA
Fraction of Experiments with Loss  x
0.9
0.8
0.7
0.6
0.5
0.4
E.g., In 40% of the computational exercises
the loss was greater than 0.81.
0.3
0.2
Frequency of Loss
0.1
0
0
0.2
0.4
Loss (LC)
0.6
0.8
1
Method C pays a significant price for its lack of collaboration
and crude representation of the information in the assertions.
This illustrates the need (and payoff) for fully collaborative
environments in which models and data can be shared,
allowing sophisticated global optimization-based tools to
reason quantitatively with the community information.