Transcript Using Formal Models of Utility to Guide the Chris Johnson

Using Formal Models of Utility to Guide the
Development of Safety-Critical Systems
Chris Johnson
University of Glasgow, Scotland.
http://www.dcs.gla.ac.uk/~johnson
1
Can PRA Guide Formal Methods?
2
Very High:
Failure is
almost
inevitable
1 in 2
10
1 in 3
9
High: Repeated
failures
1 in 8
8
1 in 20
7
Moderate:
1 in 80
6
Occasional
failures
1 in 400
5
1 in 2000
4
Low:
1 in 15,000
3
Relatively few
failures
1 in 150,000
2
Remote: Failure
is unlikely
1 in 1,500,000
1
Hazardous Very high severity ranking when a potential
without
failure mode affects safe operation or involves
warning
non-compliance with a government regulation
without warning.
10
Hazardous Failure affects safe product operation or involves
with
non-compliance with government regulation with
warning
warning.
9
Very High Product is inoperable with loss of primary
Function.
8
High
Product is operable, but at reduced level of
performance.
7
Moderate
Product is operable, but comfort or convenience
item(s) are inoperable.
6
Low
Product is operable, but comfort or convenience
item(s) operate at a reduced level of
performance.
5
Absolute
Design Control does not detect a potential Cause of
Uncertainty failure or subsequent Failure Mode; or there is no
Design Control
10
Very
Remote
Very remote chance the Design Control will detect a
potential Cause of failure or subsequent Failure Mode
9
Remote
Remote chance the Design Control will detect a
potential Cause of failure or subsequent Failure Mode
8
Very Low
Very low chance the Design Control will detect a
potential Cause of failure or subsequent Failure Mode
7
Very Low Fit & finish or squeak & rattle item does not
conform. Most customers notice defect.
4
Low
Low chance the Design Control will detect a
potential Cause of failure or subsequent Failure Mode
6
Minor
Fit & finish or squeak & rattle item does not
conform. Average customers notice defect.
3
Moderate
Moderate chance the Design Control will detect a
potential Cause of failure or subsequent Failure Mode
5
Very
Minor
Fit & finish or squeak & rattle item does not
conform. Discriminating customers notice defect.
2
4
None
No effect
1
Moderately Moderately high chance the Design Control will
High
detect a potential Cause of failure or subsequent
Failure Mode
High
High chance the Design Control will detect a
potential Cause of failure or subsequent Failure Mode
3
Very High
Very high chance the Design Control will detect a
potential Cause of failure or subsequent Failure Mode
2
Almost
Certain
Design Control will almost certainly detect a
potential Cause of failure or subsequent Failure Mode
1
3
Classical Decision Theory
(f1,p1;f2,p2;…;fn,pn)
 s  S: V(s) = (ni=1 pi).u(f1f2…fn)
 s, s1  S: s1 risk s  V(s) > V(s1).
4
Applications of Classical Decision Theory
pump_v(exchanger_error, 0.0000000003, bbv_error, 0.00000241) risk
analyser_v(analyser_error, 0.0000000003)
compound_failure 
exchanger_error  bbv_error 
AX(display_exchanger_warning  display_bbv_warning)
ordered_response_failure 
compound_failure  analyser_failure 
AX(start_standby_pump 
EF(display_analyser_warning  reroute_analysis))
5
Decision Theory and Formal Methods
so |= AX (f ; y) iff:
#{ Fk  F| (s0, s1)  Fk  s1 |= f  Fk} = y
#{ Fj  F| (s0, s1)  Fj  Fj}
F
so |= AX [f P y] iff:
#{ sx  P| sx |= f  sx} = y
#{ sx  P  sx}
S0
P
6
RPN Paradoxes
7
Decision Theory and Formal Methods
• Issues with probability:
– limited incident data;
– relational databases;
– poor interpretation.
8
Why Bother with Utility?
y
0
x
Point of diminishing
returns
Expenditure
Expected marginal utility of resources
Expected marginal utility of resources
y
x
0
Point of diminishing
returns
Expenditure
9
Why Bother with Utility?
X2
1
2
1
2
0
X1
10
Why Bother with Utility?
4000
3500
3000
2500
Total cost
2000
Euros
1500
Cost of failure
1000
Cost of
maintenance
500
0
1
3
5
7
9
11
13
15
17
Maintenance Interval
(Months)
19
21
23
H. Kortner and A. Kjellsen, Det Norske Veritas - 2000.
11
Why Bother with Utility
Attribute A
(eg ride quality)
Attribute B (eg speed)
12
Standard Models of Utility
• Users have a consumption set X.
• Trade-offs exist between elements of X:
• There are preference relations over X:
– (x1, x2)  
“x1 is at least as good as x2”
• Axioms avoid paradoxes & define “rationality”.
13
Rationality Axiom 1: Completeness
• For any x1 x2X either x1  x2 or x1  x2
• Implication 1. The Completeness Axiom
makes an unrealistic assumption that designers
will be able to distinguish between the different
strategies or plans that they can exploit.
14
Rationality Axiom 2: Reflexivity
• For all xX x  x
• Implication 2 The Reflexivity Axiom states that any
alternative is at least as good as itself but designers
may associate different values with different means
of obtaining the same outcome.
15
Rationality Axiom 3: Transitivity
• For any x1,x2, x3 X
if x1  x2 and x2  x3 then x1  x3
• Implication 3 The Transitivity Axiom makes an
unrealistic assumption that users act as “rational”
consumers in a technical environment that they
may not fully understand.
16
Preference Topologies
• Definition 1 ( preference):
constrained to satisfy rationality axioms.
• Definition 2 (>> strict preference):
x1 >> x2 iff x1  x2 and (x2  x1)
• Definition 3 (~ indifference):
x1 ~ x2 iff x1  x2 and x2  x1
17
Preference Topologies
• For some point x0 = (x01, x02):
•
•
•
•
•
At least as good as: {x | x  X, x  x0}.
No better than: {x | x  X, x0  x}
Worse than: {x | x  X, x0 >> x}
Preferred:{x | x  X, x >> x0}
Indifferent: {x | x  X, x ~ x0}
18
Preference Topologies
Quantity Preferred
X2
- Shows X as a 2D vector of reals.
- Paradox to left of x0
- So introduce additional axioms.
>>(x0)
~(x0)
x0
<<(x0)
X1
0
Quantity Preferred
19
Axioms of Taste: Continuity
• For all x  Rn both  and  are closed.
• Implication 4 The Continuity Axiom
ensures topological nicety and is neutral
with respect to safety-critical development.
20
Axioms of Taste: Strict Monotonicity
• For all x0, x1  Rn+ if x0 is greater than or
equal to x1 then x0  x1 while if x0 is
strictly greater than x1 then x0 >> x1.
• Implication 5 The Axiom of Strict Monotonicity fails to
characterise certain aspects of safety-critical
development in which more of a resource can yield a
worse design.
21
Axioms of Taste: Strict Monotonicity
Continuity reduces indifference region.
Monotonicity ensures all preferred sets
are strictly above indifference sets
(non-satiation).
X2
Quantity Preferred
x1
0
x
>>(x0)
t
x2
x0
<<(x0)
X1
Quantity Preferred
22
Axioms of Taste: Strict Convexity
• If x1x0 and x1x0 then tx1+(1-t)x0 >> x0 for
all t[0, 1]
• Implication 6 The Axiom of Strict Convexity
reflects a “balanced” approach to resource
allocation or substitution. As one of the preference
axioms of taste, however, it is inappropriate for all
forms of safety-critical development.
23
The Way Forward
• Perhaps I’m missing the point.
• Quantitative analysis unimportant?
• But we keep getting PRA wrong:
– Formal methods might help?
24
Wider Conclusions
• Question use of convex utility curves:
– in risk analysis and decision theory;
– (in stochastic multiplexing; caching etc.)
• Things are more complex than I thought:
– subjectivity; ceteris paribus; risk homeostasis.
25
26
National Attitudes to Risk
3.4
3.2
3
2.8
2.6
2.4
France Germany
Italy
Portugal
UK
27
Caveat
• Preference relation orders the consumption set, X.
• Utility functions map preferences onto numeric scale.
• Utility functions “inherit” complete, reflexive,
transitive, continuous and strictly monotonic
properties.
• Time to examine these assumptions...
28