Transcript Uncertainty Surrounding the Cone of Uncertainty Todd Little IEEE Software, May/June Managing the Coming Storm Inside the Tornado Project Kickoff When will we get the requirements? All.

Uncertainty
Surrounding the
Cone of Uncertainty
Todd Little
IEEE Software, May/June
2006
Managing the Coming Storm
Inside the Tornado
Project Kickoff
When will we get the requirements?
All in good time, my little pretty, all in good time
But I guess it doesn't matter anyway
Just give me your estimates by this afternoon
Team Unity
Not so fast! Not so fast! ... I'll have to give the matter a little
thought. Go away and come back tomorrow
No, we need something today!
Ok then, it will take 2 years.
No, we need it sooner.
Doesn't anybody believe me?
I already promised the customer it will be out in 6 months
You're a very bad man!
We’re not in Kansas
Anymore
Developer Hero
I may not come out alive, but I'm goin' in there!
Reorg
The Great and Powerful Oz has got matters well in hand.
My! People come and go so quickly here!
Testing
"Hee hee hee ha ha! Going so soon? I wouldn't hear
of it! Why, my little party's just beginning!
The Cone of Uncertainty
(Boehm)
10
Minimum
Maximum
2
1
0.5
Cone of Uncertainty: The amount of possible error in a
software project estimate, which is very large in the
early stages of a project and shrinks dramatically as
the project nears completion – Steve McConnell
0.1
Feasibility
Concept of
Operation
Requirements
Spec
Product
Design Spec
Detail Design
Spec
Accepted
Software
Hurricane Rita
Landmark Product Suite
Reservoir /
Fluid data
Production data
Seismic data
Structural /
Stratigraphic data
Velocity data
Common Model
Representation
Well data
Data from LGC
Initial Estimate vs. Actual Project Duration (from LGC Portfolio Database)
1200
1000
y = 1.6886x
Actual
800
600
LGC Data
Ideal
400
Linear (LGC Data)
200
0
0
100
200
300
400
500
Initial Estimate
600
700
800
900
1000
Data from Tom DeMarco
It’s déjà vu all over again
Industry data from Tom DeMarco
1200
1000
Actual Effort
800
600
Actual
400
2X
Ideal
200
0
0
100
200
300
400
500
Estimated Effort
600
700
800
900
1000
Cumulative Distribution Curve
for Actual/Estimate (DeMarco)
DeMarco Cumulative Distribution Function
1
0.9
0.8
CDF Probability
0.7
0.6
p(10)
p(50)
p(90)
0.5
0.79
1.74
3.81
0.4
Data
Log Normal Curve
0.3
0.2
0.1
0
0.1
1
Ratio
10
CDF Distribution Curve
(LGC)
Landmark Graphics Cumulative Distribution of Portfolio Projects
1
0.9
Cummulative Probability
0.8
0.7
0.6
p(10)
p(50)
p(90)
0.5
0.96
1.76
3.23
0.4
0.3
0.2
0.1
0
0.1
1
Ratio of Actual to Estimate
10
Probability Distribution
Curve
Distribution Curve of Actual/Estimated (DeMarco data vs. LGC)
(Demarco data is Effort/Effort; LGC data is Duration/Duration)
2.5
2
Frequency
1.5
DeMarco
LGC
1
0.5
0
0
1
2
3
4
(Actual/Estimated)
5
6
7
8
Log Normal Distribution
 Estimation Accuracy follows a Log Normal
distribution
Median
Mean
0.1
1
10
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
“It’s tough to make predictions,
especially about the future.”
Yogi Berra, Niels Bohr
That idea is so damned nonsensical and impossible
that I'm willing to stand on the bridge of a
battleship while that nitwit tries to hit it from the
air.
Newton Baker, U.S.
secretary of war in
1921, reacting to the
claim of Billy Mitchell
(later Brigadier
General Mitchell) that
airplanes could sink
battleships by
dropping bombs on
them.
“It’s tough to make predictions,
especially about the future.”
Yogi Berra, Niels Bohr
Heavier-than-air flying machines are impossible.
Lord Kelvin, British mathematician, physicist, and
president of the British Royal Society, spoken in 1895.
How does Estimation Accuracy
Improve Over Time? (Boehm)
10
Minimum
Maximum
2
1
0.5
0.1
Feasibility
Concept of
Operation
Requirements
Product
Detail Design
Spec
Design Spec
Spec
Accepted
Software
So what does LGC data
look like?
1200
1000
800
Ideal
From Start
Post Env
Post Plan
Post Dev
600
400
200
0
0
100
200
300
400
500
600
Landmark Cone of
Uncertainty
But is Uncertainty Really
Reduced?
“Take away an ordinary person’s illusions and
you take away happiness at the same time.”
Henrik Ibsen--Villanden
Cumulative Distribution
(CDF) Curve
1
0.9
0.8
0.7
0.6
Initial
Post Env
0.5
Post Plan
Post Dev
Log Normal
0.4
0.3
0.2
0.1
0
0.1
1
10
100
Remaining Uncertainty
The Pipe of Uncertainty
10
Minimum
Maximum
2
1
0.5
0.1
Envisioning
Planning
Developing
Stabilizing
Does Landmark Suck at
Estimation?
A severe depression like that of 1920-21 is outside
the range of probability.
Harvard Economic Society, Weekly Letter, November
16, 1929.
Does Landmark Suck at
Estimation?
I think there is a world market for about five
computers.
Thomas J. Watson, chairman of IBM, 1943.
Does Landmark Suck at
Estimation?
They couldn't hit an elephant at this dist…
General John B. Sedgwick, Union Army Civil War
officer's last words, uttered during the Battle of
Spotsylvania, 1864
Estimation Quality Factor
(EQF)
Value to be Estimated
Actual Value
Blue Area
EQF =
Initial Estimate
Red Area
Actual
End Date
Elapsed Time
Link to article by Tim Lister
EQF from Lister/DeMarco
EQF
Comment
1.8
Typical disaster project
3.8
Median of Medians
5.0
Pretty Good
8.0
Excellent sustained Median EQF
10.0
Not Sustained
EQF Distribution Curve
(LGC)
Landmark Graphics Cumulative Distribution of EQF
1
0.8
CDF/Prob
0.6
EQF for duration
has a theoretical
minimum of 2.0
p(10)
p(50)
p(90)
0.4
2.80
4.78
11.68
0.2
Raw EQF
LogNormal
0
0.1
1
10
-0.2
EQF
100
We slip one day at a time,
EQF=2
Value to be Estimated
Actual Value
Blue Area
EQF =
Initial Estimate
Elapsed Time
Red Area
Actual
End Date
(EQF-2) Distribution
Curve (LGC data)
Landmark Graphics Cumulative Distribution of EQF
1
0.9
0.8
0.7
CDF/Prob
0.6
p(10)
p(50)
p(90)
0.5
2.80
4.78
11.68
0.4
EQF Data - 2
0.3
Log Normal Curve
0.2
0.1
0
0.1
1
10
EQF
100
From the home office in
Duncan, Oklahoma Dubai, UAE
Top Ten reasons why we are late in
2008
Top Ten reasons why we
are late in 2008
10: Requirements, what Requirements?
What you want, baby I
got it
R-E-Q-U-I-R-E
Find out what it
means to me
Top Ten reasons why we
are late in 2008
9: Dependencies on other groups that were late
Top Ten reasons why we
are late in 2008
8: Over-optimistic Schedule Estimation
Always look on the bright side of code
.......
Always look on the bright side of code
.......
The code’s a piece of $#!^,
when we look at it
We can always overlook a minor kink . . .
It probably compiles, it might even link . .
Surely that must mean it doesn’t stink
Top Ten reasons why we
are late in 2008
7: Those weren’t MY estimates
Scheduling Ritual
How low can you go!
Top Ten reasons why we
are late in 2008
6: Not enough testers or documentation
resources.
Who needs them anyway? We
put those bugs--I mean features-in there on purpose. Besides, it
was difficult to program, it should
be difficult to use.
Top Ten reasons why we
are late in 2008
5: Offshore and Outsourcing issues
My source code lies over the ocean,
My source code lies over the sea .
My source code lies over the ocean,
Oh bring back my source code to me
.....
Bring Back, Bring Back,
oh bring back my source code to me, to me
Bring Back, Bring Back,
oh bring back my source code to me
Top Ten reasons why we are
late in 2008
4: One word, Ch-ch-ch-changes
Top Ten reasons why we
are late in 2008
3: I can’t get no, System Admin





I can’t get no, CM action
‘cause I try,
..and I try,
….and I try,
……and I try….
Top Ten reasons why we
are late in 2008
2: You didn’t give me the headcount
that you promised
Top Ten reasons why we
are late in 2008
1: Weren’t you doing the backups!?
You can get better
How to get better




Start Measuring
Understand bias
Collaborate
Manage Uncertainty
Start Measuring
Understand Bias
 "What gets us into trouble is not what we don't
know. It's what we know for sure that just ain't
so.“
» Mark Twain
Uncertainty
Know that
we know
Know that
we don’t know
Don’t know that
we know
Knowable
Don’t know that
we don’t know
Unknowable
Uncertainty
Know that
Planning
we know
Know that
Risk Management
we don’t know
p10
p50
Don’t know that
we know
Knowable
Don’t
Uncertainty
know that
we
Management
don’t know
Unknowable
p90
Just Double the Estimate
100%
90%
80%
70%
CDF
60%
50%
40%
30%
20%
10%
0%
0.1
1
Actual/Estimate Ratio
10
Defending an Unpopular
Schedule
 Developers tend to be temperamentally opposed to the use of
negotiating tricks. Such tricks offend their sense of technical
accuracy and fair play. Developers don't want to offer
lopsidedly high initial estimates even when they know that
customers, marketers, or bosses will start with lopsidedly
low bargaining positions.
 Steve McConnell
http://www.stevemcconnell.com/ieeesoftware/bp03.htm
We want this
Negotiation Bias
 "It is difficult to get a man to understand
something when his salary depends upon his not
understanding it.“
» Upton Sinclair:
Trust: The Cooperative or
Non-Cooperative Game?
s
Customer perception
Customer
perception
of Team's ability to
ofDeliver
Team's ability to
Deliver
R
Team’s
ability
Team’s
ability
to to
Deliver
against
targets
Deliver
against
targets
o
o
Desire to set
aggressive targets
Test 1 (Jørgensen)
Group
Guidance
A
800
B
40
C
4
D
None
Result
160
Test 1
Group
Guidance
Result
A
800
300
B
40
100
C
4
60
D
None
160
Test 2
Group
Guidance
Result
A
Minor
Extension
New
Functionality
Extension
50
B
C
Test 2
Group
Guidance
Result
A
Minor
Extension
New
Functionality
Extension
40
B
C
80
50
Test 3
Group
A
B
Guidance
Future work at
stake, efficiency
will be measured
Control
Result
100
Test 3
Group
A
B
Guidance
Future work at
stake, efficiency
will be measured
Control
Result
40
100
Collaborate: Planning Poker
 Collaborative Estimation
Activity
 Modified Fibonacci Sequence
 0,½,1,2,3,5,8,13,20,40,100,∞,?
 Interactive version at
www.planningpoker.com
Exercise
Low
Distance from Las Vegas to
Houston, Texas
Height of the Empire State
Building
Population of Sweden
U.S. Oil Consumption/day
Water in a 100 gallon vat filled
with sand
Med High
Exercise
Low Med
Distance from Las Vegas to
Houston, Texas
Height of the Empire State
Building
Population of Sweden
1222
U.S. Oil Consumption/day
20MM
Water in a 100 gallon vat filled
with sand
35
1453
9MM
High
Managing for Uncertainty
 We expect uncertainty and manage for it
through iterations, anticipation, and
adaptation.
The Cone of Uncertainty
Uncertainty
Target
Best possible scenario if
everything went perfectly.
Value
Plan
Planned scope for the
release at the optimal time
that it can be released.
Contract
Minimum scope for the
release at the latest date
that it can be released.
Tim e
The A/B/C List sets proper
expectations
A
MUST be completed in order to ship the product.
B
SHOULD be completed in order to ship the
product.
MAY be completed prior to shipping the product
C
if time allows.
Only “A” features may be committed to customers.
“A” features must fit in a p90 confidence schedule. No more
than 50% of the planned effort can be allocated to “A” items
A/B/C List
50%
25%
25%
B
C D
A
Typical Delivery
1.2
1
0.8
A
0.6
B
Backlog Plan
C
0.4
0.2
50%
Target
Delivery Date
D
ec
em
be
r
r
em
be
N
ov
ct
O
m
be
te
Se
p
100%
ob
er
r
t
us
Au
g
ly
Ju
Ju
ne
ay
M
il
Ap
r
ar
ch
M
ry
ua
Fe
br
Ja
nu
ar
y
0
A/B/C List
50%
25%
25%
A B
Uncertainty Risk
C D
1.2
1
0.8
A
0.6
B
Backlog Plan
C
0.4
0.2
50%
Target
Delivery Date
D
ec
em
be
r
r
em
be
N
ov
ct
O
m
be
te
Se
p
100%
ob
er
r
t
us
Au
g
ly
Ju
Ju
ne
ay
M
il
Ap
r
ar
ch
M
ry
ua
Fe
br
Ja
nu
ar
y
0
Uncertainty
Value
C
B
A
Tim e
Uncertainty
Value
Maximal Schedule
Success Flexible
Scope
Flexible
Minimal
Success
Failure
Tim e
Estimation accuracy improves
(Eveleens and Verhoef)
Estimation accuracy constant
(Eveleens and Verhoef)
Estimation accuracy decreases
(Eveleens and Verhoef)
Take Aways
 Estimation follows log-normal patterns
 There is a cone, sorta
 Relative uncertainty can reduce, but often
may stay relatively constant
 Improvement is possible
 Watch out though
Successful Projects?
Customer Satisfaction
Contact
 Todd Little
 Senior Development Manager, Landmark
Graphics Corporation, www.lgc.com
 [email protected]
 www.toddlittleweb.com
Coming Soon from Addison-Wesley!
Stand Back and Deliver: A Leaders Guide to the
Agile Enterprise
The Cone of Uncertainty
Your Questions?