Transcript John Miller

Pitfalls of Project Estimating:
An Applied Lesson from Dr. Deming’s Funnel Experiment
John Miller, PMP, CLSSBB
4/28/10
Contents
•
•
•
•
•
•
•
•
•
•
•
Introduction
Dr. Deming’s Funnel Experiment
What is a Process?
Statistical Measures
Estimating Methods
Concepts
Averages and Standard Deviation
Data Points to Data Trends
Control Charts
Lessons from the Funnel Experiment
Using Excel to Avoid Common Mistakes
J. Miller, 4/28/10
Introduction
Parametric and evidenced-based estimating tools have become very sophisticated.
Many essential benefits of these tools can inexpensively be provided using Excel, but
project managers need to be aware of risks associated with improperly adjusting
estimating metrics.
Dr. Deming’s Funnel experiment serve’s as a model for improperly adjusting
estimating metrics. A simple example will demonstrate how statistically valid estimating
methods can be duplicated in Excel.
Dr. Deming’s Funnel Experiment
“A experiment that demonstrates the effects of tampering [with a process]. Marbles are
dropped through a funnel in an attempt to hit a flat-surfaced target below. The experiment
shows that adjusting a stable process to compensate for an unstable result or an
extraordinarily good result will produce output that is worse than if the process had been
left alone.”
Donna C. Summers, Quality Management 2nd Edition, p 546
J. Miller, 4/28/10
DR. Deming’s Funnel Experiment
Funnel Comparison to Project Estimating
Funnel = Estimating process
Target = Goal -- project time goal or cost goal or resource goal
Marble = Actual project or estimate that creates measurable output – cost, schedule, etc.
Each marble drop = actual project or part of project
Moving the Funnel = Adjusting the project resources after each project to try to meet
the target on the next project. (Ex. Adding people or tools, if schedules are being
missed, removing resources if budgets are being missed)
Moving the Target = Adjusting the time or cost estimate estimating process
after each project based on the results from the last project.
J. Miller, 4/28/10
DR. Deming’s Funnel Experiment
Funnel Comparison to Project Estimating
Each of the four rules is discussed below.
Rule 1: Leave the funnel fixed over the target. Why don't we simply adjust the funnel
after each drop so the next drop will be closer to the target? Don’t change the estimating
process, accept the variation, the difference between the estimate and actual projects.
Rule 2: For every drop, the marble will come to rest a distance "z" from the target.
Rule 2 is to move the funnel a distance -z from its last position. Move the funnel based on
the funnels last position. Chang the estimating process (metrics) after each project or
difference between estimate and actual.
Rule 3: Move the funnel a distance -z from the target after each drop of the marble
that ends up a distance z from the target. Note that Rule 2 moves the funnel based on the
funnel's last position. Rule 3 moves the funnel a distance from the target.
Rule 4: Rule 4 is simply to set the funnel over where the last drop came to rest.
J. Miller, 4/28/10
What is a Process?
•
A series of steps or actions that convert input to output.
•
Project estimating is a process.
•
Processes are described by statistical measures – numbers that describe “groups”.
•
Groups are made of data points.
•
What do we need to better understand this group of data points?
Centering – Mean
Spread - Stand Deviation
Shape – Normality, data distribution
(assume all distributions here are Normal)
•
We need data.
J. Miller, 4/28/10
Statistical Measures
• Dimensional attributes
• Product or Process metrics such as:








# or % defects
% on time deliveries
Efficiency (productivity, ratios)
# units of output / unit of time (may also include inputs)
Time / # units of output
xx units / hr
Turn Around Time (TAT), xx Points / unit
Cost/Unit
Which can be averages?
J. Miller, 4/28/10
Estimating Methods
PMBOK 3rd
6.4.2 Activity Duration Estimating: Tools and Techniques
.1 Expert Judgment
Activity durations are often difficult to estimate because of the number of factors that can influence
them, such as resource levels or resource productivity. Expert judgment, guided by historical
information, can be used whenever possible. The individual project team members may also
provide duration estimate information or recommended maximum activity durations from prior
similar projects. If such expertise is not available, the duration estimates are more uncertain and
risky.
.2 Analogous Estimating
Analogous duration estimating means using the actual duration of a previous, similar schedule
activity as the basis for estimating the duration of a future schedule activity. It is frequently used to
estimate project duration when there is a limited amount of detailed information about the project
for example, in the early phases of a project. Analogous estimating uses historical information
(Section 4.1.1.4) and expert judgment. Analogous duration estimating is most reliable when the
previous activities are similar in fact and not just in appearance, and the project team members
preparing the estimates have the needed expertise.
J. Miller, 4/28/10
Estimating Methods
PMBOK 3rd
6.4.2 Activity Duration Estimating: Tools and Techniques
.3 Parametric Estimating
Estimating the basis for activity durations can be quantitatively determined by multiplying the
quantity of work to be performed by the productivity rate. For example, productivity rates can be
estimated on a design project by the number of drawings times labor hours per drawing, or a cable
installation in meters of cable times labor hours per meter. The total resource quantities are
multiplied by the labor hours per work period or the production capability per work period, and
divided by the number of those resources being applied to determine activity duration in work
periods.
.4 Three-Point Estimates
The accuracy of the activity duration estimate can be improved by considering the amount of risk in
the original estimate. Three-point estimates are based on determining three types of estimates:
• Most likely. The duration of the schedule activity, given the resources likely to be assigned, their
productivity, realistic expectations of availability for the schedule activity, dependencies on other
participants, and interruptions.
• Optimistic. The activity duration is based on a best-case scenario of what is described in the most
likely estimate.
• Pessimistic. The activity duration is based on a worst-case scenario of what is described in the
most likely estimate.
An activity duration estimate can be constructed by using an average of the three estimated
durations. That average will often provide a more accurate activity duration estimate than the single
point, most-likely estimate.
J. Miller, 4/28/10
Estimating Methods
.1 Expert Judgment
Expert judgment, guided by historical information, can be used whenever possible.
.2 Analogous Estimating
Analogous estimating uses historical information and expert judgment.
.3 Parametric Estimating
Estimating the basis for activity durations can be quantitatively determined by multiplying the
quantity of work to be performed by the productivity rate.
.4 Three-Point Estimates
An activity duration estimate can be constructed by using an average of the three estimated
durations.
• Mean
The mean is the average data point value within a data set. To calculate the mean, add all of
the individual data points then divide that figure by the total number of data points.
• Published estimating methods overlook critical statistical attributes.
What’s missing?
J. Miller, 4/28/10
Estimating Methods – Averages
What’s missing?
Standard Deviation
Mean
If I use the “average” of past projects to estimate future projects, what is virtually guaranteed?
J. Miller, 4/28/10
Concepts
• Statistical measure – numbers that describe groups. A
numerical value, such as standard deviation or average, that
characterizes the sample or population from which it was
derived.
• What do we need to better understand this group of data
points?
 Centering – Mean
 Spread - Stand Deviation
 Shape – Normality, data distribution
(assume all distributions here are Normal)
J. Miller, 4/28/10
Averages and Standard Deviation
• Mean
The mean is the average data point value within a data set. To calculate the
mean, add all of the individual data points then divide that figure by the total
number of data points.
• Standard Deviation
A statistic used to measure the variation in a distribution. Standard deviation is a
measure of the spread of data in relation to the mean. It is the most common
measure of the variability of a set of data.
J. Miller, 4/28/10
Averages and Standard Deviation
Standard Deviation Example
Suppose we wished to find the standard deviation of the data set consisting of the values
3, 7, 7, and 19.
Step 1: find the arithmetic mean (average) of 3, 7, 7, and 19
(3 + 7 + 7 + 19) / 4 = 9.
Step 2: find the deviation of each number from the mean,
3−9=−6
7−9=−2
7−9=−2
19 − 9 = 10.
Step 3: square each of the deviations, which amplifies large deviations and
makes negative values positive,
( − 6)2 = 36
( − 2)2 = 4
( − 2)2 = 4
(10)2 = 100.
Step 4: find the mean of those squared deviations,
(36 + 4 + 4 + 100) / 4 = 36.
Step 5: take the non-negative square root of the quotient (converting squared
units back to regular units), so, the standard deviation of the set is 6.
J. Miller, 4/28/10
Standard Deviation Discovered
Assume: Mean = 30, Std Dev = 6
-1s=-6
- 2 s = 2(-6) = -12
- 3 s = 3(-6) = -18
- 4 s = 4(-6) = -24
- 5 s = 5(-6) = -30
- 6 s = 6(-6) = -36
s
-6
1s=6
2 s = 2(6) = 12
3 s = 3(6) = 18
4 s = 4(6) = 24
5 s = 5(6) = 30
6 s = 6(6) = 36
s
-5
s
-4
s
-3
s
-2
s
-1
-36 -30 -24 -18 -12 -6
-6
0
6
12
18
24
30
s
1
s
2
s
3
s
4
s
5
s
6
+6 +12 +18 +24 +30 +36
36
42
48
54
60
66
J. Miller, 4/28/10
Standard Deviation Discovered
Assume: Mean = 30, Std Dev = 6
30
-6
0
6
12
18
24
36
42
48
54
60
66
J. Miller, 4/28/10
Standard Deviation is Critical
Assume:
There are two projects, each using the Three Point Estimate Method
Project A
Optimistic = 1
Most likely = 50
Pessimistic = 99
Project B
Optimistic = 49
Most likely = 50
Pessimistic = 51
0
10
1
20
30
40
50 60
49 51
70
80
90
100
99
Project A
Project B
Data points are 1, 50 ,99
Mean = 50
StdDev = 49
Data points are 49, 50, 51
Mean = 50
StdDev = 1
J. Miller, 4/28/10
Normal Distribution with Std Dev
Estimating Methods PMBOK 3rd
6.4.3 Activity Duration Estimating: Outputs
.1 Activity Duration Estimates
Activity duration estimates are quantitative assessments of the likely number
of work periods that will be required to complete a schedule activity. Activity
duration estimates include some indication of the range of possible results. For
example:
• 2 weeks ± 2 Points to indicate that the schedule activity will take at least eight
Points and no more than twelve (assuming a five-Point workweek).
• 15 percent probability of exceeding three weeks to indicate a high
probability—85 percent—that the schedule activity will take three weeks or
less.
Let’s take a closer look at making a valid estimate…
J. Miller, 4/28/10
Data Points to Data Trends
J. Miller, 4/28/10
Point 7
Point 6
Point 5
Point 4
Point 3
Point 2
Point 1
Data Points to Data Trends
J. Miller, 4/28/10
Data Points to Data Trends
J. Miller, 4/28/10
Data Points to Data Trends
J. Miller, 4/28/10
P1
P2
P3
P4
P5
P6
P7
Data Points to Data Trends
J. Miller, 4/28/10
Data Points to Data Trends
J. Miller, 4/28/10
Point 7
Point 6
Point 5
Point 4
Point 3
Point 2
Point 1
Data Points to Data Trends
J. Miller, 4/28/10
Data Points to Data Trends
J. Miller, 4/28/10
Data Points to Data Trends
J. Miller, 4/28/10
P1
P2
P3
P4
P5
P6
P7
Data Points to Data Trends
J. Miller, 4/28/10
P1
P2
P3
P4
P5
P6
P7
Data Points to Data Trends
J. Miller, 4/28/10
UCL
X
s?
LCL
P1
P2
P3
P4
P5
P6
P7
Data Points to Data Trends
J. Miller, 4/28/10
Data Points to Data Trends
J. Miller, 4/28/10
Process Exercise
What about when we don’t have data for all process steps?
What if we had a way to estimate the data?
Creating a Control Chart
1. Define the metric: time/unit of output, delta of actual vs. planned time or cost, etc
2. Must be homogeneous
3. Data collection planning
1. Who will collect data?
2. What aspect of the process will be measured?
3. Where, or at what point in the process will the measurement be taken?
4. When or how frequently will the data be collected?
5. Why is this sample being taken?
6. How will the data be collected?
7. How many samples will be taken?
4. Collect the data
Step Step Step Step Step Step Step Step Step Step
#
1
2
3
4
5
6
7
8
9
10 Steps
Project 1
45
32
78
12
67
54
87
43
54
87
10
Project 2
35
67
9
64
55
75
33
58
8
Project 3
56
45
29
54
39
66
63
70
8
What
isn’t collected
every66
project
Projectif4data43
43
56 for31
40 or period?
69
45
75
9
Project 5
41
67
9
59
56
71
43
67
78
9
Project 6
60
45
14
54
67
76
65
74
8
Project 7
54
39
63
61
59
44
60
7
Project 8
41
74
13
63
73
49
59
71
8
Total
Mean
System Step
Time Duration
559
55.9
396
49.5
422
52.8
468
52.0
491
54.6
455
56.9
380
54.3
443
55.4
Creating a Control Chart
Step Step Step Step Step Step Step Step Step Step
1
2
3
4
5
6
7
8
9
10
Project 1
45
32
78
12
67
54
87
43
54
87
Project 2
35
67
9
64
55
75
33
58
Project 3
56
45
29
54
39
66
63
70
Project 4
43
43
56
31
66
40
69
45
75
Project 5
41
67
9
59
56
71
43
67
78
Project 6
60
45
14
54 OR
67
76
65
74
Project 7
54
39
63
61
59
44
60
Project 8 Step Step41 Step74 Step13 Step Step63 Step73 Step49 Step59 Step71
Project 9 1 51 2 45 3 72 4 12 5 65 6
7 87 8 46 9 56 1085
Project
49
34
60
67
56
75
Project10
1
45
32
78
128
67
54
87
43
54
87
Project 2
35
67
9
64
55
75
33
58
Project 3
56
45
29
54
39
66
63
70
Project 4
43
43
56
31
66
40
69
45
75
Project 5
41
67
9
59
56
71
43
67
78
Project 6
60
45
14
54
67
76
65
74
AsProject
long 7as the
54 units
39 are the
63 same, the
61 model
59 works! 44
60
Project 8
41
74
13
63
73
49
59
71
Project 9
51
45
72
12
65
87
46
56
85
Project 10
49
34
60
8
67
56
75
Total
Mean
# System Step
Steps Time Duration
10
559
55.9
8
396
49.5
8
422
52.8
9
468
52.0
9
491
54.6
8
455
56.9
7 Total
380 Mean
54.3
443 Step
55.4
# 8 System
519 Duration
57.7
Steps9 Time
349
49.9
107
559
55.9
8
396
49.5
8
422
52.8
9
468
52.0
9
491
54.6
8
455
56.9
7
380
54.3
8
443
55.4
9
519
57.7
7
349
49.9
Creating a Control Chart
Step Step Step Step Step Step Step Step Step Step
#
1
2
3
4
5
6
7
8
9
10 Steps
Project 1
45
32
78
12
67
54
87
43
54
87
10
Project 2
35
67
9
64
55
75
33
58
8
Project 3
56
45
29
54
39
66
63
70
8
Project 4
43
43
56
31
66
40
69
45
75
9
Project 5
41
67
9
59
56
71
43
67
78
9
Project 6
60
45
14
54
67
76
65
74
8
Project 7
54
39
63
61
59
44
60
7
Project 8
41
74
13
63
73
49
59
71
8
Project 9
51
45
72
12
65
87
46
56
85
9
Project 10
49
34
60
8
67
56
75
7
Total
Mean
System Step
Time Duration
559
55.9
396
49.5
422
52.8
468
52.0
491
54.6
455
56.9
380
54.3
443
55.4
519
57.7
349
49.9
Total # Projects
10
10
10
10
10
10
10
10
10
10
Recorded # Projects
8
8
9
9
9
9
9
7
8
7
% of Projects Reporting 80.0% 80.0% 90.0% 90.0% 90.0% 90.0% 90.0% 70.0% 80.0% 70.0%
Mean 49.9 39.3 64.7 15.2 61.9 54.3 75.4 43.3 60.3 77.1
Standard Deviation
6.6
5.1 10.1
8.6
5.2
9.4
7.3
5.0
4.5
6.6
Variance 44.1 25.9 102.0 74.4 27.1 88.0 53.0 24.9 19.9 43.8
J. Miller, 4/28/10
Creating a Control Chart
Total # Projects
10
10
10
10
10
10
10
10
10
10
Recorded # Projects
8
8
9
9
9
9
9
7
8
7
% of Projects Reporting 80.0% 80.0% 90.0% 90.0% 90.0% 90.0% 90.0% 70.0% 80.0% 70.0%
Mean 49.9 39.3 64.7 15.2 61.9 54.3 75.4 43.3 60.3 77.1
Standard Deviation
6.6
5.1 10.1
8.6
5.2
9.4
7.3
5.0
4.5
6.6
Variance 44.1 25.9 102.0 74.4 27.1 88.0 53.0 24.9 19.9 43.8
5. Calculate the centerline – Sum of Means
6. Calculate the Upper and Lower Control Limits.
UCL = Sum of Means + 3 σ
LCL = Sum of Means - 3 σ
Sum of Means
Sum of Variances
541.4
503.3
Sq Rt of Variances
22.4
σ = Sq Rt of Sum of Variances
-3
StdDev
-3* 22.4
-67.2
474.2
-2
StdDev
-2* 22.4
-44.8
496.6
-1
StdDev
-1* 22.4
-22.4
519.0
Mean
541.4
541.4
+1
StdDev
1* 22.4
22.4
563.8
+2
StdDev
2* 22.4
44.8
586.2
+3
StdDev
3* 22.4
67.2
608.6
Creating a Control Chart
-3
StdDev
-3* 22.4
-67.2
474.2
-2
StdDev
-2* 22.4
-44.8
496.6
-1
StdDev
-1* 22.4
-22.4
519.0
Mean
541.4
541.4
+1
StdDev
1* 22.4
22.4
563.8
+2
StdDev
2* 22.4
44.8
586.2
+3
StdDev
3* 22.4
67.2
608.6
After calculating the control limits, place the center line and control
limits on the chart
608.6
541.4
474.2
Creating a Control Chart
A Guide to the Project Management Body of Knowledge (PMBOK® Guide) Third Edition
2004 Project Management Institute, Four Campus Boulevard, Newtown Square, PA 19073-3299 USA
J. Miller, 4/28/10
Determine if the Process is Stable Over Time
• Check the date for trends
• Check the date for patterns
J. Miller, 4/28/10
Process Stable Over Time?
Learn how to decide if the process is stable and if the data can be used for
further analysis.
Data is analyzed using a control chart.
(1) Point beyond UCL and LCL (beyond 3 sigma)
J. Miller, 4/28/10
Process Stable Over Time?
(2) Seven consecutive points on the same side of center line.
J. Miller, 4/28/10
Process Stable Over Time?
(3) Trend: 6 consecutive points steadily increasing or decreasing:
J. Miller, 4/28/10
Process Stable Over Time?
(4) Repeating pattern and cycles.
J. Miller, 4/28/10
Process Stable Over Time?
I-MR Chart of Northeast
Individual Value
1400
U C L=1367.5
1200
_
X=1000.0
1000
800
LC L=632.4
600
1
3
5
7
9
11
O bser v ation
600
13
15
17
19
Can the above data be used for any further analysis and decision making?
1
Moving Range
1
450
300
Observations:
150
No point
is beyond UCL and LCL
0 no pattern repetition
There is
1
3
5
7
9
11
13
15
17
There is no continuous increase or decrease
of 6 data points
O bser v ation
There is NO 7 consecutive points on the same side of center line
U C L=451.6
__
M R=138.2
LC L=0
19
So, this data can be used for future predictions
J. Miller, 4/28/10
Process Stable Over Time?
I-MR Chart of Northwest
Individual Value
1400
1
U C L=1298.0
1200
_
X=1025.9
1000
800
LC L=753.8
1
600
1
3
5
7
9
11
O bser vation
13
15
17
Can the above data be used for any further analysis and decision making
19
U C L=334.3
M oving Range
300
Observation :
There are points beyond UCL and LCL
200
There is no pattern repetition
There is no continuous increase or decrease of 6 data points
100
There is 7 consecutive points on the same side of center line
0
__
M R=102.3
LC L=0
This1tells, there
is a special
cause
influencer,
which
is causing
this
Before
making
3
5
7
9
11
13
15 to happen.
17
19
bser vation
any predictions for the future, need toOanalyze
the special cause
J. Miller, 4/28/10
Process Stable Over Time?
I-MR Chart of Southwest
Individual Value
1600
U C L=1534.5
1400
_
X=1195.9
1200
1000
LC L=857.3
1
3
5
7
9
11
O bser vation
13
15
Moving Range
Can400
the above data be used for any further analysis and decision making :
300
Observation
:
No point is beyond UCL and LCL
200
There
is no pattern repetition
There
100 is no continuous increase or decrease of 6 data points
There is 7 consecutive points on the same side of center line
0
17
19
U C L=416.0
__
M R=127.3
LC L=0
1 using 3
7 predictions,
9
11
13
15 cause 17
So, Before
the data5for future
need
to analyze
the
for data 19
points to be on
O bser vation
same side (during the period) and if required separate those points from the rest, in making the
decision
J. Miller, 4/28/10
Determine if the Process is Stable Over Time
• Check the date for trends
• Check the date for patterns
Apply Lessons from the Funnel Experiment
Rule 1: Leave the funnel fixed over the target. Why don't we simply adjust the funnel
after each drop so the next drop will be closer to the target? Don’t change the estimating
process, accept the variation, the difference between the estimate and actual projects.
Rule 2: For every drop, the marble will come to rest a distance "z" from the target.
Rule 2 is to move the funnel a distance -z from its last position. Move the funnel based on
the funnels last position. Chang the estimating process (metrics) after each project or
difference between estimate and actual.
Rule 3: Move the funnel a distance -z from the target after each drop of the marble
that ends up a distance z from the target. Note that Rule 2 moves the funnel based on the
funnel's last position. Rule 3 moves the funnel a distance from the target.
Rule 4: Rule 4 is simply to set the funnel over where the last drop came to rest.
J. Miller, 4/28/10
Using Excel to Avoid Common Mistakes
Questions?
Contact
John H. Miller
[email protected]
J. Miller, 4/28/10