Transcript Document

1
"Why software projects fail, how to
diagnose potential failure, and how
to make your projects succeed.
With Focus on “12 Tough Questions”
By Tom Gilb
[email protected]
www.Gilb.com
Duration: 90 minutes
Univ. of Westminster, March 11 2009, 6-8PM
www.Gilb.com
Slide 1
2
3 of 500 Sw projects Succeed
•
Letters to the Editor ("The Times"), August 19, 2002, No easy task to start IT projects
–
•
•
From Professor Martyn Thomas,
Sir, Your leading article (August 14) rightly focuses attention on the unacceptable rate of failure of IT
projects, although contrary to what you suggest the problems are not necessarily worse in the public
sector.
A survey of 1,027, mainly private sector, IT projects published in the 2001 British Computer Society
Review showed that only 130 (12.7 per cent) succeeded.
•
Software development projects were worst: of more than 500 projects surveyed,
only three were regarded as having succeeded.
•
As you report (Business, August 14) the problems are similar in the US. I have audited many software
projects in both public and private sector organisations and in my experience the many and complex
reasons for failures boil down to this :
– most customers and most suppliers fail to recognise that complex software development
is an engineering task.
Every step needs engineering rigour, based on sound computer science and supported by formal
quality control.
•
–
–
–
•
Most of the worldwide software industry treats software development as a craft that requires only
technician-level skills.
–
•
Companies who work this way produce software much closer to their planned timescales and at lower cost.
Their customers are asked detailed questions to fix the requirements early in the project, and every subsequent
change can be quickly analysed to see what it will cost, in money and time.
Their software contains so few errors that maintenance costs are negligible.
Until that changes, projects will continue to suffer massive delays and cost overruns.
Regards,MARTYN THOMAS,Visiting Professor of Software Engineering,Oxford University Computing
Laboratory,The Wolfson Building,Park Road, Oxford OX1 3QD.August 14, 2002, [email protected]
www.Gilb.com
Slide 2
3
Talk Summary
1.  90% of all software projects, according to several
surveys, are partly or totally failed.
2. This is due to the project management methods we
have been using, mainly from the Anglo-American
culture.
3. There are a large number of systematic causes of
project failure.
4. This talk will list some fundamental failure causes and
their cures.
5. It will not deal with all possible causes.
6. The first part is methods for diagnosing failure:
knowing you are on the wrong path.
7. The second part is avoiding or preventing failure
and
8. the third part is creating and controlling successful
projects.
www.Gilb.com
Slide 3
10 principles of failure analysis
4
Things you can do to ‘sense’ failure during a project
1.
You must decide and enforce rules of requirement specification
Rule: All quality requirements must be quantified.
2.
You must decide the level of specification-defects (rule violations) that is acceptable
(max. 1 Major Defect/Page?)
3.
You must focus on the top level requirements, and they must be quantified
Top 10 critical requirements: drive entire project, and can be tracked continuously
4.
You must define key stakeholders, and their values, to derive complete requirements
35 Key Stakeholders, 1 or more requirements each: not just ‘users’
5.
You must define top level strategies that measurably meet your objectives within your
constraints - and estimate that they do
Top 10 strategies for meeting top requirements: estimate their numeric impact
6.
You must prioritize early partial delivery of ‘highest stakeholder value’, first
Estimate value at each step. Choose step content with highest value of all possible steps now.
7.
You must control costs, together with benefits, on a continuous basis
Estimate benefits/costs in weekly steps, measure benefits/costs end of week, cumulative
8.
You must plan to measure stakeholder results on an early, frequent, and continuous basis
Spend 1 hour ±30 minutes per week to test and measure value delivered to stakeholders
9.
You must plan to change all plans, based on current and cumulative experience feedback
Be prepared to change, requirements, designs, time estimates, cost estimates, work processes
10.
You must plan to reward your team for benefits actually delivered - not effort.
Reward teams for actual measured required results testably delivered.
www.Gilb.com
Slide 4
5
TWELVE TOUGH QUESTIONS
1. Why isn't the improvement 7. How do you know it works
quantified?
that way? Did it before?
2. What is degree of the risk 8. Have we got a complete
or uncertainty and why?
solution? Are all objectives
3. Are you sure? If not, why
satisfied?
not?
9. Are we planning to do the
4. Where did you get that
'profitable things' first?
from? How can I check it
10. Who is responsible for
out?
failure or success?
5. How does your idea affect
11. How can we be sure the
my goals, measurably?
plan is working, during the
6. Did we forget anything
project, early?
critical to survival?
12. Is it ‘no cure, no pay’ in a
contract? Why not?
There is a detailed paper on these “12 Tough Questions” at www.Gilb.com
www.Gilb.com
Slide 5
Questions to Get Management Involved
in avoiding project failure
6
The 12 Tough questions
Are a way to get managers interested in
metrics
The Tough Questions lead directly
• to quantified management objectives
• to quantified project and product
requirements
• to the use of impact estimation tables
–For evaluating alternative solutions
quantitatively
www.Gilb.com
Slide 6
•Figure 1: Real (NON-CONFIDENTIAL version) example of an initial draft of
setting the objectives that engineering processes must meet.
www.Gilb.com
7
Slide 7
Strategy Impact Estimation
www.Gilb.com
8
Slide 8
1. NUMBERS
9
Why isn’t the
improvement
quantified?
www.Gilb.com
Slide 9
10
2. RISK
What’s degree of
the risk or
uncertainty and
why?
www.Gilb.com
Slide 10
3. DOUBT
11
Are you sure?
If not, Why
not?
www.Gilb.com
Slide 11
12
4. SOURCE
Where did you
get that from?
How can I
check it out?
www.Gilb.com
Slide 12
13
5. IMPACT
How does your
idea affect my
goals,
measurably?
www.Gilb.com
Slide 13
14
6. ALL CRITICAL FACTORS
Did we forget
anything critical
to survival?
www.Gilb.com
Slide 14
7. EVIDENCE
15
How do you know it works that way?
Did it before?
"The most formidable weapon against errors of every kind is
reason."
--Thomas Paine
"Facts are stubborn things; and whatever may be our wishes,
our inclinations, or the dictates of our passions, they cannot
alter the state of facts and evidence.”
--John Adams
(Argument in Defense of the [British] Soldiers in the
Boston Massacre Trials [December 1770]
www.Gilb.com
Slide 15
8. ENOUGH
16
Have we got a
complete solution?
Are all objectives
satisfied?
www.Gilb.com
Slide 16
9. PROFITABILITY FIRST
17
Are we planning
to do the
profitable
things first?
www.Gilb.com
Slide 17
10. COMMITMENT
18
Who’s
Responsible
for failure or
success?
www.Gilb.com
Slide 18
19
11. PROOF
How can we be
sure the plan is
working,
•during the project,
•early?
www.Gilb.com
Slide 19
20
12. NO CURE
Is it ‘no cure, no
pay’, in the
contract?
Why not?
No cure
No pay!
See paper: No Cure No Pay (2006, Tom Gilb). Roots 2005
www.Gilb.com
Slide 20
10 Principles of Preventing Project
Failure
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
21
Keep in touch with critical stakeholders, early, continuously
Do the critical top requirements quantitatively and with rich
background information (justification, authority, benchmarks etc.)
Drive majors defects per page of specs down from over 100 to <10 in six
months; and then in a year or so to less than one
Make sure you have real-project time process improvement at the
project level. Learn fast what works.
Measure real benefits delivered, not tasks and effort
Plan to deliver critical benefits very early, and cumulate them, and lock
them in
Reward success, and punish failure, continuously and early
Give project team power to find the means; focus on controlling the
results, not the means
Make sure all strategies (designs, architectures) are estimated with
Impact Estimation, using evidence, uncertainty and sources.
Make sure suppliers are on an evolutionary contract, and paid by results
only
www.Gilb.com
Slide 21
The Simplest and Best Agile Project Method; ‘XE’
1
Process Description
22
1. Gather from all the key stakeholders the top few (5 to 20) most
critical goals that the project needs to deliver.
2
Give each goal a reference name (a tag).
2. For each goal, define a scale of measure and a ‘final’ goal level.
3
For example: Reliable: Scale: Mean Time Before Failure, Goal: 1
month.
3. Define approximately 4 budgets for your most limited resources
(for example, time, people, money, and equipment).
4
4. Write up these plans for the goals and budgets
(Try to ensure this is kept to only one page).
5. Negotiate with the key stakeholders to formally agree the goals
and budgets.
6. Plan to deliver some benefit
5
(that is, progress towards the goals)
in weekly (or shorter) increments (Evo steps).
7. Implement the project in Evo steps.
8
6
Report to project sponsors after each Evo step (weekly, or shorter)
with your best available estimates or measures, for each
performance goal and each resource budget.
On a single page, summarize the progress to date towards achieving
the goals and the costs incurred.
7
8. When all Goals are reached: ‘Claim success and move on’
a. Free remaining resources for more profitable ventures
www.Gilb.com
Slide 22
23
Agile project Management; XE Policy
Policy
1.
The project manager, and the project, will be judged
exclusively on
1.
2.
3.
2.
The team will be paid and rewarded for
1.
2.
3.
4.
the relationship of progress towards achieving the goals
versus the amounts of the budgets used.
The project team will do anything legal and ethical to deliver the
goal levels within the budgets.
benefits delivered
in relation to cost.
The team will find their own work process and their own
design.
As experience dictates, the team will be free to suggest to the
project sponsors (stakeholders) adjustments to ‘more realistic
levels’ of the goals and budgets.
www.Gilb.com
Slide 23
10 Principles for Project Success.
24
How to make sure you deliver what is really valued
1.
2.
3.
Keep in intimate contact with key stakeholders (weekly basis, Evo delivery)
Quantify, and verify, and keep updating - the critical few stakeholder values
Derive project requirements from the stakeholder values on the basis of
most profitable stakeholder values
are supported by specific project requirements
4.
Build systems evolutionarily, using existing systems and environments as departure
basis, for delivering results - even if longer vision is radically different.
Prove concepts and deliver real results.
5.
Do your project from a systems engineering point of view - not a ‘programmers view’.
Plan and do all the things it takes to successfully deliver real results to real stakeholders.
6.
7.
8.
Line up rewards system so that stakeholder view, management view, project view is
pulling in same direction and rewarded for same thing
Teach all employees good planning and specification practices, using formal rules,
formal sampling measures of rule conformance, and minimum quality level exit control:
don’t even let ‘junk specs’ out your process door!
Avoid all pressure in the direction of bureaucratic development processes
(like CMM, CMMI, Waterfall Model) - focus on results.
9.
Consider changing and improving your entire organizational climate,
a CEO Job, to be more receptive to change.
10.
Get top management (CEO, MD, President, COO, CTO) to establish and follow up simple
results oriented policy.
(no majors in specs, measurable benefits early, highest benefit to costs first. That’s all.)
www.Gilb.com
Slide 24
25
Last Slide
[email protected]
+47 920 66 705
+44 77 1 267 0707
www.Gilb.com
Slide 25
Backup Slides
26
• To answer questions
• To illuminate in more detail
• In case there is sudden additional time to the
30 minutes
www.Gilb.com
Slide 26
Sensing Problems in Existing Projects
27
How would you know that a project is on a failure
path?
Early?
Quantitative sampling of requirements specs show high Major
Defect Density (like 100 majors per page)
Early Planned Evolutionary result delivery steps fail to deliver
planned value, or take far more time to do so than planned.
Quantitatively? See above
Credibly? See above
www.Gilb.com
Slide 27
28
Example of Evo Step Tracking
Step 9 is a weekly step
Courtesy Trond Johansen, FIRM AS, Norway.
Paper: “From waterfall to EVO in a medium size Norwegian software house. The path and
experiences”.
2004
www.Gilb.com
Slide 28
29
FIRM EVO week
www.Gilb.com
Slide 29
Real Inspection of System Requirements Specification (SRS) of 82 pages.
Major US Corporation.
30
This presentation shows how to carry out a short specification quality control
with senior/middle managers.
The purpose is to
make managers aware
that they play a key-role in creating project delays
by approving poor quality of requirements documents.
The inspection results shown in this real-life example
successfully predicted a project delay of at least 2 calendar years.
Poor quality marketing requirements documents prove time and again
to be a good predictor of project delays.
The clue is that requirements documents
with a high defect density
are an indicator of a truly unprofessional engineering culture.
www.Gilb.com
Slide 30
Framework
31
Demonstration of power of Inspection
8 Managers
2 hours
4 real requirements specifications
www.Gilb.com
Slide 31
“Rules”:
Best Practice Strong Advice
32
Introduce the following three rules for inspecting a requirements document:
Three Rules for Requirements:
1. Unambiguous to intended Readership
2. Clear enough to test.
3. No Design specs (= ‘how to- be good’) mixed in
Mixed up in the Requirements
(Reqts. = ‘how good - to be’) not how to be good!
MARK Design as “D”
Except if it is a conscious Design Constraint (which is a requirement type)
www.Gilb.com
Slide 32
Specification Defect
33
Explain the definition of a Defect:
A Spec Defect is a violation of a Rule
Note: If there are 10 ambiguous terms in a single requirement
then there are 10 defects!
9 of 10 removed, leaves 1 defect
www.Gilb.com
Slide 33
Severity
34
Explain:
• the definition of Major Defect
• the checkers must focus on finding Major Defects
Major: a defect severity where there is
potential of
high (10 lost engineering hours) loss
later downstream (test, field).
www.Gilb.com
Slide 34
Exit?
35
Agree with the management team on a numeric exit condition:
Is 1,000 Majors per page OK? 100, 10, 1?
Exit Conditions: (when Requirements can
go to Design, Test etc with little risk)
Maximum 1 Major Defect/ (Logical) Page
Logical Page = 300 Non commentary words.
www.Gilb.com
Slide 35
The Job
I gave the 8 managers
(4 did page 81, 4 did page 82)
36
You have up to 30 minutes
for checking 1 requirements page
from an 82 page document
Count all potential Rule Violations
= Defects
Classify defects as Major or minor
www.Gilb.com
Slide 36
Report for Page 81
37
(reported inspection results on requirements document, 4 managers)
24= Total, 15=Majors, 5 = Design
44,
15,
19
55,
20,
4
22,
4,
2
Defect Density estimation for this page =
Total for group 20 x2 = 40 Majors assume are unique
If 33.333% effective, total in page = 3x 40=120
Of which 2/3 or 80 were not yet found.
If we fix all we found (40), then the estimated remainder of Majors
would be 80 (not found) +8 not fixed for real = 88 Majors remaining.
www.Gilb.com
Slide 37
38
Report for page 82
(reported inspection results on requirements document, 4 other managers)
Total,
Majors,
Defects
M+m
M
Design (Design = part of Total and M&m)
--------------------------------41,
24,
D=1
33,
15,
D=5
30,
44,
24,
3,
60
120
180
D=10
D=5
Team would (we estimate) log unique Majors about 2x30=60 actually found
Which is 1/3rd of total existing , so estimated total this page is about 180 Majors
About 2/3, = 120 Majors , were not found by the team during the agile spec QC process.
Not all 180 spec defects (only 25%-35%) would actually become product defects, before testing.
We don’t know which ones in advance: it depends on human interpretation, on the day.
Many (only 25%-35%) of these Major spec defects would cause product defects (bugs in product, not just spec)
These would be found later in test,and product use as bugs or major defects
The product defect rework would cause serious project (product development) delay
No human review process is better, even when continuously improved for years, than 60% (source code record) to 88% (the ‘logic spec’ record) at defect
identification. <- IBM MN.
If we attempt to fix the 60 defects we logged, and correctly fix 5/6 of them; then 10 defects are ‘failed fixes’, so:
The total remaining after inspection and editing = 10+120 =130 Majors per page.
10 failed fixed +
120 not detected major defects
= 130 Major defects per page ( 300 non commentary words) remaining after the review and fixing process - before text.
Conclusion:
Human defect removal by Inspections/reviews/SQC is a hopeless cause: not worth it
Spec QC can be used, in spite of imperfect effectiveness, to accurately estimate major defect level per page.
This measurement can be used to motivate engineers to dramatically (100x! Over about 7 learning cycles) reduce their defect insertion ( rule violation) to a
practical exit level ( like < 1.0 Major/page)
www.Gilb.com
Slide 38
Extrapolation to
Total Majors in Whole Document
39
Page 81: 120 majors/p
Page 82: 180 Majors/p
Average 150 Majors/page x 82 page = 12,300 Majors in
the document.
----------------If a Major has 1/3 chance of causing loss
And each loss is avg 10 hours then total project Rework
cost is about 41,000 hours loss.
(This project was over a year late)
1 year = 2,000 hour 10 people
www.Gilb.com
Slide 39