Transcript Slide 1

Software Quality and Team
Development Practices based on
CERN Experience
Rostislav Titov,
GS-AIS-EB Section Leader,
CERN
The reality today
Failure
 31.1% of IT projects will be canceled before they ever get
completed
 52.7% of projects will cost 189% of their original estimate
 More than 50% of software projects fail today.
 2009: highest failure rate in over a decade!
Success
 Only 16.2% software projects that are completed on time
and on budget
 In the large companies only 9% of their projects come in on
time and on budget.
Catastrophes
– Ariane 5 (7bn dev, 500 million)
numerical conversion error
CERN
AIS
Typical Failures

User requirements not met

Software unreliable

It works great for me, now deploy it for 10,000 users…

Too late (You took so long that our requirements have
changed…)

You did what I asked. but I didn’t say what I meant…
CERN
AIS
Reasons

Unrealistic Schedules
– Race through the requirements, produce a
superficial design and rush into coding.
 Changing Requirements During Development
“Writing software from requirements is like walking
on water – its easier when frozen”

Inappropriate Staffing
Fast
 Poor-Quality Work
There's a saying about software quality:
“If it doesn't have to work, we can build
it really fast.”
Choose
two
Cheap

Believing in magic
– Pitfalls of commercial products
CERN
AIS
Good
Project Success factors
Project Success Factors and % of Responses
1
User Involvement
15.90%
2
3
4
Executive Management Support
Clear Statement of Requirements
Proper Planning
13.90%
13.00%
9.60%
5
6
7
8
Realistic Expectations
Smaller Project Milestones
Competent Staff
Ownership
9
10
Clear Vision & Objectives
Hardworking, Focused Staff
Other
Project Challenged Factors and % of Responses
2.90%
2.40%
13.90%
1
2
Lack of User Inputs
Incomplete Requirements & Specifications
12.80%
12.30%
3
4
Changing Requirements & Specifications
Lack of Executive Support
11.80%
7.50%
5
6
7
8
Technology Incompetence
Lack of Resources
Unrealistic Expectations
Unclear Objectives
Other
7.00%
6.40%
5.90%
5.30%
23.00%
CERN
AIS
8.20%
0 - 7.7%
7.20%
5.30%
Analysis

State Goal
“send a man to the moon before end of the decade &
return him safely to Earth”, JF Kennedy


Specify the problem not the solution
Classification
Projects completed by the largest
M
Must
American companies have only
o
S
Should
approximately 42% of the originally
C
Could
proposed features and functions.
0
W

Would
Concept of Operations document
“This is my understanding of what you want”

Beware of requirements ‘gold plating’

Verifiable : use-cases
CERN
AIS
Good Design Principles

Consider alternative approaches
Not tunnel vision

Traceable to the requirements
Correct & complete

Not reinvent the wheel
Use a pattern

Adaptability Accommodate Change

Maximize Cohesion

Minimize Coupling

Understandabilty
Mars Polar Lander
($165 million) – design error
“A designer knows he has achieved
perfection not when there is nothing left to
add, but when there is nothing left to take
away”
- Antoine de Saint Exupéry
A design must be understandable if it is to support modification.
CERN
AIS
Good & Bad design
Good Design
Bad Design


One conceptual change
requires changes to many
parts of the system.

Logic has to be duplicated.

Cost of a bad design
becomes overwhelming.

Can't remember where all the
implicitly linked changes
have to take place.

Can't add a new function
without breaking an existing
function.

Complexity




CERN
AIS
Change in one part of the
system doesn't always
require a change in another
part of the system.
Every piece of logic has one
and one home.
The logic is near the data it
operates on.
System can be extended with
changes in only one place.
Simplicity
Coding
“It doesn’t matter if I write poor code…
the compiler will tell me if there is
problem”
“It doesn’t matter if I make a mistake…
it will come out in testing”
• Mars Climate Orbiter ($125 million)
metric conversion error
CERN
AIS
Bugs

Experienced software engineers inject one defect in about
every 10 lines of code.

The programmers aren't incompetent or lazy - they're just
human.

All humans make mistakes, but in software, these mistakes
result in defects.

This means that a modest-size program of 100,000 lines of
code typically would start with about 10,000 defects.

Examples :
INTEL Pentium : no more than 80-90 Bugs
Cell Phone (200 000 loc) up to 600 errors.
Windows-95: 10 Mill. loc: up to 200 000 errors.
CERN
AIS
The Cost of Change
Cost
Time
CERN
AIS
Code Reviews - guidelines

CERN
AIS
Form
– Product is guilty until proven innocent
– Producer is innocent because he/she is not on
trial
– More likely to find bugs if you assume they are
there
– Evaluate product not producer
– Emphasize "review" aspect; do not "fix it
here".
– Raise problems. Do not discuss solutions
Code Reviews - guidelines

CERN
AIS
Management Involvement
– NONE!
– Not a manager's status meeting
– Management is not represented during
inspections
– Inspections must not be used as a tool to
evaluate workers
– … Not a committee, not a working group, not a
status report & not an appraisal instrument …
Benefits

Primary objective
– remove defects as early as possible in the development
process

Other benefits :
–
–
–
–
–
–
–
–
–
CERN
AIS
Early Testing
Project Tracking
Educational – share best practices
Training of new/junior programmers
Improved Communication
Improved Individual Quality
Cross-training
Process-improvement
Shared Responsibility – no individual blame
The “Yes, buts...”

I don’t have time for this...

Good programmer’s code doesn’t need
reviewing

Its only a ‘minor’ piece of code

Code Changes, then what?
– 2nd pair eyes rule
– Pair programming
CERN
AIS
Coding Standards – why?
Why ?
 80% of the lifetime cost of a piece of software goes to
maintenance.
 Hardly any software is maintained for its whole life by the
original author.
 Code conventions improve the readability of the software,
allowing engineers to understand new code more quickly
and thoroughly.
Cannot review unless you have standards...
 endless debate – was driving too fast? Cannot answer
without defined speed limits
 Recommend best practices, avoid bad practices
 Maintainable & reliable software is key
Produces
 Common Code Ownership
CERN
AIS
Now imagine you have
A multi-domain, multi-lingual
horizontal software application
supporting over 16’000 users in 42
countries composed of :
1.5 million
6,000
10,000
many
Lines of Code
Java classes
HTML templates
other files
Welcome to EDH!
CERN
AIS
EDH: Good Old Days (1991-98)
“Imagination rules the world”
Mac or PC or Unix?
C or C++ or ?
University atmosphere
Freedom & Individualism
CERN
AIS
Choice, choice, choice...
Results

Healthy outside,
but unhealthy inside

Evolution from freedom to Chaos!
Development Platform : Mac, PC & Unix
Code : C,C++,Python, Prolog, ProC, PL/SQL, Perl...
Comments : Spanish, Italian, French & English...
Bugs : “Y2K don’t care”

Obvious code never reviewed : Why would you
show your code to anybody? Never did at
university... Results count!

Consequence : Maintenance became the primary
resource-consuming activity
CERN
AIS
From University to Industry
Individual Development Practices
Team Development Practices
Freedom of Choice for
Development Environment
Uniform development environment
Free selection of tools
Common set of tools
Choice of language & technology
Single technology choice
Individual Code Responsibility
(& blame)
Common Code Ownership
(& learning)
Quality of the Product
Quality of the Process
... driven by the members of the team
CERN
AIS
(not management imposed)
Coding: Tools

Atlassian JIRA
– Issue tracking and project tracking
– EDH: every change must have a JIRA
• Process should be as lightweight as possible

Atlassian GreenHopper
– Agile project management (Scrum)
– EDH: 2-week sprints

Atlassian Confluence
– Documentation (WIKI style)
CERN
AIS
Coding: Tools (2)

Atlassian Crucible
– Online code reviews
– EDH: Every production line of code must be reviewed

Atlassian FishEye
– Browse version control repository (CVS, SVN)
– Real-time notifications of code changes
– Web-based reporting, visualisation and code sharing

Atlassian Bamboo
– Continuous integration

Atlassian Clover
– Java code coverage metrics
CERN
AIS
Coding: Tools (3)
CERN
AIS
Testing
Bugs
 Standard Software: 25 bugs per 1000 lines of program.
Good Software: 2 errors per 1000 lines.
Space Shuttle Software: < 1 errors per 10000 lines.
Example Handy (Cellular Phone):
200 000 lines of program: up to 600 errors.
Windows-95: 10 Mill. lines: up to 200 000 errors.

Sept 24 2004 – Jpeg buffer overrun bug in MS windows
“an attacker who successfully exploited this vulnerability could take
complete control of an affected system, including installing programs;
viewing, changing, or deleting data; or creating new accounts with full
privileges”
Defects
 Testing ≠ Debugging
You may have zero bugs, but s/w may not meet requirements, scale,
respond-in time…
CERN
AIS
Testing

Software testing is an art : requires a
tester's creativity, experience and
intuition, together with proper techniques.

Most of the testing methods and practices
are not very different from 20 years ago.

Testing is more than just debugging.

Testing is expensive. Automation helps.

Complete testing is infeasible. Tradeoff

Testing, while essential, may not be the
most effective method to improve
software quality.
CERN
AIS
What do you test?

Correctness testing

Security testing

Reliability testing

Stress testing

Scalability testing

Performance testing

Usability testing
CERN
AIS
And after that?
Software Maintenance

80% of lifetime of software

The Legacy Crisis

The relative cost for maintaining
software and managing its evolution now
represents more than 90% of its total cost.
 Example costs
– Annual software maintenance cost in USA has
been estimated to be more than $70 billion
– Y2K : $8.38 billion US dollar, $90 million for
Nokia

CERN
AIS
50% of their time is spent in the process of
understanding the code!!!
Types of Maintenance

Corrective Maintenance (21%)
– A process that includes diagnosis and correction of
errors.

Adaptive Maintenance (25%)
– Activity that modifies software to properly interface with
a changing environment (hardware and software).

Perfective Maintenance (50%)
– Activity for adding new capabilities, modifying existing
functions and making general enhancements.
– This accounts for the majority of all effort expended on
maintenance.

Preventive Maintenance (4%)
– Activity which changes software to improve future
maintainability or reliability or to provide a better basis
for future enhancements.
– Still relatively rare.
CERN
AIS
Your challenge !

Come in the 9% of projects on time &
in budget

Engineer your software (design,
review & maintenance in mind)

Control the spiraling IT costs &
improve the reputation of the
industry
CERN
AIS
Thank You
For More Information
E-mail:
[email protected]
CERN
AIS