Transcript The Future of Systems Engineering and Software Processes

Computing and SE II
Chapter 2: A Brief History of SE
Er-Yu Ding
Software Institute, NJU
A View of 20th and 21st Century
Software Engineering
Barry Boehm
ICSE 2006 Keynote Address
May 25, 2006
[email protected]
http://sunset.usc.edu
Modified by Er-Yu Ding
Outline
• Motivation and Context
• A 20th Century View
• A 21st Century View
• Conclusions
– Timeless principles and aging practices
05/25/06
3
Motivation and Context
• Importance of historical
perspective
• We are losing our history
05/25/06
4
Why Software Projects Fail
- Average overrun: 89.9% on cost, 121% on schedule, with 61% of
content
05/25/06
5
Importance of Historical
Perspective
• Santayana half-truth:
– “Those who cannot remember the past are
condemned to repeat it”
• Don’t remember failures?
– Likely to repeat them
• Don’t remember successes?
–
–
–
–
05/25/06
Not likely to repeat them
Our objectives
Understanding SE more deeply
Organizing important
techniques/methods/methodologies of SE systematically
6
A Hegelian View of Software Engineering
Evolution
Theses
Engineer
Software
like
Hardware
Many defects
Formality,
Waterfall
Compliance
Software
Value-Add
PlanDriven
Software
Maturity
Models
COTS
Integrated
Sw-Systems
Engineering
Soft
SysE
Process Overhead
Syntheses
Scalability,
Risk Mgmt.
Productivity;
Reuse;
Objects;
Peopleware
Risk Mgmt.
Domain Engr.
Risk-Based
Agile/Plan
-Driven
Hybrids;
Model-Driven
Development
Value-Based
Methods;
Collaboration;
Global
Development;
Enterprise
Architectures
Autonomy; BioComputing
Scalability
Prototyping
Antitheses
05/25/06
Software
Differences,
Engineer
Shortages
Software
as Craft
1950's
1960's
Global
Systems
of
Systems
Time to Market,
Rapid Change
1970's
1980's
1990's
Agile
Methods
2000's
2010's
7
Outline
• Motivation and Context
• A 20th Century View
• A 21st Century View
• Conclusions
– Timeless principles and aging
practices
05/25/06
8
1950’s Thesis: Engineer Software Like
Hardware
• Hardware-oriented:
– Software applications: airplanes,
bridges, circuits
– Economics: Boehm supervisor, 1955
• “We’re paying $600/hour for that computer, and
$2/hour for you, and I want you to act accordingly.”
– Software Processes: SAGE (SemiAutomated Ground Environment)
• 1 MLOC air defense system, real-time, user-intensive
• Successful development of highly unprecedented
system
• Hardware-oriented waterfall-type process
05/25/06
9
The SAGE
Software
Development
Process
OPERATIONAL PLAN
MACHINE SPECIFICATIONS
- (Benington,
1956)
“We were
successful
because we
were all
engineers”.
05/25/06
OPERATIONAL SPECIFICATIONS
PROGRAM SPECIFICATIONS
CODING SPECIFICATIONS
CODING
PARAMETER TESTING (SPECIFICATIONS)
ASSEMBLY TESTING (SPECIFICATIONS)
SHAKEDOWN
SYSTEM EVALUATION
10
1950’s Thesis: Engineer Software Like
Hardware
• Software Techniques:
– Machine-Centric Application
– For special usage, research mainframes
• 1GL, 2GL
– Organization software with statements
• Coding tips for a optimized program are
important
05/25/06
11
1960’s, techniques progress
• Complier can optimizing statements well
• Organizing program with function
– Function was derived from λ calculus
• software get more complex
05/25/06
12
1960’s Antithesis: Software Is Not Like
Hardware
- Four Brooks factors plus two
• Invisibility: like the Emperor’s Magic Cloth
– Visibility and controllability of project is always a
topic of SE
• Complexity: we have see this last lecture
• Conformity: executed by computers, not
people
• Changeability: up to a point, then becomes
difficult
• Doesn’t wear out: different reliability,
05/25/06
13
1960’s Antithesis: Software Crafting
• Flexible materials, frequent changes as
above
• SW demand exceeded supply of engineers
– Cowboy programmers as heroes
– Code-and-fix process
– Hacker culture (Levy, 1984)
• Judge programmers by the elegance of their code
05/25/06
14
1960’s Progress and Problems
• Better infrastructure: OS, compilers, utilities
• Computer Science Departments
• Product Families: OS-360, CAD/CAM,
math/statistics libraries
• Some large successes: Apollo, ESS, BofA
check processing
• Problems: 1968, 1969 NATO Reports
05/25/06
– Failure of most large systems
– Unmaintainable spaghetti code
15
1970’s Antithesis: Formal and Waterfall
Approaches
• Structured Methods
– Structured programming (Bohm-Jacopini: GO TO
unnecessary)
• Formal programming calculus: Dijkstra, Hoare, Floyd
• Formalized Top-Down SP: Mills, Baker
• Waterfall Methods
– Code and fix too expensive (100:1 for large
systems)
– Precede code by design (De Marco SD, Jackson
JSD/JSP)
05/25/06
16
1970’s Antithesis: Formal
Methods
• “formal methods” branch that focused on
program correctness, either by
mathematical proof, or by construction via
a “programming calculus”
• Over the decade of the '60s theoretical
computer science achieved standing as a
discipline recognized by both the
mathematical and the computing
05/25/06 communities, and it could point to both
17
1970’s: Formal Methods
• Organized programs with
– Function
• Algorithm
– Three control logic
– Type and type check
• When compiler deal with functions well,
organized programs with
– Modularity
• This things gets importance with modularity
05/25/06
– Information hiding
18
1970’s: Formalized Top-Down
SP
• Function decomposition
– divide and rule
– Process modeling
• DFD
– Data modeling
• ERD
05/25/06
19
Increase in Software Cost-to-fix vs. Phase
(1976)
1000
Relative cost to fix defect
Larger Software Projects
500
IBM-SSD
200
GTE
•
100
50
•
80%
20%
•
SAFEGUARD
20
•
10
Smaller Software Projects
•
5
2
•
Median (TRW Survey)
•
1
Requirements
Design
Code
Development
test
Acceptance
test
Operation
Phase in Which defect was fixed
05/25/06
20
The Royce Waterfall Model
(1970)
- Explicit feedback
SYSTEM
REQUIREMENTS
SOFTWARE
REQUIREMENTS
- “Do it twice”
PRELIMINARY
PROGRAM
DESIGN
ANALYSIS
PROGRAM
DESIGN
PRELIMINARY
DESIGN
CODING
ANALYSIS
PROGRAM
DESIGN
TESTING
CODING
TESTING
OPERATIONS
USAGE
05/25/06
21
1970’s: Problems with Formal Methods
• Successful for small, critical programs
• Largest proven programs around 10 KSLOC
• Proofs show presence of defects, not
absence
– Defects in specification, proofs happen
• Scalability of programmer community
– Techniques require math expertise, $500/SLOC
– Average coder in 1975 survey:
05/25/06
• 2 years of college, SW experience
• Familiar with 2 languages, applications
22
Large-Organization HW/SW Cost Trends (1973)
100
80
% of
60
total cost
Hardware
40
Software
20
0
1955
1970
1985
Year
05/25/06
23
1970’s: Problems with Waterfall Model
• Overly literal interpretation of sequential
milestones
– Prototyping is coding before Critical Design Review
– Mismatch to user-intensive systems
• Heavyweight documentation hard to review,
maintain
– 7 year project with 318% requirements change
– Milestones passed but not validated
• Mismatch to COTS, reuse, legacy SW
– Bottom-up vs. top-down processes
05/25/06
24
A Hegelian View of Software Engineering
Evolution
Theses
Engineer
Software
like
Hardware
Many defects
Formality,
Waterfall
Compliance
Software
Value-Add
PlanDriven
Software
Maturity
Models
COTS
Integrated
Sw-Systems
Engineering
Soft
SysE
Process Overhead
Syntheses
Scalability,
Risk Mgmt.
Productivity;
Reuse;
Objects;
Peopleware
Risk Mgmt.
Domain Engr.
Risk-Based
Agile/Plan
-Driven
Hybrids;
Model-Driven
Development
Value-Based
Methods;
Collaboration;
Global
Development;
Enterprise
Architectures
Autonomy; BioComputing
Scalability
Prototyping
Antitheses
05/25/06
Software
Differences,
Engineer
Shortages
Software
as Craft
1950's
1960's
Global
Systems
of
Systems
Time to Market,
Rapid Change
1970's
1980's
1990's
Agile
Methods
2000's
2010's
25
1980’s Synthesis: Productivity, Reuse,
Objects
• Worldwide concern with productivity,
competitiveness
– Japanese example: autos, electronics, Toshiba
SW reuse
• Major SW productivity enhancers
– Working faster: tools and environments
– Working smarter: processes and methods
– Work avoidance: reuse, simplicity; objects
– Technology silver bullets: AI, transformations,
05/25/06
26
05/25/06
27
Tools, Environments, and Process
• 1970s, some requirements and design tools
developed, significant tool progress had been
mode in such areas as test tools and SCM tools
• Major emphasis in 1980s was on integrating tools
into support environments: IPSEs to CASEs
– Integrated Project Support Environment, Computer Aided
Software Engineering
– Requirements, design, planning and control, office support
– Formatted vs. formal specifications
• 1980s: Resource estimations and middleware is
underway
• Process-driven CASEs
05/25/06
– Use process knowledge as tool integration framework
– Some overkill in locking people into roles
28
Process: Industrial
Engineering
• Frederick W. Taylor : scientific management
05/25/06
– First. They develop a science for each element
of a man's work
– Second. They scientifically select and then
train, teach, and develop the workman
– Third. They heartily cooperate with the men so
as to insure all of the work is being done in
accordance with the principle of the science
which has been developed.
– Fourth. There is an almost equal division of the
work and the responsibility between the
29
Process
• Process execution support: “SW processes
are SW too”
– What’s good for products is good for processes
– Reuse, prototyping, architecture, programming
• Process compliance support: Standards and
CMMs
05/25/06
30
Reuse: Mechanical
Engineering
• In modular software, clearly the
"standardized units or dimensions" should
be standards such that software modules
meeting the standards may be conveniently
fitted together (without "trimming") to
realize large software systems. The
reference to "variety of use" should mean
that the range of module types available
should be sufficient for the construction of
a usefully large class of programs.
05/25/06
31
Reuse and Object Orientation
• 1950’s: Math routines, utilities
• 1960’s: McIlroy component marketplace,
Simula – 67
• 1970’s: Abstract data types, Parnas program
families
• 1980’s: Smalltalk, Eiffel, C++, OO methods,
reuse libraries
• 1990’s: Domain engineering, product lines,
UML, components, patterns, pub-sub
05/25/06
32
No Silver Bullet: Brooks
• Automated solutions are good for “accidental”
software problems
– Simple inconsistencies, noncompliance, inferences
• They do not do well on “essential” software
problems
– Changeability: adapting themselves to unanticipated
changes
– Conformity: working out everything the computer
needs to “know”
• Devoid of intuition, commonsense reasoning
05/25/06
– Complexity: integrating multiple already- complex
programs
33
People: The Most Important Factor
- SW engineering is of the people, by the people, and for the
people
• 1970’s: Weinberg Psychology of Computer
Programming
• 1980’s: DeMarco-Lister Peopleware
• 1990’s – 2000’s: Importance emphasized
in both Agile and CMM cultures
– Individuals and interactions over process and
tools
– People CMM, Personal Software Process,
Team Software Process
05/25/06
34
Dual 1990’s – Now: software
progress
• 1950s-1960s: machine-centric software
• 1960s-1980s: application-centric software
• 1990s-now: enterprise-centric software
–
–
–
–
Distributed, large-scale
Design patterns, Software architectures, UML
Requirement engineering
Middleware, Components, CDE, Product Line, COTS,
etc.
• Collaborative development environment, Commercial off the
shelf
• Future?
05/25/06
35
Dual 1990’s – Early 2000’s: software
techniques
• OO methods were strengthened
through
– OOP ( Java )OO Design (1995)OO
Analysis
– UML
– OO design principles
– Design patterns
05/25/06
36
Dual 1990’s – Early 2000’s Antithesis:
- Maturity Models and Agile Methods
• Predictability and Control: Maturity
Models——CMMI
– Reliance on explicit documented knowledge
– Heavyweight but verifiable, scalable
• Time to Market and Rapid Change: Agile
Methods
05/25/06
– Reliance on interpersonal tacit knowledge
– Lightweight, adaptable, not very scalable
37
Other 1990’s – Early 2000’s
Developments
• Y2K and reverse engineering
• Mature risk-driven software model: RUP
– Rational unified process
• Nature and importance of software
architecture
• Usability and Human-Computer Interaction
• COTS, open source, and legacy software
• Software as the primary competitive
05/25/06 discriminator
38
Open source and legacy
• Open source
– From its roots in the hacker culture of the
1960’s
• Collective code ownership
• Free software, data, computing access
– Free Software
– Linux
– Raymond’s “The Cathedral and the Bazaar”
• Legacy
05/25/06
– EAI (enterprise application integration), BPR
39
COTS: The Future Is Here
• Escalate COTS priorities for research, staffing, education
– Software is not “all about programming” anymore
– New processes
required
CBA Growth Trend in USC e-Services Projects
80
70
Percentage
60
*
50
40
30
20
10
0
1997
1998
1999
2000
2001
2002
Year
• CBA: COTS-Based Application
* Standish Group CHAOS 2000 (54%)
05/25/06
40
Mid-2000’s Synthesis: Risk-Driven Hybrid
Products and Process
• Increasing integration of systems
engineering and SW engineering
– Increasing trend toward “soft systems
engineering”
• Increasing focus on usability and value
– Fit the software and hardware to the people,
not vice versa
• Model-driven development and serviceoriented architectures
05/25/06
41
05/25/06
42
Outline
• Motivation and Context
• A 20th Century View
• A 21st Century View
• Conclusions
– Timeless principles and aging practices
05/25/06
43
The Future of Systems and
Software
• Eight surprise-free trends
1.
2.
3.
4.
5.
6.
7.
8.
Increasing integration of SysE and SwE
User/Value focus
Software Criticality and Dependability
Rapid, Accelerating Change
Distribution, Mobility, Interoperability, Globalization
Complex Systems of Systems
COTS, Open Source, Reuse, Legacy Integration
Computational Plenty
• Two wild-card trends
05/25/06
9. Autonomy Software
10.Combinations of Biology and Computing
44
Outline
• Motivation and Context
• A 20th Century View
• A 21st Century View
• Conclusions
– Timeless principles and aging practices
05/25/06
45
Timeless Principles (+) and Aging Practices
(-)
• From the 1950’s
+ Don’t neglect the sciences
+ Look before you leap (avoid premature
commitments)
- Avoid inflexible sequential processes
• From the 1960’s
05/25/06
+ Think outside the box
+ Respect software’s differences
- Avoid cowboy programming
46
Timeless Principles (+) and Aging Practices
(-)
• From the 1970’s
+ Eliminate errors early
+ Determine the system’s purpose
- Avoid top-down development and reductionism
• From the 1980’s
+ These are many roads to increased productivity
+ What’s good for products is good for processes
- Be skeptical about silver bullets
05/25/06
47
Timeless Principles (+) and Aging Practices
(-)
• From the 1990’s
+ Time is money and value to people
+ Make software useful to people
- Be quick, but don’t hurry
• From the 2000’s
05/25/06
+ If change is rapid, adaptability trumps
repeatability
+ Consider and satisfice all of your stakeholders’
value propositions
48
Timeless Principles (+) and Aging Practices
(-)
• For the 2010’s
+ Keep your reach within your grasp
+ Have an exit strategy
- Don’t believe everything you read
- “It’s true because I read it on the Internet”
05/25/06
49
The END
• Recommend papers
– A View of 20th and 21st Century Software
Engineering
– Fifty Years of Progress in Software Engineering
– Finding a History for Software Engineering
• Next Lecture
– Software developments activities
05/25/06
50