Transcript Kein Folientitel - u

Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Software Engineering Evidence
Dieter Rombach
([email protected])
Technical University of Kaiserslautern
Computer Science Department
Software Engineering Chair
Kaiserslautern, Germany
wwwagse.informatik.uni-kl.de
Fraunhofer Institute
for Experimental Software
Engineering (IESE)
Kaiserslautern, Germany
www.iese.fhg.de
Slide 0
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Contents
• Experimental Software Engineering
• Software Engineering Evidence
• Towards a Theory of Software Engineering Evidence
• Empirical Methods
• Existing Body of Knowledge
• Experimental Software Engineering in Kaiserslautern
• Practical Examples
• Agenda for Research, Tech Transfer & Teaching
• Outlook
Slide 1
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Experimental Software Engineering (1 of 3)
• Computer Science is one of the scientific base
disciplines for the “engineering of large (software)
systems”
Systems Engineering
Mechanical
Engineering
Software
Engineering
Physics
Computer
Science
Mathematics
Mathematics
Economics
… Psychology
Slide 2
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Experimental Software Engineering (2 of 3)
• All research in computer science & software
engineering should be application oriented
– basic research has a longer time-scale than applied
research
– non-application oriented research is only justified in
natural sciences!
Which testing technique
– application needs (in engineering) are defined via
G == f (P,C)
promises/guarantees
least constraints
95 defect
detection
• at
engineering
(e.g., staff
experience levels,
development technology such as OO/UML)
effectiveness for large
OO/UML-based systems?
• goals (e.g., reliability of >= 0.99)
Slide 3
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Experimental Software Engineering (3 of 3)
Experimental SE
• Software Engineering
comprises
– (formal) methods (e.g.,
techniques,
Experimental
Software modeling
Engineering
Formal
Methods
description languages)
– system technology (e.g.,
Process
System
- recognizes the nature architecture,
of our field
modularization,
Technology
Theory
OO,design
product lines)
(engineering
in a human-based
domain)
– process
technology
(e.g.,
- applies empirical studies
in order
to
life-cyle models, processes,
determine engineering management,
effects (f)measurement,
organization, planning QS)
in such a design domain
Empiricism
– empiricism (e.g.,
experimentation, experience
capture, experience reuse)
Slide 4
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Software Engineering Evidence (1 of 2)
• Empirical studies aim to capture quantitative evidence
regarding (P)
– product characteristics (definition, behavior)
“What is the complexity of a product?”
• “What is the performance of a system?”
•
G == f (P,C)
– process characteristics (definition, behavior)
“What is the inherent degree of parallelism?”
• “How much effort does it take?”
•
– process-product relationships
•
“How does design complexity affect test effort?”
• Issues
– How deterministic are studies?
– How easy/hard is it to test results via replication?
Slide 5
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Software Engineering Evidence
G == f (P,C)
• Traditional (quantitative) empirical evidence
– controlled experiments (variation in
C is controlled)
– case studies (C is a constant – reflecting some
environment)
Statistical
Significance
increases
• Questionnaires, Action Research, …. (mostly qualitative)
• Expert consensus (like in medicine)
Practical
Acceptance
increases
Slide 6
Scientists (aiming at testable cause-effect relations) prefer controlled expriments!
Practitioners (aiming at low-risk technology infusion) prefer case studies & expert consensus
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Towards a Theory of SE Evidence (1 of 3)
<var>
• Elements of Evidence (G == f (P,C))
– Review Technique P has effectiveness (80%, 50%) for (experienced,
inexperienced) developers
– Life cycle model P consumes resources (effort distribution model 1/model 2)
for (large/small x functional/OO design based) projects
• G:
any quality, cost & time model (e.g., reliability, effectiveness)
• P:
any SE process, method or technique (e.g., testing, design)
• C:
any project environment characteristic that defines the validity of
G
Allmapping
evidence
should
• f:
from (P,C
) onto Gbe packaged like (G x P x C )
in order
to be accessible for engineering
• ==: mapping
is defined empirically (because lack of physical laws!)
•
<var>:
some measure of certaintydecisions!
(e.g., +- X%)
This is also important for scientific publications!
Slide 7
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Towards a Theory of SE Evidence (2 of 3)
• Aggregation (vertical: P constant)
– to increase significance (i.e. reduce <var>)
– to increase generality, i.e. variation of C := C1 x C2 x C3 x
C4 by increasing the range of each Ci
• Significance increase
– experiment
replicationProblem
(e.g., inspection
area)
Complexity
(C coverage):
– formal aggregation depends on measure for significance (e.g. #
replications are incremented)
Maturing
(observations
-> laws -> theories) based on
• Variation
increase
studies(e.g.,
ALLONE
takes
too long!
– empirical
experiment variation
applications,
experiences,
…)
– complexity: simple coverage for 5 variables with 4 values each
requires “4 to the power of 5” = 1024 experiments???
Expert consensus
„when oberservations are considered laws“
– new hidden context variables HC appear in meta analysis!
is necessary!
• E.g., (G1, P, C) & (G2, P, C)  (G1/G2, P, C x (HV))
Slide 8
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Towards a Theory of SE Evidence (3 of 3)
• Aggregation (horizontal: P increases)
– to scale up to “larger” processes
P (e.g., Cleanroom
software development process)
– perform controlled experiments in “key elements” (e.g.,
unit inspections vs. testing)
– perform integration case studies
– acceptance of scaled-up evidence must be confirmed by
expert consensus
(organization
Complexity
Problem or
(P community)
size):
Performing controlled experiments on large processes
is practically impossible!
Expert consensus is required to accept evidence!
Slide 9
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Empirical Methods (1 of 3)
• Science in general involves
– modeling of software product & process artifacts
– empirical validation of hypotheses regarding their
characteristics & behavior in testable form
• Empirical foundation includes methods for
There exists
a comprehensive body of empirical methods!
• relating goals to measurements (GQM)
•
piggy-bagging empirical studies on real projects (QIP)
(ISERN
Workshops
& reuse (EF)
• organizing
empirical
observations for
Conference
inexperimental
Los Angeles,
• ISESE
specific activities
such as
design, data
analysis
week of 16 August 2004)
– importance of combining quantitative & qualitative
analysis
No excuse for not applying it!
Slide 10
Forum of the John von Neumann Society
Empirical Methods (2 of 3)
Generic Form of Empirical
SE Contributions:
G == f (P,C,HC)
Budapest, Hungary, 10 March 2004
•
For non-deterministic processes
•
Function f can only be determined
empirically
•
Variance of function f depends on
– sufficient # data points
– inclusion of relevant C
– stability of non-included C
•
important use of qualitative analysis:
assure that there are no non-identified
context factors
•
Variance may increase when
– repeated in/included with new
environment
– reason: hidden context factor (i.e.,
factor which was constant in one or
both environents) becomes
influential
– consequence: New empirical cycle
OR keep two different domains (i.e.
identical recommendation exists for
product lines!)
<var>
- G: any quality aspect such as
What
is
our
equivalent
to
physical
laws?
- P: process, technique, method,
reliability or maintainability
tool under investigation
- congitive „laws“?
- HC: hidden context factors
that only
- empirically
validated?
show up at certain levels of
- always context dependent?
generalization
- C: context factors such as
experience, lifecycle model
Slide 11
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Empirical Methods (3 of 3)
• Physics offers laws for electrical eng.
– precise
– not circumventable
Physical laws
• Computer Science & …. offer “laws” for SE
– empirically precise
Cognitive Laws
– circumventable (e.g., you may increase the complexity of
any system and it still may work!
•
is this really true?
– not if one includes evolution!
• what defines bounds?
– models that capture the negative consequences if you
exceed complexity bounds
Slide 12
(Com <= Co  Test_Effort <= To ! Test_Effort > To)
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Existing Body of Knowledge
• There exists more knowledge than we typically recognize
– mostly in terms of context-specific empirical observations
– rarely in terms of generalized “laws”
• There exist already more empirical “laws” than we typically
recognize
– book (Endres/Rombach, Addison, 2003)
– inspections
– design principles
• More studies need to be done
– repeat (with variation)
– generalize
Slide 13
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Fraunhofer IESE Series
Editorial Board includes
- Dave Parnas
- Dieter Rombach
- Ian Sommerville
- Marv Zelkowitz
Handbook capturing existing
body of knowledge
Students can learn
about existing body of knowledge
Practitioners can avoid negligance
of due dilligance
Slide 14
Additions are welcome
for next edition of book
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Experimental Software Engineering in Kaiserslautern (1 of 8)
• Empirical study driven software engineering research requires a
laboratory environment for
– in vitro: controlled student experiments
– In vivo: case studies on real industry projects
• “Mother” of all lab environments
– NASA SEL (GSFC, CSC & UMD)
•
combination of stake-holders
• long-standing trusting relationship
• sustained accomplishments
None of
the SEL success stories would have
– reduction of cost (50%) & increase in quality (no slipping
defects) without empirical studies!
happened
– sustained via empirically modeled “laws”
– regular use of “formal” methods (e.g., functional semantics in
Cleanroom)
– overcoming of development “factoids” (e.g., testing is best
defect detection approach)
Slide 15
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Experimental Software Engineering in Kaiserslautern (2 of 8)
Fraunhofer IESE
SME‘s
1
RL
State
2
1 2
SW&Sys.Eng
Large
Comp‘s
RL
(Bosch)
Univ. Kaiserslautern
DFG Res. Institutes
Slide 16
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Experimental Software Engineering in Kaiserslautern (3 of 8)
• Experimentation (EXP)
(Experimental designs, QIP/GQM/EF, Analysis techniques)
• Requirements and Usability Engineering (RUE)
(Requirements Engineering, Usability Engineering)
• Component-based Software Engineering (CBE)
(Component-based software development mainly in the field
of embedded and realtime systems)
• Software Product Lines (SPL)
(Architecture recovery, Architecture development &
evaluation, Scoping & modeling of domains & decisions)
Slide 17
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
ESE in Kaiserslautern (IESE, Research Competences) (4
of 8) • IT Security & Safety (ITS)
(Security Assessments, EF for Security, Automated checking
of infrastructures)
• Quality and Process Engineering (QPE)
(Goal-oriented measurement GQM, Descriptive process
modeling, Goal-oriented product/process assessment,
Subcontracting, COTS Acquisition, Risk Management)
• Systematic Learning & Improvement (SLI)
(QIP-based SPI planning & setup, EF buildup and
Organizational Learning)
• Certifiable Education & Training in SE (CET)
(Evaluation and Certification, Technology-enabled Learning,
Simulation-based learning and decision support)
Slide 18
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
ESE in KL (IESE, Maintain intellectual Control over System
Complexity) (5 of 8)
Cost
Avoiding deterioration
of system complexity
SE principles &
over time yields linear
technical engineering
cost curve!
processes
: actual
cost curve
: desired
cost curve
Product Functions
(Releases) over time
1
2
3
Slide 19
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
ESE in KL (IESE, Maximize Software Reuse) (6 of 8)
Cost
Product Line
Reuse
Exploiting reuse
potential with proper
apriori scoping (tool
support) yields
constant cost curve!
: PL cost
curve
: desired
cost curve
Product Functions
(Releases) over time
1
2
3
Slide 20
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
ESE in KL (IESE, Achieve Predictability for Software
Development) (7 of 8)
–
Cost
X
Management
Principles
Measurement-based
project management
models yield small
bandwidth prediction
capabilities!
X
X
X
X
X
: maximum
oscillation
X
X
: desired
cost curve
X
1
2
Product Functions
(Releases) over time (Size)
3
Slide 21
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
ESE in KL (IESE, Business Areas) (8 of 8)
Reliable Software for
Embedded Systems
•
•
•
•
•
•
Secure Software for
Infrastructure
Facilities and Services
• Telecommunication
• Providers
• Telematic Service Providers
Flexible Software for
IT-based Business
Processes
• Commerce (B2B, B2C)
• Medical Information Systems
• Financial Service Providers /
Insurance
• eGovernment
Software-based
Products and Services
• Software Companies
• Continuing Education & Training
Service Providers
• Consulting Companies
Automotive Industry and Railroad
Mechanical Engineering
Air and Space Industry
Industrial and Consumer Electronics
Chemical and Pharmaceutical Industry
Medical Technology
Slide 22
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Practical Examples (Inspections @ NASA) (1 of 4)
• NASA SEL Experience (see Basili, JSS, 1997)
– stepwise abstraction code reading vs. testing (Basili/Selby, TSE, 1987)
•
controlled experiment at UMD & NASA/CSC
•
effectiveness & cost (SAR > testing)
•
self-assessment (SAR > testing)
– stepwise abstraction code reading in regular SEL project
•
case study at NASA/CSC
•
SAR did not show any benefits
•
diagnosis: People did stewise abstraction code reading not as well as they
should have as they believed that testing would make up for their mistakes
– Cleanroom vs. standard SEL software development
•
controlled experiment at UMD
•
more effective application of reading, less effort and more schedule
adherence
– stepwise abstraction code reading in SEL Cleanroom projects
•
case study(ies) at NASA/SEL
•
improved failure rates (- 25%) and productivity (+30%)
Slide 23
Forum of the John von Neumann Society
Industrial
Success
Stories
Research
- Studies
- Methods
NASA/SEL:
NASA/SEL:
code inspect.
inspections
are effective
were
(- 30/50%
not lived!
code def)
Budapest, Hungary, 10 March 2004
…….
Defect
Selby Study Cleanroom
Cost study (inspection > Study (Basili
(Boehm)
testing)
et al)
Allianz:
requirem.
inspections
are effective
(- 50 % def)
PBR studies
(MD & IESE)
Functional Stepwise Ab- Cleanroom Perspective
Slide 24
Semantics straction Reading Process based Read.
(Mills)
(Mills, Basili) (Mills et al) (Basili et al)
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Practical Examples (Inspections @ IESE) (3 of 4)
•
Further IESE work on inspections
– investigation of effects in OO/UML environment (Laitenberger)
•
•
defined PBR for OO/UML (packaging of reading unit across views
controlled experiments
– students at UKL (SE class)
– PBR of requirements spec (UML) vs CBR
– effectiveness & cost (PBR > CBR)
•
replication of existing (see NASA/SEL) studies in varying contexts
(application domains, technology domains)
– variation of existing studies to address new questions
•
optimal effort for preparation phase in inspection process (exists as
demonstrated at Bosch; is used to manage inspection process)
•
Industrial relevance
– helped establish inspections with sustained success in several
companies (e.g., Allianz, Bosch)
– focus on inspections (with measurement-based feedback) matures
development organizations (e.g., Bosch unit with inspections went from
CMM1 to CMM 3 in one step!)
Slide 25
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Practical Examples (Design & Documentation @ IESE) (4 of
4) • IESE Studies on OO/UML (Briand, Bunse, Daly)
– operationalized good design principles such as coupling, information
hinding & cohesion
– hypotheses:
•
•
#1: “Good” OO designs are better understood
– measured by the correctness of answers to a set of questions
#2: Impact analysis on “good” OO designs is performed beter and faster
– measured by the time & correctness of all changes to perform a set of
given change requests
– controlled experiments at UKL
– 2 systems (“good”, “bad”); 2x2 factorial design
– results
•
•
all results significantly in favor of “good” design
students made important self-experience regarding a set of engineering
principles
Slide 26
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Agenda (for Research) (1 of 3)
• SE Research results require “some form of evidence”
– notations, techniques, methods & tools w/o evidence arenot
accepted (e.g., PhD theses)
Partners for EU-NoE
– collaboration with SE/CS experts
on „SE Evidence“
• Current research focus (SE techniques)
(SE, ESE)?
– notations like Java, UML, …
– architecture (specifically “product line architectures”)
– processes such as inspection & testing, design requirements
FROM
– quality (reliability, security, safety) perspective
– efficiency (cost, time) & effectiveness (“dependable SE”)
• Current research focus (ESE methods)
– Theory for “Evidence”
Partners for EU-STREP
on „Theory for Evidence“?
Slide 27
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Agenda (for Tech Transfer) (2 of 3)
• Apply “ESE” as transfer vehicle to create sustained improvements
• Use empirical studies to
– evaluate major process-product relations prior to offering to
industry (e.g., in vitro controlled experiments)
– method prototyping: Evaluate new methods together with
industry experts in order to provide ROI potential insight for
decision makers (e.g., Ricoh, Bosch, German Telecom)
– motivate candidate pilot project (developers & managers) with
semi-controlled training experiment
– evaluate pilot project (in vivo case studies) in order to adapt &
motivate
– continuously evaluate wide-spread use in order to motivate &
optimize
Without empirical evidence, no human-based process is lived!
Slide 28
This has contributed to the grwoing gap between research & practice!
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Agenda (for Teaching & Training) (3 of 3)
•
Learning is based on
•
•
•
reading
doing
experiencing
•
Human-based engineering activities depend heavily on the latter approaches
to learning
•
Teaching must reflect this
– first analysis, then construction
– perform “self-experience” experiments
•
At Technical University of KL/CS department
– 1st semester: NO programming (just reading & changing)
– SE experiments (final UG class)
•
•
•
#1: Unit inspection more efficient than testing
#2: Traceable design documentation reduces effort & risk of change
#3. Informal (req) documents can be inspected efficiently (> 90%)
– practical semester-long team projects with “data collection & process
improvements”
Slide 29
Forum of the John von Neumann Society
Budapest, Hungary, 10 March 2004
Outlook
• SE has the chance to become a respected engineering discipline
– automotive companies have more software than hardware
engineers (since 2000)
– mature software engineering includes empiricism (to create
evidence)
– system & service engineering require mature software
engineering
• Techncial University of Kaiserslautern
– has leading laboratory settings for empirically driven software
engineering research
– is ranked #6 in “Software & Systems Eng Research” world-wide
by JSS (12/03)
– maintains international network (USA, Asia, Europe)
– looks for international partners in SE & ESE (provided they
accept experimental framework!)
THANK YOU for your attention!
QUESTIONS?
Slide 30