Transcript Software Maintenance - Center for Software Engineering

Measurement and Estimating Models
for Software Maintenance Workshop
(COCOMO Forum)
November 2011
Approved for Public Release, Distribution A
Workshop Goals
• Three goals for workshop:
– Expose members of the cost estimating
community to recent Army software maintenance
study findings and recommendations
– Gather feedback from the community on our
software maintenance WBS and initial influence
factor analysis
– Build consensus on what the most important
factors are via Delphi survey
• Build a community of interest in
maintenance that supports development
of new measures and models for
improved software maintenance
budgeting, estimation and management
Approved for Public Release, Distribution A
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next steps
• WBS review
• Influence factor review
• Delphi survey
• Roundtable discussions
• Summary and break
Approved for Public Release, Distribution A
Background
• The changing defense environment has placed
a renewed emphasis on the performance of
U.S. Army software maintenance, sustaining
engineering, and operational support efforts
• Accurate and objective cost estimates are
required to ensure that sufficient resources
are available to execute the work required to
keep systems operational and mission
capable
• To develop accurate estimates, the Army has
been working collaboratively with the Air
Force and the Navy to collect and analyze
past cost performance data and build a
software maintenance cost database
Approved for Public Release, Distribution A
Goals of Army Maintenance Study
• Goal - provide the Army with objective
decision information to accurately estimate,
budget, and allocate the software
maintenance, sustaining engineering, and
operational support resources (collectively
referred to as software maintenance
resources) required to meet evolving mission
and service affordability requirements
– Three years of effort to date focusing on the work
done and issues experienced by Army and Air Force
life cycle support centers
• Findings to-date seem universal across the
service weapons systems community
Approved for Public Release, Distribution A
Information Requirements
• Accurate estimates of software
maintenance resources mainly for
weapons systems from a:
– Product Perspective
– Organizational Perspective
– Enterprise Perspective
– Project and Release Perspective
• Objective portrayal of the dynamic
PDSS/PPSS maintenance environment
– Consistent with work being performed
Approved for Public Release, Distribution A
Information Requirements
• Consistent description of software
maintenance tasks
– Software Maintenance
– Software Sustaining Engineering
– Software Support Infrastructure –
Facilities
– Program/Project Management
• Quantitative understanding of the key
performance factors that influence
software maintenance resource
estimation
Approved for Public Release, Distribution A
Study Approach
• Direct interface with DoD and Industry
software support organizations performing
software maintenance tasks under contract
and at DOD Life Cycle Support Centers:
– Task identification - definition - allocations
– Historical cost data collection - budget submissions
and actuals
– Organizational context data - performance factors
• Stakeholder collaboration - protected sharing
of data, information, findings:
– Air Force, Army, Navy, DoD Agencies and potentially
allies
– Industry, Academia and Professional Groups
Approved for Public Release, Distribution A
Study Approach
• Develop approaches and mechanisms to
capture cost data and context information
architectures and data stores
• Perform data modeling - analysis:
– Parametric cost model calibration - CER
development
– Performance “meta” model - factor relationships
– Policy and decision information analysis
• Make recommendations for improvements
and increased affordability
– Develop software with maintenance in mind
– Smooth the transition from development to
maintenance, sustaining engineering, and
operational support
Approved for Public Release, Distribution A
Army Study Approach
• Investigate maintenance
–
–
–
–
–
What are the tasks?
Who does them?
What are the costs?
How they are estimated?
What impacts future costs?
• Understand
–
–
–
–
Current costs and risks
Current estimating practice
Current budgeting approach
What changes are needed
Approved for Public Release, Distribution A
Expected Outputs
• Context-driven software maintenance performance
model - configurable to product, organization, and
enterprise activities and scope
• Calibrated cost models for maintenance and cost
estimating relationships for ACAT 1A systems
– Multiple task categories
– Domain specific with validated cost relationships
• Software maintenance information architecture common service database
• Changes to policy and budgetary guidance needed
to support systems transitions and workload
growth
• Software maintenance reporting requirements like
the SRDR
• More accurate estimates for the POM
Approved for Public Release, Distribution A
Performance Meta-Modeling
No. of Releases
Done in Parallel
Complexity
Domain
Who – Contractor
Or Government?
Funding
Approach
Constraints and Influence Factors
Color
of
Money
•
•
•
•
Work Model
Maintenance workload
Sustaining engineering tasks
Infrastructure tasks
Program/Project Mgmt tasks
Approved for Public Release, Distribution A
Project
Factors
-
Process
People
Product
Summary of Study Findings
• Distribution of work
much different than
expected
- Eight Army and AF
Centers visited
• Testing is the major
- Over 100 interviews
maintenance activity
- Industry consulted
• Transition and transfer
• Findings
is done poorly
- Maintenance centers do
• Estimates and budgets
more than just updates
don’t cover all the work
and repairs
- Sustaining engineering
• Products
• Over 250 projects
surveyed
-
Reports, papers,
briefings, etc.
Web site
Initial maintenance cost
and quality database
-
Product field & user
support
Regression testing
• Efficiencies are needed
to cope with workload
Approved for Public Release, Distribution A
Army Study Findings
• Maintenance is done differently by
contractors than government shops
– Contractors develop to requirements,
government supports testing and field support
• As a minimum, four software releases are
being prepared by maintenance shops in
parallel during the calendar year
– Development release
- Fielded release
– To be fielded release
- Requirements release
• Work for software maintenance differs
from development – more test-directed
and constrained by environment
Approved for Public Release, Distribution A
Current Work Distribution
Notes
• About seventy percent
of their work involves:
- Maintenance
- Sustaining Engineering
- Independent V&V
• The other thirty percent
is devoted to other
tasks:
- Acquisition
management
- Software development
• Maintenance staff
includes both
government and inhouse contractor
personnel
Approved for Public Release, Distribution A
Testing is Primary Maintenance Activity
• As much as 55-70% of
the technical work
done during
maintenance supports
retesting and
qualifying the system
10%
20%
55%
• Testing is much harder
when developers fail to
transition and turnover
the needed set of
regression tests
• Support tasks are
performed to maintain
system integrity and
support field
operations
Approved for Public Release, Distribution A
Not All Of The Work Is Funded
• Estimates formulated
based on effort needed
to make updates and
repairs
• Other activities like
sustaining engineering
and testing not fully
covered
• Unfunded mandates like
Info Assurance not
adequately covered
• Small projects done on
LOE basis
• Licenses may need to
be funded by enterprise
• Resulting budgets force
maintenance staff to
play backlog reduction
games
• Shops make the
updates and repairs
that they can with
resources allocated
• Cost models & cost
estimating relationships
(CERs) used perpetuate
status quo
• Shortfalls in funding
need to be corrected
Approved for Public Release, Distribution A
Development System Not Ready for
Maintenance
• Transition requirements often waived,
avoided or delayed
- Facilities, tactical equipment and tools often
not available when needed
- Ownership rights to tools and special test
equipment often an issue
- Development SIL seldom transitioned for
maintenance
• Many aspects of “technical debt” are
not addressed
• Contractor often the only resource
available to maintain system
Approved for Public Release, Distribution A
On-Going Tasks
• Data collection
- Questionnaire/instruments
- Maintenance cost and
quality database
- Data administration,
protection and
management
• Architecture
development
- Information needs
- Data modeling
• Analysis
- Gap analysis
- Indicators
• Stakeholder Program
“Working one-on-one”
- Web site
- Case studies
- Outreach
▪ Collaborators
▪ Conferences
▪ Presentations
- Publications
- Working groups
• Project management
- Status and progress
reviews
Approved for Public Release, Distribution A
Going Forward
• Characterize software maintenance,
sustaining engineering, and
operational support
– Understand commonalities and differences
among services, domains, programs, and
maintenance organizations
– Clarify the differences between perceptions
and realities
• Coordinate the efforts of the Services
and industry to collect relevant
information
– Understand the characteristics of post
deployment software activities
Approved for Public Release, Distribution A
Collaborative Working Group
• Develop working group with government and
industry to explore the identified issues and
provide recommendations
– Requires collecting data to support the findings
– Data collection must be done hands-on to reduce
noise and increase confidence in results
• Develop viable software maintenance cost
estimation methods, models, and practices
• Work with academia and the DAU to improve
the education of the workforce on the
realities of software maintenance,
sustainment, and operational support
Approved for Public Release, Distribution A
In Summary
• We invite you to
participate in our
joint efforts
• We ask you to help
us populate our
software
maintenance cost
database
• We are interested in
any success stories
you would like to
share
Approved for Public Release, Distribution A
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next steps
• WBS review
• Influence factor review
• Delphi survey
• Roundtable discussions
• Summary and break
Approved for Public Release, Distribution A
WBS Revision
• In order to understand the factors that
impact maintenance cost, we are
developing a performance meta-model
• When discussing the meta-model, we
will summarize tasks around four
major activities:
– Software maintenance
– Software sustaining engineering
– Software support infrastructure & facilities
– Program/project management
Approved for Public Release, Distribution A
WBS Revision
Contractor
• Typically mimic software development
contracts
• Often require delivery to government sites
that handle distribution, certify the software,
perform field support and perform test and
evaluation
Government
• Perform maintenance with real operational
equipment and boots on the ground
• Maintenance is only part of the tasks they
perform (often fix hardware, do acquisition
support, etc.)
Approved for Public Release, Distribution A
WBS Revision
Software Maintenance:
1.1 Software maintenance
1.1.1
1.1.2
1.1.3
1.1.4
1.1.5
1.1.6
1.1.7
1.1.8
Release requirements
Release planning
Architecture analysis
Hardware defect repair
Software defect repair
Hardware enhancements
Software enhancements
Release integration & test
1.1.9 Release qualification & delivery
Approved for Public Release, Distribution A
WBS Revision
1.3 Independent test and verification
1.3.1
1.3.2
1.3.3
1.3.4
Test planning
Test preparation
Test conduct
Independent analysis &
verification
1.3.5 Certifications
1.5 Information assurance
1.5.1 Protection services
1.5.2 DIACAP
1.5.3 IAVA
Approved for Public Release, Distribution A
WBS Revision
Software Sustaining Engineering
1.2 Sustaining engineering
1.2.1
1.2.2
1.2.3
1.2.4
Analysis and studies
Emergency repairs
User training
External support
1.6 Acquisition support
1.7 Operational support
1.9 Field support
Approved for Public Release, Distribution A
WBS Revision
Software Support Infrastructure & Facilities
1.4 Product support
1.4.1 Configuration management
1.4.2 Quality assurance
1.4.3 Process management (peer reviews)
1.4.4 Supplier management
1.4.5 Security
1.8 Facility support
1.8.1 Maintenance facility sustainment
1.8.2 SIL sustainment
1.8.3 Equipment sustainment
1.8.4 Specialized test equipment and tools
1.8.5 Network operations and administration
Approved for Public Release, Distribution A
WBS Revision
1.11 Parts
1.12 Spares
1.13 Licenses
Program/Project Management
1.10 Management
1.10.1 Release management
1.10.2 Sustaining engineering management
1.10.3 Risk management
1.10.4 Measurement analysis
1.14 Contractual capabilities set FY (XX/XX)
1.15 Contractual system mission capability
1.16 Cost item general
Approved for Public Release, Distribution A
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next steps
• WBS review
• Influence factor review
• Delphi survey
• Roundtable discussions
• Summary and break
Approved for Public Release, Distribution A
Influence Factors
• Business Factors
– Extent of policy coverage - governance
– Low level technical & business decision
autonomy
– Diverse organizational task and activity
portfolios
– Program and domain characteristics
– Product and data rights
– Source and color of money
– Resourcing business models
– Estimation/budgeting approaches
– Information system capabilities
Approved for Public Release, Distribution A
Influence Factors
• Complexity factors
– Legacy software architectures
– Legacy software technologies
– Backfit security requirements
– Backfit safety and other certification
requirements
– System of system integration requirements
Approved for Public Release, Distribution A
Influence Factors
• Resource and task alignment factors
– Policies, budgets, resources, tasks and outputs
– Autonomous personnel and funding decisions
– Plan versus execution
– Alignment of task and funding models
– Management reserves (reduced allocations)
– Top-level expectations versus realities
– Overhead versus direct funded functions
– Amount of “technical debt”
Approved for Public Release, Distribution A
Influence Factors
• Execution factors
– Event-driven requirements and reprioritizations
– Short-term mission driven execution schedules
– Multiple customers – direct user involvement
– Multiple funding streams
– No. of releases being done in parallel (using
same resources)
– Backlog at start of release
– Uncertainty of planning parameters
– Organizational capability – flexibility - stability
Approved for Public Release, Distribution A
Size Drivers
Enterprise
Release
• No. of change
requests
• No. of repairs
– Defects by priority
and type
– No. of patches
• Backlog (technical
debt)
– Defect by priority,
age and type
• No. of COTS
packages updated
• No. of programs/
projects competing
for resources
• No. of releases
being supported by
program/project
• Stability of
releases over time
as measured by
change rate
• Stability of core
funding
Approved for Public Release, Distribution A
Maintenance Effort Multipliers
• People
– Analyst capability
– Programmer
capability
– Personnel continuity
– Applications
experience
– Platform experience
– Language and tool
experience
• Product
– Platform/domain type
– Application type
– Language type
– Requirements
volatility (change)
– Product complexity
– Data base size
– Required reliability
– Degree of reuse
– Documentation
match to needs
– Execution time
constrain
– Main storage
constrain
– Platform volatility
Approved for Public Release, Distribution A
Effort Multiplers
• Project
• Process
– Degree of
precedentedness
– Development
flexibility
– Architecture/risk
resolution
– Team cohesion
– Use of software tools
– Multi-site capability
– Required
development
schedule
– Resource dedication
– Acquisition method
– Development
method
– Development
standard
– Use of modern
programming
practices
– Process maturity
– Process volatility
(change)
• Other
Approved for Public Release, Distribution A
Effort Multipliers
• Because of time limitations, we will
not look at size and effort multipliers
for:
– Software sustaining engineering
– Software support infrastructure and
facilities
– Program/project management
• We will address these at next year
either by email or at another
conference like PSM
Approved for Public Release, Distribution A
Delphi Survey
• Goal – Determine what factors that you
believe have greatest impact on software
maintenance projects
• Scope:
– ID tasks which your maintenance shop performs
– ID factors to which their effort is most sensitive
– Schedule not considered because it is fixed in
maintenance
– Size of project considered because many
maintenance projects are small
• Can view influence factors as constraints
imposed on either enterprise or project
Approved for Public Release, Distribution A
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next steps
• WBS review
• Influence factor review
• Delphi survey
• Roundtable discussions
• Summary and break
Approved for Public Release, Distribution A
Roundtable Discussion
• What did you think
were the three
biggest influence
factors?
• What do you believe
are the three drivers
that the
maintenance effort
is most sensitive to?
Approved for Public Release, Distribution A
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next steps
• WBS review
• Influence factor review
• Delphi survey
• Roundtable discussions
• Summary and break
Approved for Public Release, Distribution A
In Summary
• We have summarized the
results of our study
• We have reviewed the
maintenance WBS that we
have developed
• We have conducted a
Delphi to identify the
influence factors and cost
drivers important to
estimating
• We will summarize findings
and present them on Friday
• If you wish to be on our
distribution, let us know
Approved for Public Release, Distribution A
This is why Don lives
in
Prescott, AZ
In