Transcript Advanced Capability Maturity Model Concepts Presented to

Capability Maturity
Model Integration:
Implications for
Quality Assurance and
Testing
Presentation to SCQAA
20 June 2002
Rick Hefner, TRW
[email protected]
(310) 812-7290
Agenda
• Project History
• CMMI Structure
• Comparisons with SW-CMM v1.1, SE-CMM, and EIA/IS 731
• Assessment Methodology
• Transition Timelines and Strategies
• Implications for Quality Assurance Professionals
SCQAA - Rick Hefner
2
What is a CMM?
• Capability Maturity Model: A reference model of mature
practices in a specified discipline -- industry best practices
• CMMs differ by
– Discipline (software, systems, acquisition, etc.)
– Structure (staged versus continuous)
– How Maturity is Defined (process improvement path)
– How Capability is Defined (institutionalization)
• “Capability Maturity Model®” and CMM® are used by the
Software Engineering Institute (SEI) to denote a particular class
of maturity models
Capability Maturity Model®, CMM®, CMM Integration, and CMMI are service marks and registered trademarks of
Carnegie Mellon University
SCQAA - Rick Hefner
3
Commonly Used CMMs
Software CMM
staged
software development
System Engineering CMM
continuous
system engineering
System Engineering Capability Model
continuous
system engineering
Software Acquisition CMM
staged
software acquisition
System Security Engineering CMM
continuous
security engineering
Personal Software Process
staged
individual software
development
FAA-CMM
continuous
software engineering,
systems engineering,
and acquisition
IPD-CMM
continuous
integrated product
development
People CMM
staged
workforce
SPICE Model
continuous
software development
SCQAA - Rick Hefner
4
What is the Problem?
•
•
•
•
Different structures, formats, terms, ways of measuring maturity
Causes confusion, especially when using more than one CMM
Hard to integrate them in a combined improvement program
Hard to use multiple models in supplier selection
Systems
Engr
CMM
Software
CMM
IPD
CMM
FAA
iCMM
People
People
CMM
CMM Software
Acq
ZZZ
Security
CMM
Systems
CMM
Systems
Security
Engr
Engr CMM
CMM
SCQAA - Rick Hefner
CMMI
5
The CMMI Project
• DoD sponsored collaboration
between industry, Government, academia
• Over 100 people involved
•
•
•
•
•
•
•
•
•
•
•
•
•
•
U.S. Army, Navy, Air Force
Federal Aviation Administration
National Security Agency
Software Engineering Institute
ADP, Inc.
AT&T Labs
BAE
Boeing
Computer Sciences Corporation
EER Systems
Ericsson Canada
Ernst and Young
General Dynamics
Harris Corporation
•
•
•
•
•
•
•
•
•
•
•
•
•
Honeywell
KPMG
Litton
Lockheed Martin
Motorola
Northrop Grumman
Pacific Bell
Q-Labs
Raytheon
Rockwell Collins
Sverdrup Corporation
Thomson CSF
TRW
SCQAA - Rick Hefner
6
CMMI Models
Source Models
• Capability Maturity
Model for Software V2,
draft C (SW-CMM V2C)
• EIA Interim Standard
731, System Engineering
Capability Model (SECM)
• Integrated Product
Development Capability
Maturity Model, draft
V0.98 (IPD-CMM)
CMMI Models
Disciplines
– systems engineering
– software engineering
– integrated process and product
development (IPPD)
– supplier sourcing
– combinations of the above
Representations
– staged
– continuous
SCQAA - Rick Hefner
7
Maturity Levels
• Each level is a layer in the foundation
for continuous process improvement
Level 5
Optimizing
Focus on process
improvement
Level 4
Quantitatively
Managed
Process measured
and controlled
Process characterized
for the organization and
is proactive
Process characterized for
projects and is often
reactive
Process unpredictable,
poorly controlled and
reactive
Level 3
Defined
Level 2
Managed
Level 1
Performed
SCQAA - Rick Hefner
8
CMMI Process Areas
Level
Focus
Process Areas
5 Optimizing
Continuous
process
improvement
Causal Analysis and Resolution
Organizational Innovation and Deployment
4 Quantitatively Quantitative
management
Managed
3 Defined
2 Managed
Process
standardization
Basic
project
management
Quantitative Project Management
Organizational Process Performance
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Risk Management
Decision Analysis and Resolution
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Requirements Management
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
1 Performed
SCQAA - Rick Hefner
9
Process Area Group Relationships
Process Management
empower
Organizational Process Focus
Organizational Process Definition
Organizational Training
Organizational Process Performance
Organizational Innovation and Deployment
analyze
standardize processes
Project Management
Support
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Integrated Project Management
Risk Management
Quantitative Project Management
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Decision Analysis and Resolution
Causal Analysis and Resolution
analyze
Engineering
employ
Requirements Development
Requirements Management
Technical Solution
Product Integration
Verification
Validation
SCQAA - Rick Hefner
measure & assist
10
Common Features For Each Process Area
• Commitment to Perform
– Establish an Organizational Policy
• Ability to Perform
– Plan the Process
– Provide Resources
– Assign Responsibility
– Train People
– Establish a Defined Process
• Directing Implementation
– Manage Configurations
– Identify and Involve Relevant Stakeholders
– Monitor and Control the Process
– Collect Improvement Information
• Verification
– Objectively Evaluate Adherence
– Review Status with Higher-Level Management
SCQAA - Rick Hefner
11
CMMI Staged Model Structure
Maturity Levels
Process Area 1
Common Features
Activities
Commitment
Performed
to Perform
Subpractices
Subpractices
Subpractices
Amplifications
Amplifications
Amplifications
Elaborations
Elaborations
Elaborations
Process Area n
Generic Goals
Specific Goals
Specific
Specific
Specific
Practices
Practices
Practices
...
Process Area 2
Ability
to Perform
Directing
Implementation
Verification
Generic
Specific
Specific
Practices
Practices
Practices
Subpractices
Subpractices
Subpractices
Amplifications
Amplifications
Amplifications
Elaborations
Elaborations
SCQAA - Rick Hefner
Elaborations
12
Required, Expected, Informative
Maturity Levels
Process Area 1
Common Features
Activities
Commitment
Performed
to Perform
Subpractices
Subpractices
Subpractices
Amplifications
Amplifications
Amplifications
Elaborations
Elaborations
Elaborations
Process Area n
Generic Goals
Specific Goals
Specific
Specific
Specific
Practices
Practices
Practices
...
Process Area 2
Ability
to Perform
Required
Needed to satisfy
CMMI
Directing
Implementation
Generic
Specific
Specific
Practices
Practices
Practices
Subpractices
Subpractices
Subpractices
Amplifications
Amplifications
Amplifications
Elaborations
Elaborations
SCQAA - Rick Hefner
Elaborations
Verification
Expected
Typical practices to
meet goals;
Used as a starting
point in assessments
Informative
Ideas to consider
13
SW-CMM v1.1 vs. CMMI
Process Areas
LEVEL 5
OPTIMIZING
LEVEL 4
MANAGED
Defect Prevention
Technology Change Mgmt
Process Change Management
Causal Analysis and Resolution
Organizational Innovation & Deployment
Quantitative Process Mgmt
Software Quality Mgmt
Organizational Process Performance
Quantitative Project Management
Organization Process Focus
Organization Process Definition
Training Program
Integrated Software Mgmt
Organization Process Focus
Organization Process Definition
Organizational Training
Integrated Project Management
Risk Management
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Decision Analysis and Resolution
Software Product Engr
LEVEL 3
DEFINED
LEVEL 2
REPEATABLE
Intergroup Coordination
Peer Reviews
Requirements Management
Requirements Management
Software Project Planning
Project Planning
Software Project Tracking & Oversight Project Monitoring and Control
Software Subcontract Mgmt
Supplier Agreement Management
Software Quality Assurance
Product & Process Quality Assurance
Software Configuration Mgmt
Configuration Management
Measurement and Analysis
14
SW-CMM v1.1 vs. CMMI
Common Features
SW-CMM v1.1 Common Features
Commitment to Per form
Est ablish an Organizational Policy
Abi li ty to Per form
Prov ide Resources
Assign Responsibility
Train People
Activities Per form ed
Plan the Process
Perf orm the Process
Monitor and Control t he Process
CMMI Common Features
Commitment to Per form
Est ablish an Organizational Policy
Abi li ty to Per form
Plan the Process
Prov ide Resources
Assign Responsibility
Train People
(Specif ic Practices)
Dir ecting Impl ementati on
Collaborat e with Rel. Stakeholders
Manage Conf igurat ions
Monitor and Control t he Process
Measurement & Analysi s
Measure the Process
Analy ze the Measurements
Ver ifyi ng Impl ementati on
Review with Org. Management
Review with Project Management
Object iv ely Verif y Adherence
Ver ifyi ng Impl ementati on
Review with Higher-Lev el Mgt
Object iv ely Verif y Adherence
15
SE-CMM and SECM (EIA/IS 731) vs. CMMI
Common Features - Engineering
SE-CMM
SECM
Engineering
Technical
PA06 Understand
Customer Needs
and Expectations
PA02 Derive and Allocate
Requirements
PA03 Evolve System
Architecture
PA01 Analyze Candidate
Solutions
PA05 Integrate System
PA07 Verify and Validate
System
PA04 Integrate
2.3
Disciplines
1.1 Define
Stakeholder and
System Level
Requirements
1.2 Define Technical
Requirements
1.3 Define Solution
1.4 Assess and
Select
1.5 Integrate System
CMMI
Engineering
Requirement Mgmt
Requirement
Development
Technical Solution
Decision Analysis
and Resolution
Product Integration
1.6 Verify System
Verification
1.7 Validate System
Validation
SCQAA - Rick Hefner
16
SE-CMM and SECM (EIA/IS 731) vs. CMMI
Common Features - Project Management
SE-CMM
SECM
Project
PA12
PA11
PA10
PA09
PA08
Management
2.1 Plan and Organize
Plan Technical
2.2 Monitor and
Effort
Control
Monitor and
2.3 Integrate
PA04
Control
Disciplines
Technical Effort
2.4 Coordinate with
PA18
Manage Risk
Suppliers
Manage
2.5 Manage Risk
Configurations
2.6 Manage Data
Ensure Quality
2.7 Manage
Configurations
L2 GP
2.8 Ensure Quality
*
*
NotSCQAA
in SE-CMM
- Rick Hefner
CMMI
Project Management
Project Planning
Project Mon and Con
Supplier Agree Mgt
Integrated Proj Mgt
Risk Management
Support
Configuration Mgt
Proc and Prod QA
Meas and Analysis
17
SE-CMM and SECM (EIA/IS 731) vs. CMMI
Common Features - Process Management
SE-CMM
Organization
PA13 Define Organization’s
SE Process
PA14 Improve
Organization’s SE
Process
PA17 Provide Ongoing
Knowledge and Skills
PA15 Manage Product Line
Evolution
PA16 Manage SE Support
Environment
2.4
PA18 Coordinate With
Suppliers
SECM
Environment
3.1 Define and
Improve the SE
Process
3.2 Manage
L4 GP
Competency
3.3 Manage
L4 GP
Technology
L5 GP
3.4 Manage SE
Support
Environment
SCQAA - Rick Hefner
CMMI
Process Management
Org Process Focus
Org Process Def
Organizational Tng
Quant Project Mgt
Org Process Perf
Causal An & Res
Org Innovation and
Deployment
(IPPD Extension)
Organizational
Environment
Integration
18
Summary of Changes
• Organizations that are currently using SW-CMM v1.1, SE-CMM,
or EIA/IS 731 should be able to smoothly transition
• Will require additional infrastructure
– Identification and involvement of relevant stakeholders
– Configuration management of intermediate work products
• Will require additional processes
– Measurement and Analysis
– Decision Analysis and Resolution
– Requirements Development
– Risk Management
• Additional focus on engineering processes
Software organizations adopting CMM-based improvements
for the first time should use the CMMI
SCQAA - Rick Hefner
19
Appraisal Requirements for CMMI (ARC)
• Similar to the current CMM Appraisal
Framework (CAF) V1.0
– A guide to assessment method developers
• Specifies the requirements for classes of
assessment methods
– Class A: Full, comprehensive assessment methods
– Class B: Initial, incremental, self-assessments
– Class C: Quick-look
• Method developers can declare which class their method fits
• Method buyers will understand the implications of the method
proposed
SCQAA - Rick Hefner
20
Standard CMMI Assessment Method for
Process Improvement (SCAMPI) v1.1
• Similar to CBA IPI method
• Led by authorized Lead Assessor
• Team of 4-9 trained assessors
• Tailorable to organization and
model scope
• Products:
– SCAMPI Method Description Document
– Maturity questionnaire(s), electronic work aids, templates
The method was updated to version 1.1 to improve efficiency
SCQAA - Rick Hefner
21
CMMI Transition Plan
Development Phase
• Development of CMMI products
• Verification and validation of CMMI products
Transition Phase
• Approval of initial CMMI products for
public release
• Evidence of sufficient use
• Transition planning to help organizations
use CMMI products
Sustainment Phase
• Upkeep and continuous improvement
of the product suite
• Additional evidence of adoption and use
Pilots
V1.0
V0.2
Aug 1999 Aug 2000
V1.1
Dec 2001
SW-CMM, EIA 731 phased out
Aug 2003
SCQAA - Rick Hefner
22
Role of the Quality Assurance Professional
• QA has a defined role in every process area, and a process area
of their own
SCQAA - Rick Hefner
23
Generic Practices
• GP 2.9 (VE1) Objectively Evaluate Adherence
– Objectively evaluate adherence of the process and the work
products and services of the process to the applicable
requirements, objectives, and standards, and address
noncompliance.
• Quality assurance must be performed in all processes,
work products, and services
– Engineering
– Support
– Project Management
– Organizational process management
SCQAA - Rick Hefner
24
Process and Product Quality Assurance - 1
• Purpose
– Objectively review activities and work products for their adherence to
applicable requirements, process descriptions, standards, and
procedures, and communicate the results to staff and management.
• P&PQA ensures planned processes are implemented;
Verification ensures the specified requirements are satisfied.
• QA can be embedded in the process, but:
– Objectivity must be maintained
– All QA performers must be trained
– An independent reporting channel must be maintained
• QA should participate in establishing the plans, processes,
standards and procedures to ensure they:
– Fit the project’s needs
– Will be useable for performing quality assurance evaluations.
SCQAA - Rick Hefner
25
Process and Product Quality Assurance - 2
SG 1 Objectively Evaluate Processes and
Work Products
SP 1.1 Objectively Evaluate Processes
Objectively evaluate the
designated performed processes
against the applicable process
descriptions, standards and
procedures.
SP 1.2 Objectively Evaluate Work
Products and Services
Objectively evaluate the
designated work products and
services against the applicable
process descriptions, standards,
and procedures.
• Projects must designate which
processes and work products
they intend to audit
– What processes and work products
drive overall quality?
– Where is insight into quality the most
valuable?
– Assessor typically expect all
processes to be audited
SCQAA - Rick Hefner
26
Process and Product Quality Assurance - 3
SG 2 Provide Objective Insight
SP 2.1 Communicate and Ensure
Resolution of Noncompliance
Issues
Communicate quality issues and
ensure resolution of
noncompliance issues with the
staff and managers.
• Requires a independent
reporting channel for
noncompliance issues
– Makes senior management
responsible for adjudicating quality
versus schedule conflicts
SP 2.2 Establish Records
Establish and maintain records of
the quality assurance activities.
SCQAA - Rick Hefner
27
Process and Product Quality Assurance - 4
GG 2 Institutionalize a Managed Process
GP 2.1 (CO 1) Establish an Organizational Policy
GP 2.2 (AB 1) Plan the Process
GP 2.3 (AB 2) Provide Resources
• Quality assurance activities
must be fully supported by
the project and organizational
infrastructure
GP 2.4 (AB 3) Assign Responsibility
GP 2.5 (AB 4) Train People
GP 2.6 (DI 1) Manage Configurations
GP 2.7 (DI 2) Identify and Involve Relevant
Stakeholders
GP 2.8 (DI 3) Monitor and Control the Process
GP 2.9 (VE 1) Objectively Evaluate Adherence
GP 2.10 (VE 2) Review Status with Higher-Level
Management
SCQAA - Rick Hefner
28
Verification - 1
• Purpose
User needs
– Assure that selected work products
meet the specified requirements
(e.g., peer reviews, testing)
System requirements
• Key Issues
– Recognizing verification is an
incremental process that occurs
throughout the development of the
product and work products
» Verification of the Requirements
» Verification of the evolving work
products
» Verification of the completed
product
– Identifying where verifications will
occur (as part of program strategy)
Code
testing
SCQAA - Rick Hefner
Software requirements
Software design
Code peer
review
Software code
Test cases
Test results
29
Verification - 2
SG 1 Prepare for Verification
Preparation for verification is conducted.
SP 1.1 Select Work Products for
Verification
Select the work products to be verified
and the verification methods that will
be used for each.
SP 1.2 Establish the Verification
Environment
Establish and maintain the
environment needed to support
verification.
SP 1.3 Establish Verification Procedures
and Criteria
Establish and maintain verification
procedures and criteria for the selected
work products.
• Projects identify (and document)
the work products they intend to
verify
• Verification environment typically
refers to a test bed
– Could also imply the facilities and
tools needed for peer reviews
• “Procedures” are not typically
called out in the CMMI – implies
the need for more detailed
planning
SCQAA - Rick Hefner
30
Verification - 3
SG 2 Perform Peer Reviews
Peer reviews are performed on selected work
products.
SP 2.1 Prepare for Peer Reviews
Prepare for peer reviews of selected
work products.
• Different verification methods:
peer review, inspection,
demonstration, test, analysis
• Preparing for peer review
includes identification of:
SP 2.2 Conduct Peer Reviews
Conduct peer reviews on selected work
products and identify issues resulting
from the peer review.
SP 2.3 Analyze Peer Review Data
Analyze data about preparation, conduct,
and results of the peer reviews.
– Staff involved, roles
– Review criteria
– Schedule
• Peer review data are analyzed to
improve review effectiveness
SG 3 Verify Selected Work Products
Selected work products are verified against their
specified requirements.
SP 3.1 Perform Verification
Perform verification on the selected work
products.
SP 3.2 Analyze Verification Results and
Identify Corrective Action
Analyze the results of all verification
activities and identify corrective action.
SCQAA - Rick Hefner
–
–
–
–
–
–
Size of the product
Composition of review team
Type of peer review
Preparation time per reviewer
Length of the review meeting
Number of defects found, type and
origin of defect, etc.
31
Verification - 4
GG 2 Institutionalize a Managed Process
GP 2.1 (CO 1) Establish an Organizational Policy
GP 2.2 (AB 1) Plan the Process
GP 2.3 (AB 2) Provide Resources
• Verification activities must be
fully supported by the project
and organizational
infrastructure
GP 2.4 (AB 3) Assign Responsibility
GP 2.5 (AB 4) Train People
GP 2.6 (DI 1) Manage Configurations
GP 2.7 (DI 2) Identify and Involve Relevant
Stakeholders
GP 2.8 (DI 3) Monitor and Control the Process
GP 2.9 (VE 1) Objectively Evaluate Adherence
GP 2.10 (VE 2) Review Status with Higher-Level
Management
SCQAA - Rick Hefner
32
Implications for the
Quality Assurance Professional
• More companies are adopting the CMMI (and CMM)
– Required by customers
– Improves predictability and efficiency; reduces rework and cost
– A competitive discriminator
• The CMMI/CMM requires a significant QA presence
– Involvement in all projects, all processes
– Significant support required to satisfy PPQA process area
• CMMI/CMM provides an opportunity to educate the community
on the proper role of QA
SCQAA - Rick Hefner
33
Summary
• CMMI is a natural evolution in industry maturity
• Organizations that are currently using SW-CMM v1.1, SE-CMM,
or EIA/IS 731 should be able to smoothly transition
• Quality assurance professionals have an important role in the
CMMI
See http://www.processimprovement.com
for more information and resources
SCQAA - Rick Hefner
34