7/31/2007

Standards & Assessments CMMI, ISO 9000, TL9000

Sources: ASQ CSQE Primer Introduction to CMMI CMMI Distilled SE 652 2007_7_31_CMMI_Software_Quality.ppt

1

August 4 Class

CMMI Introduction & Configuration Management Appraisal ISO 9000/TL-9000 Due today (31-July): Cycle 2 Design & Code, hand off to System Tester System Test Plan Inspected & Baselined Project notebook updates including inspection records, meeting minutes, etc.

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

2

Topics

Audits & Assessments CMM / CMMI & SCAMPI ISO 9000: ISO 9001:2000, ISO 9000-3:1997, TickIT Q9000, TL9000 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

3

Capability Maturity Model (CMM)

Created in 1987 by Software Engineering Institute (SEI) 5 level model based on proficiency in Key Process Areas (KPAs) Migrating to Capability Maturity Model Integration (CMMI) Three source models: – CMM for Software – Systems Engineering Capability model – Integrated Product Development CMM CMMI v1.1

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

4

What is it?

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

5

Why Would I want one?

Required – Contractual – Senior Management Decree (e.g. ROI of 7 to 1) Sales Tool Want to improve 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

6

Schedule Example

Drop Page Fields Here

Organization 1 Project Schedule Performance

4 Count of Months Late 3 2 1 0 1 2 3 Months Late 4 5 7 Drop Series Fields Here 2 1 0 4 3 5 Count of Months Late 4 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

Drop Page Fields Here

Organization 2 Project Schedule Performance

5 Months Late 6 Drop Series Fields Here 7

Process Capability

Ability of a process to produce planned results • Predictable • Measureable 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

8

Process Models

CMMI is model based Model = structured collection of elements that describes characteristics of effective processes Process Area = cluster of related practices that when performed collectively, satisfy a set of goals considered important for making significant improvement in that area Processes selected are those proven by experience to be effective (i.e. best practices, practical knowledge from previous endeavors) Notes: A process area is not a process A model is not a process

models show what to do, not how to do it!

Philosophy “All models are wrong, some are useful” –

George Box

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

9

CMMI Models

Model Options: Software Engineering (SW) Systems Engineering + Software Engineering (SE/SW) Systems Engineering + Software Engineering + Integrated Process & Product Development (SE/SW/IPPD) … + Supplier Sourcing (SE/SW/IPPD/SS) Representation Options: Staged (Maturity Levels) Migration from CMM to CMMI Continuous (Capability Levels) Migration from EIA/IS-731 to CMMI Recommended order for process improvements, but not prescribed … 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

10

Levels

Zero – Ad Hoc One – Doing it (in Continuous, Ad Hoc in Staged) Two – Process performed for individual projects Three – Process focus at organizational level Four – Projects and processes are quantitatively managed Five – Projects and processes being optimized based on performance data & results 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

11

Representations Revisited

Continuous Model – 25 Process Areas each assessed at level 0-5 Configuration Mgmt = capability level 3 Risk Mgmt = not done (capability level 0) Requirements Mgmt = capability level 2 – Result can be presented as a Kiviat chart Staged Model – 25 Process Areas assigned to each of 4 Maturity Levels (see next slide) – Result is a grade (1-5) 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

12

7/31/2007

Staged Representation Maturity Levels (MLx)

5 Optimizing Focus on process improvement 4 Quantitatively Managed Process measured & controlled 3 Defined Process characterized by organization is proactive 2 Managed Process characterized for project & often reactive 1 Initial Process unpredictable, poorly controlled & reactive SE 652 2007_7_31_CMMI_Software_Quality.ppt

13

Staged Representation Process Area Mapping to Maturity Levels

5. Optimizing 4. Quantitatively Managed 3. Defined 2. Managed 7/31/2007 1. Initial Continuous Process Improvement Organizational Innovation & Deployment Causal Analysis & Resolution Quantitative Management Organizational Process Performance Quantitative Project Management Process Standardization Basic Project Management Requirements Development Technical Solution Product Integration Verification Validation Organizational Process Focus Organizational Process Definition Organizational Training Risk Management Decision Analysis & Resolution Requirements Management Project Planning Project Monitoring & Control Supplier Agreement Management Measurement & Analysis Process & Product Quality Assurance Configuration Management

None

SE 652 2007_7_31_CMMI_Software_Quality.ppt

15

Continuous Representation Process Areas

Process Management –

Organizational Process Focus (OPF-3)

– – – –

Organizational Process Definition (OPD-3) Organizational Training (OT-3) Organizational Process Performance (OPP-4) Organizational Innovation & Deployment (OID-5)

Engineering – – – – – –

Requirements Management (REQM-2) Requirements Development (RD-3) Technical Solution (TS-3) Product Integration (PI-3) Verification (VER-3) Validation (VAL-3)

Project Management –

Project Planning (PP-2) Project Monitoring & Control (PMC-2)

Support –

Configuration Management (CM-2)

–

Process & Product Quality Assurance

– – – – – – –

Supplier Agreement Management (SAM-2) Integrated Project Management (IPM-3) Risk Management (RSKM-3) Integrated Teaming (IT-3) Integrated Supplier Management (ISM-3) Quantitative Project Management (QPM-4)

– – – –

(PPQA – 2) Measurement and Analysis (MA-2) Decision Analysis and Resolution (DAR-3) Organizational Environment for Integration (OEI-3) Causal Analysis and Resolution (CAR-5)

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

16

CMMI Assessment Cheat Sheet

Institutionalization –

Ingrained

Way of Doing Business that an organization follows

routinely

as part of its corporate culture Specific Goals –

Required

model component that describes the unique characteristics that must be present to satisfy the process area Specific Practice –

Expected

model component that is considered important to achieving the associated specific goal. The specific practices describe the activities expected to result in achievement of the specific goals of a process area.

(In continuous representation – every specific practice (SP) is associated with a CL, in staged – all SPs are treated equally)

Generic Goal – Required model component that describes the characteristics that must be present to satisfy the

institutionalization

of the processes that implement a process area Generic Practice –

Expected

model component that is considered important in achieving the associated generic goal. The generic practices describe the activities that are expected to result in achievement of the generic goal and contribute to the institutionalization of the processes associated with a process area.

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

17

CMMI Assessment Cheat Sheet (continued)

Managed Process: – Performed process planned & executed in accordance with policy – Employs skilled people – Adequate resources – Produces controlled outputs – Involves relevant stake holders – Monitored, controlled & reviewed – Evaluated for adherence to process description Defined Process: – Managed process tailored from the organizational standard processes – Maintained process description – Contributes work products, measures & other process info to organizational process assets Performed Process – Accomplishes needed work to produce work products – Specific goals of the process area are satisfied Establish & Maintain – Includes documentation & usage: • Planned • Documented & • Used 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

18

Configuration Management (CM) Assessment

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

19

DeMarco & Lister on Process

Organizations driving to be SEI Level 5 (at least level N+1) Standards are good, but … Most success centered around standard interfaces Mandating a “best practice” is a bad practice Process improvement is good, but process improvement programs aren’t Competent people improve processes all the time (pride, growth, etc.) Formal process improvement moves responsibility from the individual to the organization Process improvement programs focus on process rather than product (making a poor product efficiently is often worse than making a good product poorly) Focus on process “level” tends to make organizations risk averse

“The projects most worth doing are the ones that will move you down one full level on your process scale!”

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

20

Break

 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

21

Quality Standard Rationale

Customers want & need assessments of supplier quality Means: Individually audit (i.e. qualify) vendor: Specific products Processes (e.g. manufacturing, design & development, support) Alternative: Common Quality Assurance standards & audits 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

22

Major Audit Types

First Party Audit Within own company (aka internal audit) Used to measure own performance, strengths & weaknesses against internally established procedures & systems Second Party Audit Performed by customer on their supplier (aka external audit) Third Party Audit Outside, independent auditor contracted to audit on behalf of company or a supplier (e.g. ISO 9000 registration audit) Assessments (e.g. SCAMPI) Similar to first party audit, but typically performed by external assessors 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

23

Other Audit Types

System Audit – examination of bigger picture of organization &/or project Typical cross organizational, cross process & cross product Process Audit – verify inputs, actions & outputs in accordance with defined requirements (e.g. software inspections) Product Audit – final product or service for “fitness for use” Customer oriented Compliance Audit Regulatory – audit to government regulations Management – audit to organizational rules, effectiveness & conformance Quality – systematic & independent of quality activities vs. established procedures 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

24

ISO 9001:2000

Objective Provide confidence that vendor can produce quality products Assumptions: good practices will produce good products Standard for assessing organization’s Quality Management System (QMS) – Processes – Activities – Behaviors – Training

But

, ISO focuses on

Quality Assurance

not

Quality Control

ISO-9001 certification

does not

guarantee quality products!

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

25

Tenants of ISO 9001

1) Say what you do 2) Do what you say 3) Prove it!

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

26

ISO 9000 Audits

Customers write requirements for current ISO-9001 certification into purchasing contracts Organizations apply for 3 rd party audit, end result is ISO-9001 certification ISO

International Accreditation Forum (IAF)

board Audits

national accreditation boards

(i.e. one board each nation) Who register individual

registrars

(e.g. Lloyd’s, DNV) Who audit organization

internal auditors

(e.g. Lucent Optical Networking) & spot check Who audit

design, development, manufacturing & support teams

within the organization 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

27

ISO Alphabet Soup

ISO 9000:2000 Overall framework, fundamentals of quality management systems & terminology ISO 9001:2000 Requirements for quality management systems (qms) & what is required to demonstrate compliance ISO 90003 2004 (previously 9000-3) Guidelines for the application of ISO 9001:2000 to computer software ISO 19011 Guidelines for auditing quality and environmental management systems 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

28

What is wrong with ISO 9001?

Vendors ISO-9001 certified, but quality still elusive!

No visibility into supplier quality levels Not getting quality levels they wanted Solution: TL9000 (Quest forum, telecommunications) QS9000 (automotive) 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

29

TL9000

ISO on steroids Wholly subsumes ISO 9001-2000 Requires vendors prove they are actually improving Metrics focused on cost drivers of service providers: Know vendor is measuring Visibility into quality improvement results 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

30

TL9000 Top Management Requirements

Monitor & improve customer satisfaction Set long & short term objectives for organization effectiveness Set targets for TL9000 product performance metrics Use an explicit life-cycle model Establish a quality improvement program Periodic management review of quality system 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

31

TL9000 Metrics

Cross-discipline metrics – # of problem reports – Problem report fix response time – Overdue problem report fix responsiveness – On-time delivery Hardware & Software measurements – System Outages Hardware measurements – Return rates Software measurements – Software installation & release application aborts – Corrective patch quality – Feature patch quality – Software update quality 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

32

TL9000 Common Audit Questions

• • • • • • • Do you know how to find your Quality Policy, QMS and the processes you should be using for your work?

Do you know your organization’s product delivery & improvement goals and what you must do to support them?

Do you know what skills you should have?

Do you know what you have to do to approve/baseline/finalize your documents, designs & code?

Do you know how to store & find records of reviews, inspections, key decisions, etc.?

Do you know what to do if a problem is found with the product or process?

Do you know your organization’s performance with respect to customer satisfaction, quality of delivered products & process execution?

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

33

TL9000 Sample Requirements

Planning – Must have methods for estimating & tracking – Determine where you will do reviews & tests – Risk management plans, customer, user & supplier involvement in reviews & evaluation Software Outputs – Requires architecture, detailed designs, code & user documentation – Each design thread must be reviewed at some point prior to integration or system test Software Testing – All testing must have test plans; test process must be documented – Plans must include test cases with inputs, output & test success criteria – Plans must include types of testing, requirements traceability, coverage definition & measurement, test environment, defect handling, et.al.

– Integration testing specifically required 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

34

Team Project Postmortem

Tracking process improvements during project Process Improvement Proposals (PIP) Port-Mortem Areas to consider Better personal practices Improved tools Process changes 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

35

Postmortem process

Team discussion of project data Review & critique of roles 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

36

Postmortem process

Review Process Data Review of cycle data including SUMP & SUMQ forms Examine data on team & team member activities & accomplishments Identify where process worked & where it didn’t Quality Review Analysis of team’s defect data Actual performance vs. plan Lessons learned Opportunities for improvement Problems to be corrected in future PIP forms for all improvement suggestions Role Evaluations What worked?

Problems?

Improvement areas?

Improvement goals for next cycle / project?

7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

37

Cycle Report

Table of contents Summary Role Reports Leadership – leadership perspective Motivational & commitment issues, meeting facilitation, req’d instructor support Development Effectiveness of development strategy, design & implementation issues Planning Team’s performance vs. plan, improvements to planning process Quality / Process Process discipline, adherence, documentation, PIPs & analysis, inspections Cross-team system testing planning & execution Support Facilities, CM & Change Control, change activity data & change handling, ITL Engineer Reports – individual assessments 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

38

Role Evaluations & Peer Forms

Consider & fill out PEER forms Ratings (1-5) on work, team & project performance, roles & team members Additional role evaluations suggestions Constructive feedback Discuss behaviors or product, not person 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

39

Project Notebook

Updated Requirements & Design documents Conceptual Design, SRS, SDS, System Test Plan, User Documentation* Updated Process descriptions Baseline processes, continuous process improvement, CM Tracking forms ITL, LOGD, Inspection forms, LOGTEST Planning & actual performance Team Task, Schedule, SUMP, SUMQ, SUMS, SUMTASK, CCR* 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

40

August 4 Class

CMMI Introduction & Configuration Management Appraisal ISO 9000/TL-9000 Due July 31: Cycle 2 Design & Code, hand off to System Tester System Test Plan Inspected & Baselined Project notebook updates including inspection records, meeting minutes, etc.

Deliverables for August 7 Project Postmortem (cycle report) Cycle 2 presentations Peer Feedback forms Completed project notebooks Cycle Exit Completed project (source, documents & all quality records) 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality.ppt

41