Transcript Beyond the CMM

Beyond The CMM:
WHY IMPLEMENTING THE SEI'S CAPABILITY
MATURITY MODEL IS INSUFFICIENT TO
DELIVER QUALITY INFORMATION SYSTEMS IN
REAL-WORLD CORPORATE IT ORGANIZATIONS
Brett Champlin, MBA, CSP/CCP
Senior Lecturer, Adjunct Faculty,
MSIS & MBA Programs, Roosevelt University
Board of Directors, DAMA International
Board of Directors, ICCP
Worker Bee, Corporate America
DAMA Michigan
January 14, 2002
Buggy Software
$59.5 Billion Annual Cost
Presented at DAMA Michigan, January 14, 2003
2
Suppose You Were…
 The Director of Application Development




Spent $1.5 Million over 2.5 years
To get from Level 1 to Level 2 (or 2 to 3)
63% increase productivity
But COO says 60% of time spent fixing data
 A New CIO
 Assessed at Level 3
 3 years ago
 Current assessment suggests back to Level 1
Presented at DAMA Michigan, January 14, 2003
3
The Point Is
 The CMM is a good model
 but, it isn’t a “silver bullet”
 The CMM is not enough
 to deliver quality information & services
 There are some problems with the
CMM
 Is there a solution?
Presented at DAMA Michigan, January 14, 2003
4
CMM History
 The Software Engineering Institute
established 1984 at Carnegie Mellon
University, Pittsburgh
 Principle customer is the Department
of Defense
 U.S. Air Force Project 1987
 Method to select software contractors
 Over 5,000 Assessments performed
since 1987
Presented at DAMA Michigan, January 14, 2003
5
The Capability Maturity Model
Optimizing
(5)
Process
Control
Quality and
Productivity
Improvement
Consistent
Execution
Standard Process
Repeatable
(2)
Stable Process
Initial
(1)
Measured Process
Defined
(3)
Process
Definition
Basic
Management
Control
Improving Process
Managed
(4)
Process
Measurement
Continuing
Improvement
Controlled
Environment
Chaotic
ad hoc Process
Presented at DAMA Michigan, January 14, 2003
6
The Capability Maturity Model
Productivity & Risk
Optimizing
5
Managed
4
3
Defined
2
Repeatable
Initial
Productivity
Risk
1
Presented at DAMA Michigan, January 14, 2003
7
SEI CMM Assessments
70%
60%
50%
Initial
Repeatable
Defined
Managed
Optimizing
40%
30%
20%
10%
0%
1995
1997
1999
2001
SEI Assessments
Presented at DAMA Michigan, January 14, 2003
8
SEI Assessments
1,200
1018
1,000
800
734
600
400
541
353
200
0
1995
1997
1999
Presented at DAMA Michigan, January 14, 2003
2001
9
Organizational Maturity Profile
Percentage By Level
45
39.1
40
35
30
27.1
23.4
25
20
15
10
5.6
4.8
Managed
Optimizing
5
0
Initial
Repeatable
Defined
August 2001 – Assessment of 1018 Organizations
Presented at DAMA Michigan, January 14, 2003
10
Federal Agencies
Percentage By Level
80
71.4
70
60
50
40
28.6
30
20
10
0
0
0
Defined
Managed
Optimizing
0
Initial
Repeatable
August 2001 – Assessment of 14 Organizations
Presented at DAMA Michigan, January 14, 2003
11
In House Development Organizations
creating software for internal use
Percentage By Level
50
45
40
35
30
25
20
15
10
5
0
46.8
37.3
12.7
2.5
Initial
Repeatable
Defined
Managed
0.6
Optimizing
August 2001 – Assessment of 158 Organizations
Presented at DAMA Michigan, January 14, 2003
12
Military Services
Percentage By Level
45
38.2
40
35
32.7
30
25
20
20
15
7.3
10
5
1.8
0
Initial
Repeatable
Defined
Managed
Optimizing
August 2001 – Assessment of 55 Organizations
Presented at DAMA Michigan, January 14, 2003
13
Commercial Organizations
creating software for profit
Percentage By Level
45
39.9
40
35
30
27.7
25
18.6
20
15
6.4
7.3
Managed
Optimizing
10
5
0
Initial
Repeatable
Defined
August 2001 – Assessment of 531 Organizations
Presented at DAMA Michigan, January 14, 2003
14
DoD Contractors
Percentage By Level
50
45
40
35
30
25
20
15
10
5
0
46.7
28.3
14.6
7.1
3.3
Initial
Repeatable
Defined
Managed
Optimizing
August 2001 – Assessment of 212 Organizations
Presented at DAMA Michigan, January 14, 2003
15
Is it worth it?
Category
Range
Median
Data pts
Cost of SPI per
software
engineer/yr
$490-2004
$1375
5
Gain in
productivity per
year
9%-67%
35%
4
Reduction in
time to market
per year
15%-23%
----
2
Reduction in
post release
defects/yr
10%-94%
39%
5
Business value
ratio
(benefit/cost)
4.0-8.8:1
5:1
5
“The Evidence for CMM-based Software Process Improvement” – SEMA-3.01, CMU, 2001
Presented at DAMA Michigan, January 14, 2003
16
Degree of SPI Success
success
throughout
8%
the organization
23%
substantial
26%
moderate
30%
limited
14%
little if any
0%
10%
20%
30%
40%
“The Evidence for CMM-based Software Process Improvement” – SEMA-3.01, CMU, 2001
Presented at DAMA Michigan, January 14, 2003
17
Caution
“Men [sic] love abstract reasoning
and neat systematization so
much that they think nothing
of distorting the truth,
closing their eyes and ears
to contrary evidence
to preserve their logical
constructions”
- Fyodor Dostoevsky
Presented at DAMA Michigan, January 14, 2003
18
Presented at DAMA Michigan, January 14, 2003
19
Presented at DAMA Michigan, January 14, 2003
20
Other Maturity Models








120+ and counting
Data MM
Training MM
Release MM
Configuration MM
Documentation MM
Business Rules MM
Etc…
Presented at DAMA Michigan, January 14, 2003
Other
Models
21
Other Maturity Models
By Category
34
24
19
14
13
5
5
4
3
3
2
So
ftw
ar
e
IT
M
gt
Pj
tM
gt
D
at
B
us a M
in
gt
es
s
M
gt
K
I
nt
no
er
w
ne
le
t
dg
O
e
th
M
er
gt
In
du
IT
s
O
pe try
ra
tio
ens
B
us
in
es
G
s
ov
't
M
gt
45
40
35
30
25
20
15
10
5
0
Presented at DAMA Michigan, January 14, 2003
22
Other Maturity Models
Percentage by Phase
6%
7%
IT Mgt
29%
7%
All
Design
Development
14%
Operations
23%
26%
Testing
Other
Presented at DAMA Michigan, January 14, 2003
23
Something Totally Different
Presented at DAMA Michigan, January 14, 2003
24
Missing Pieces?
 Are all of these other models just filling
in missing pieces?
 Isn’t the core of CMM just Project Mgt?
 What about all of these other nonsystems related Maturity Models?
Presented at DAMA Michigan, January 14, 2003
25
What’s the Problem?
 Management
 Business demands
 Turnover
 Worldwide 10-15%
 Slow
 1.5 – 2.5 years to improve 1 Level
 New Technology
 Rate of Change is increasing
 Application Development Environments
 Portals; HTML; Java; .Net; XML-SOAP-UDDI
 Application Development Methodologies
 OO/UML-RUP; XML-Web; XP/Agile Methods
Presented at DAMA Michigan, January 14, 2003
26
Management’s Role
Executive Maturity Must Precede Process Maturity
Need
Management
Program
Need
Executive
Oversight
Need
Consistency
Continuously
Improving
Process
Managing
(5)
Predictable
Process
Optimizing
(5)
Managed
(4)
Participative
(4)
Supportive
(3)
Standard,
Consistent
Process
Disciplined
Procedures
Defined
(3)
Repeatable
(2)
Recognized
(2)
Ignored
(1)
Initial
(1)
From "Data Management Maturity Model" Burton G. Parker, et. al., MITRE Software Engineering Center, McLean, Virginia July 1995;
Parker, B., Enterprise-wide Data Management Process Maturity Framework, Handbook of Database Management, Auerbach, 1999
Presented at DAMA Michigan, January 14, 2003
27
Speed
Level One to Level Two – x years?
Each Level after that 1.5 – 3 years?
From One to Five, takes 6 – 12 years?
Is it really that hard?
What is the problem with taking on two
or three level improvements at a time?
 It doesn’t take that long to revamp an
entire IT operation





Presented at DAMA Michigan, January 14, 2003
28
Improvement or Re-Design?
Process
Re-Engineering
Less
Process
Re-Design
Process
Improvement
More
Time
More
Less
Change
?
?
Value
 Process Improvement is incremental
 Process Re-Design is end-to-end re-thinking
of what we are doing
 Process Re-Engineering is a blank slate
approach
Presented at DAMA Michigan, January 14, 2003
29
What about Quality?
 Hmmm…information quality problems
aren’t always related to software
defects
 What about the data?
 What about lousy requirements?
 What about poorly designed business
processes?
 Is software development where the real
problem lies?
Presented at DAMA Michigan, January 14, 2003
30
Software or Information Quality?
 The Australian Government wanted to
know how many illegal immigrants
there were in the country.
 Matched those arriving (tourists, etc.)
with those leaving.
 From a population of 12 million they
arrived at 400,000 estimated illegals.
 The actual number was about 100,000
Presented at DAMA Michigan, January 14, 2003
31
Wanted by the FBI?
 Wanted by the FBI for questioning: Adil
Pervez. Or is it Adel Pervaiz? Adil Pervaz,
maybe?
 As it turns out, Adil Pervez wasn't wanted
after all. On Jan. 7, the FBI called off a
manhunt for Pervez and four other Middle
Eastern men. But the fact that alternative
name spellings for the five suspects were
posted online immediately after the alert
points to an increasing need for
understanding complexities of foreign
names.
Source: www.wired.com, “Wanted: what’s his name again?” Jan 13, 2003
Presented at DAMA Michigan, January 14, 2003
32
Some More Cases to Consider
 75% of companies report significant
problems due to defective data
 92% of claims Medicare paid to community
health centers over one year’s time were
“improper or highly questionable”
 Wrong price data in retail databases costs
consumers as much as $2.5 billion in
overcharges annually
 96,000 IRS tax refund checks were returned
as undeliverable one year
Presented at DAMA Michigan, January 14, 2003
33
Cost of Poor Data Quality
$600 billion!
Presented at DAMA Michigan, January 14, 2003
34
Systems Dynamics
 Focusing on software development is
sub-optimizing information and
technology delivery
 Process integration is given lip-service
at Level 3, but …
 Without integrating software
development with data and operations
management, we will continue to have
quality problems
Presented at DAMA Michigan, January 14, 2003
35
Integrated Processes
Software
Development
Data
Management
Documentation
& Training
Change Mgt
Systems Operations
Change Mgt
Business Operations
High
Quality
Information
Services
Infrastructure
Development
Presented at DAMA Michigan, January 14, 2003
36
ITIL – IT Infrastructure Library
1. Service Support
(delivering and supporting IT services that are appropriate to the business requirements
of the organization.)
2. Service Delivery
(Service Level Management, Financial Management for IT Services, IT Service Continuity
Management, Availability Management Contingency Planning and Capacity
Management.)
3. Planning to Implement Service Management
(the steps required to implement or improve IT service provision & guidance on alignment
of the business needs to IT.)
4. Applications Management
(the software development life cycle, details on business change with the emphasis on
clear requirement definitions and implementation to meet business users' needs.)
5. ICT Infrastructure Management
(network service management, operations management , management of local
processors, computer installation and acceptance and systems management.)
6. Security Management
(how to organize and maintain the management of security of the IT Infrastructure, from
the IT manager's point of view.)
7. The Business Perspective
(business relationship management, partnerships and outsourcing, continuous
improvement, exploitation of information, communication, and technology (ICT) for
business advantage.)
Presented at DAMA Michigan, January 14, 2003
37
Why Isn’t CMM Enough?
 What is the real problem/product?
 Sub-optimizes Information Services Quality
 Must be coordinated with Data Mgt, Operations
Mgt, etc.
 Too slow
 Redesign or Reengineer the process
 Needs Management Maturity to sustain
 IT Quality programs must be aligned to
Business Needs
Presented at DAMA Michigan, January 14, 2003
38
What Should We Do?
 CMM is a really good model for Software Quality
Improvement
 Apply a generic process maturity model to all IS
processes
 Take a redesign approach to all IS processes –
integrated, from the top - down
 Train all IT managers in quality management
principles
 Make sure you identify where your real problems are
before you start “fixing” things
 Align your improvement efforts to your business
needs
Presented at DAMA Michigan, January 14, 2003
39
Not the CMM…





Level 1: Initial
The software process is ad hoc, and occasionally even chaotic. Few processes are
defined, and success depends on individual effort and heroics. Frequent late nights and
hollow, sunken eyes are common. Programmers at each others' throats. Managers
mostly very angry.
Level 2: Repeatable
Should a similar project be run, it would probably be just as chaotic. Team leaders have
slim control over the programmers. Project manager has installed MS Project, and
printed out reams upon reams of Gantt charts which have already started to block
doorways and commonly used pathways, e.g. from "zoo" area to kitchen. More late nights
than not. Programmers generally civil to each other, but often murmur discontentedly
behind each others' backs. Managers have bouts of rage during progress update
meetings.
Level 3: Defined
The software process is documented, standardized, and integrated into a standard
process for the organization. Managers express keenness to use the process for any new
projects, as soon as all the current projects are finished. Programmers spend more time
filling out forms than writing software.
Level 4: Managed
Accurate metrics are collected for each project, as and when it fails. Detailed "postmortems" explain why the organization's standard process was not used yet again.
Programmers spend large proportion of day updating CV and hanging around the popular
job sites.
Level 5: Optimizing
A miracle worker was passing by one night. Programmers angry because most of the
jobs out there are made-up.
Source: http://www.bad-managers.com/rumours/cmm_level_one.shtml
Presented at DAMA Michigan, January 14, 2003
40
Thank You
I appreciate your comments and will be
pleased to answer any questions that you
may have.
You may contact me for further discussion
at:
Brett Champlin
email: [email protected]
or
[email protected]
Visit my faculty website:
http://faculty.roosevelt.edu/Champlin/
Presented at DAMA Michigan, January 14, 2003
41
Select References/Sources












“The Electronic College of Process Innovation”, DoD, http://www.c3i.osd.mil/org/bpr.html
“The Evidence for CMM-based Software Process Improvement” – SEMA-3.01, CMU,
2001
The IT Infrastructure Library, http://www.itil.co.uk/
“Manager Heal Thyself: Improving Software Processes means Changing Management
Processes”, Derby & Rothman, Cutter IT Journal, Oct 2001, Vol. 14, No. 10
“Managing the Software Process”, Watts Humphrey, 1989, Addison-Wesley
“Maturity Alone Is Meaningless: SPI Reality from Industry”, Rodenback, Lautum,
Solingen, Cutter IT Journal, Oct 2001, Vol. 14, No. 10
“Out of Alignment”, Paul Strassman, Computerworld, March 4, 2001
“Quality Is Free: The Art of Making Quality Certain”, Philip Crosby, 1979, McGraw-Hill
Software Engineering Information Repository, http://seir.sei.cmu.edu/
Software Process Improvement: Is That What We All Should Be Doing – Again?”, Kendall,
Cutter IT Journal, Oct 2001, Vol. 14, No. 10
“The Value of Software Process Improvement”, James Douglass, SEI-CMU, Oct 2001
Worldwide IT Trends & Benchmark Report 2002, Meta Group
Presented at DAMA Michigan, January 14, 2003
42
Other Maturity Models
Automated Software Testing MM– Krause
Business Processes – Dr. Jaques Hale', 1994
Business Rules – Barbara Von Halle, Apr '96
CASE Tool Selection – S.L. Pfleeger, November
1991
Change Proficiency MM– Paradigm Shift
International
Competence (Data Warehouse analyst
competence) MM – Brohman & Parent, 2001
Compliance MM – Rushmere Consulting, Inc.
Content Management MM – Steelpoint
Technologies, 2002
Course Website Development MM – Fred
Beshears, Univ of California, Berkeley
Corrective Maintenance MM – Kajko-Mattson
Customer Intimacy MM, Ken Rudin, Keane Inc.
Customer Relationship Management (CRM)
MM – EDS
Data Management MM – MITRE, 1994-96
Data Resource Management MM – Champlin,
1996
Data Warehousing MM – Marco, 2002
Documentation Management – Dr. Marcello
Visconte (Chile)
e-Business MM - Berkeley Enterprise Partners
EBusiness MM – Gardler, 2000
E-Commerce MM – Ludescher, Vienna
University of Technology and Usrey,
University of Colorado
Earned Value Management MM – Ray Stratton,
Management Technologies,
Enterprise Application Integration MM –
Schmidt & Seidel, AMS, 2001
Enterprise-wide Data Management – Burton
Parker
ERP Systems MM – Holland
FAA-iCMM, FAA’s Integrated CMM – Federal
Aviation Administration
Facilities Renovation MM – APPA
Presented at DAMA Michigan, January 14, 2003
43
Other Maturity Models
Function Point MM – Emmons, 2000
General Practice MM - Gillies
Government Technology Maturity Model for IT
Investment and Project Management –
GAO, 2002
Human Factored Interface MM – J. Earthy
Information Evolution Model – Hatcher, 2002
Information Process MM – Hackos, 1994
Inspections CM – Bob Cerady, HP
Integrated Product Development MM –
EPIC/SECAT, 1997
Integration for the Digital Enterprise MM –
Paydarfar, 2001
Internet Agility MM – Brian Neymeh, INSTEP
Inc. 1995
Internet Development (Network Maturity) MM
– IEEE
Internet Management MM– CMU
IT Architecture MM – US Department of
Commerce
IT/Business Alignment Assessment MM –
Luftman (Harvard)
IT Governance MM – CobiT
IT Service Capability MM – Frank Niessink
Knowledge MMM – Informatie Management
Nederland (IMN)
Knowledge Management MM – Gallagher,
Queen’s Univ of Belfast,
Knowledge Management MM - Langen, 2000
Learning Organization MM – Lockheed Austin
Maintenance MM – AMS
Management Information Systems MM – Ernst &
Young
Measurement Program MM – Donnellan &
Peterson
Operations-Based for Problem Management MM
- Meta Group, 2001
Organizational MM – Rosenberg
Organizational Interoperability MM – Clark &
Jones, 1998
Organizational Project Management MM - PMI
People Management MM – SEI
Performance Engineering MM – Scholz and
Schmietendorf, 2000
Portal MM – Tanning Technology Corporation
Process Improvement for Construction
Enterprises MM - (SPICE) 1998-2000
Presented at DAMA Michigan, January 14, 2003
44
Other Maturity Models
Product Development MM – Crow, DRM
Associates, 2000
Programme Management MM – Rayner & Reiss,
The Program Management Group, 2000
Programme Management and System
Engineering MM – European Software
Institute
Project Management MM – Cadence
Management Corporation, 2001
Project Management MM – Enterprise Planning
Associates, 1998
Project Management MM – IBBS Consulting
Project Management MM – Kerzner, 2001
Project Management MM – Levin, Hill, DeFillipis,
Ward, Shaltry, Richards/ESI
Project Management MM – PM Solutions, 2001
Project Management MM – Robertson, KLR
Consulting, 2001
Project Management MM – Wisdm Corp., 2001
Public Works MM – UK
Security Management MM – NSA sponsored
Service MM – Quickarrow.com
Software Change Management MM - Bendix,
1998
Software Contracting MM – August Automation,
Inc.
Software Maintenance MM – MITRE
Software Reuse MM – Davis (SPC)
Software Systems Documentation Process –
Oregon State University, 1992
Strategic Management MM – McWeeny &
Ellinger
System MM - (school’s use of IT, based on
Nolan’s Stages Model)
System Acquisition MM – Ferguson, MITRE SEI
System Administration MM – Kubicki, 1993
System Engineering MM – Malpass, SEI
Systems Requirements MM – REAIMS, 1994-97
System Security MM – Hefner
Systems Security Engineering MM – Electronic
Warfare Associates (EWA) Information and
Infrastructure Group
Technology MM – Edmin.com, 1998
Testability MM – Gelperin, 199
Testing – Bernstein, Suwannasart, and Carlson,
IIT, 1996
Presented at DAMA Michigan, January 14, 2003
45
Other Maturity Models
Testing MM – Burgess & Drabick, 1996
Total data Quality Management MM - MIT
Training MM – MITRE
Trusted Software MM – Kitson
Usability/Human Centredness MM – Earthy,
1998
Vulnerability Management – security Online,
2001
Risk Maturity Model, HVR Consulting Services,
Ltd.
Risk Maturity Model, Basil Orsini , Human
Resources Development Canada
eServices Capability Model (eSCM), CMU
CMM for Inventory Effectiveness , Jeff
Kavanaugh, Inforte Corp.
Data MM, Jeremey Janzen, British Columbia
Ministry of Forests
Data Warehouse Information Management MM,
John Ladley
Data Quality MMM, Meta Group
Enterprise Architecture MM, Meta Group
Enterprise Program & Project Management
MM, Meta Group
Information Management MM, Meta Group
Infrastructure MM, Meta Group
Internal Consulting MM (IT Operations &
Services) , Meta Group
IT Operational Process MM, Meta Group
IT Performance and Measurement
(Organizational MM), Meta Group
IT Procurement MMM, Meta Group
Measurement MM, Meta Group
Outsourcing MMM, Meta Group
Process MMM, Meta Group
Telecom MMM, Meta Group
Presented at DAMA Michigan, January 14, 2003
Back
46