Transcript Basics of Structured Software Testing

Test Management Approach (TMap)
Chris Hampton, PMP
Practice Manager, Software Control & Testing
Sogeti USA
September 2007
1
2
Agenda:
• The State of, and Business Case for, Software
Quality
• What is testing?
• Sogeti’s Test Management Approach® (Tmap®)
• Sogeti’s Test Process Improvement® (TPI®)
Methodology
3
Poor State of Software Delivery
“Software failure costs the American
economy $59.5 billion annually.”
Source: NIST (2002)
90% of projects are delivered late
66% of projects rated unsuccessful
54% of projects over budget
30% of projects cancelled early
Source: Standish 2003
4
The Business Case For QA
• Compliance
– Basel II, Sarbanes Oxley, HIPAA, etc…
– Reduced liability from defective software and defective
process
• Cost Savings
– Quality processes now reduce support costs later
– Lost revenue
• Satisfaction
– Higher User satisfaction
– Improved business alignment
• Growth
– Better quality and measurements allows development
optimization
– Increase ability to deliver
5
Where Quality Problems Originate
7%
10%
56%
27%
Code
Other
Design
Requirements
Source: James Martin
6
How Quality Provides Tremendous Savings
Source: Cutter Consortium/Forrester/Sogeti
Code
Planning
Analysis
System
Design
$140
COST TO
REPAIR
DEFECT
Unit
Test
Unit
$1000
Test
Integration
Code
Test
$2500
System
Test
$4500
Production
$14000
Certification
Test
$7000
Time cost and resource effort increase
exponentially later in the lifecycle
July 17, 2015
7
What is testing?
Testing is a process aimed at:
• Finding defects in a controlled manner
• Detecting the level of quality of the test object
• Demonstrating the gap between specifications and the
actual product
• Demonstrating that the end product functions as called
for in the requirements
8
Testing is not . . . .
Testing is not:
• Implementation
• Acceptance
• Defect repair
• A phase after development – although testing includes a
phase after development
• Training on a new system
9
Why do we test ?
• To mitigate business risks that have been
identified:
–
Validate software quality
–
Verify integration of business processes
–
Reduce time-to-market
–
Competitive purposes
• To ensure business usability of the software:
–
Release low-error/known-error software
–
Move high quality software (meets or exceeds quality
expectations) into production
–
Demonstrate the usefulness of new technology
10
Testing of information systems
What do we test?
• Application Software
• Functionality
• Hardware
• Continuity
• System Software
• Performance
• Procedures
• Usability
• Documentation
• Interoperability (between
different applications and
systems)
11
Structured Software Testing
• Testing everything is impractical
– Funding limitations
– Limited time and resources
– Diminishing value after a certain point
• Isn’t there a more effective way?
• Yes! Structured software testing: a risk-based, qualitycentric approach to testing
12
Test Management Approach (TMap®)
• Sogeti’s methodology for structured
testing of software products.
• TMap® has evolved to be the
standard for software testing in
Europe and quickly gaining traction in
the US.
– It is being used by more than three
hundred Dutch, Belgium and German
organizations.
– Industry adoption includes :






Financial Services
Insurance
Government
Consumer Electronics
Telecommunications
Medical systems
13
4 Essentials of TMap®
• Business-driven test management
(BDTM) approach
• Structured test process
• Complete tool box
• Adaptive test method
BDTM puts focus on risk-based testing—only
testing what has impacts to the organization.
14
Business Driven Test Management
• Choices must be made in what
is tested and how thoroughly.
• This is an economic decision
based on:
– The risks that an organization
thinks it will incur
– The availability of time and
resources
– The results the customer
wishes to achieve.
• The choices based on risks,
result, time and cost constitutes
the basis for the BDTM
approach.
15
The BDTM Process
1. Formulate the test goals.
2. Determine the risk class for each
combination of characteristic and
object part.
3. Determining the depth of testing
required.
•
Based on risk/resource trade off
4. Develop the test estimate and test
plan.
5. Allocate the appropriate test
techniques.
6. Throughout the test process, the
test manager provides the
stakeholders with insight into and
control options over test process
and test object.
16 16
Risk Definition
•
Testing should be based on the mitigation of risk and
validation of expected quality defined by the business
requirements
•
Definition of risk
– A “risk” is a chance of a failure occurring, related to
expected damage to the project (should the failure occur)
•
Risk Categories
– Risks identified in different categories such as business
risks, project risks, product risks, process risks
•
Risk Ranking
– Risk are ranked in criticality relative to one another, by
instituting a method of risk ranking (or “risk rating”)
17
How Risk Ranking Works
1. Assemble a team of people representing various roles
from the project
– PMs, Business Leads/SMEs, Test Leads, Tech Leads
2. Create an initial list of risks
3. Team assigns a numeric value to each risk = the
probability of occurrence of each risk
4. Team assigns a second value to each risk, representing
the impact on the project/organization
5. Multiply the two values together (the probability of
occurrence X the impact)
6. The result is a relative value for each risk
7. Order the risks by their relative risk values
– “risk rating” or “risk ranking”
– Ranking helps manage the most critical risks, especially
those falling in the middle tier of the ranking
8. Periodically update the list of risks
18
Risks – quality characteristics matrix
Multi-Level Matrix
Quality
Characteristics
High
Priority
High Priority/
Low Risk
High Priority/
Medium Risk
High Priority/
High Risk
Medium Medium Priority/ Medium Priority/ Medium Priority/
Medium Risk
High Risk
Priority
Low Risk
Low
Priority
Low Priority/
Low Risk
Low Priority/
Medium Risk
Low
Medium
Low Priority/
High Risk
High
Risks
19
Ten steps for creating a test case
Step 10
Step 9
Step 8
Step 7
Facilitated
Sessions
Step 6
Step 5
Step 4
Step 3
Step 2
Step 1
CREATE TEST CASES
Select suitable test techniques
Associate test scenarios with risks
Create test scenarios for each subsystem
Create subsystem / quality characteristic pairings
Rank relevant quality characteristics
Agree on relevant quality characteristics
Rank risks
Identify risks
Identify sub-systems / sub-processes
FUNCTIONAL AREA
20
Structured Test Process
• Master Test Plan, managing the total test process
– Acceptance and System Tests
– Developmental Test
– Supporting Processes
• Lifecycle Model
21
Master Test Plan
• Developed by the test manager
in partnership with the
appropriate stakeholders
• Jointly define:
– What will be tested for each
test level,
– When it will be tested
– To what degree of
thoroughness.
• This plan constitutes the basis
for the test plans for the
separate test levels.
–
–
–
–
Unit Testing
Systems Integration Testing
Acceptance Testing
Developmental Testing
22
Life Cycle Model
• The life cycle model is a generic model.
– Can be applied to all test levels and test types
– Used in parallel with the life cycle models for system
development.
• Test activities are divided across seven phases:
–
–
–
–
–
–
–
Planning
Control
Setting up and maintaining infrastructure
Preparation,
Specification
Execution
Completion
23
TMap - Structured testing lifecycle
PREPARATION
-Testability review
-Allocate techniques
-Requirements review
-Data cycle test
-Process cycle test
-Other tests
-Specify test cases
-Create/assess
infrastructure
-Test specification
-Data flow test
EXECUTION
-Pretest
-(Re)test
-Checking
-Assessing
-Debugger
-Record and playback
-Monitoring
-Preserve testware
-Evaluate process
SPECIFICATION
-Create test scripts
-Create infrastructure
P
S
E
C
P&C
PLANNING &
-Inspection and study
-Develop test strategy
oRisk analysis
oTest Estimation (TPA)
-Setup organization
-Prepare test plan
-Management and control
CONTROL
-Test strategy
-Test estimation
-Reports
-Management tools
-Budgeting tools
-Defect management
tool
COMPLETION
-Preserve testware for
future use
-Evaluate process
24
Complete Tool Kit
• Techniques
– Many techniques can be used
in the test process. A test
technique is a combination of
actions to produce a test
product in a universal manner.
• TMap provides techniques for
the following:






Test estimation
Defect management
Creating metrics
Product risk analysis
Test design
Product evaluation.
25
Adaptive Test Method
• Complete And Flexible
• Works with any SDLC
–
–
–
–
–
Iterative
RAD
Agile
Waterfall
RUP
• Works with any technology
–
–
–
–
Mainframe
Web
ERP
SOA
26
27
TMap® – Differentiators
• Business-Driven Test Management
• Risk-based test strategy
• Efficient testing—coverage without
overlap
• Testing off of critical path as much
as possible
• Criteria and metrics about
production readiness
• Management of testing to project
timelines
• Compliments industry tools
Proven methodology applied in hundreds of
companies internationally over the past 14 years!
28
Benefits of Structure Testing
• Defects are found early, costing less time/money to reach
production  delivering a higher quality product
• Required quality of the various test objects is tested for
and validated, by focusing on testing for quality as a risk
mitigation strategy
• By keeping a larger percentage of the testing
process/effort off the critical path, faster time-to-market
results
• Structured software testing is more cost-effective and
efficient than non-structured testing approaches
• Sound test coverage is provided, without the need to
overlap phases
• Establishes a test organization that is prepared and
efficient
• Delivers a repeatable process
29
Test Process Improvement (TPI)
• The Sogeti TPI® Model was developed to facilitate a
stepwise improvement of the testing process.
– Developed from practical experience from 30+ years of
Sogeti work in testing.
– Offers a frame of reference to determine the strengths and
weaknesses of your current testing process.
– Covers 20 key areas that need specific improvement to
achieve a well define testing process.
• Key TPI Areas:
Test Strategy
Life-Cycle Model
Moment of Involvement
Estimating and Planning
Test Specification Techniques
Static Test Techniques
Metrics
Test Automation
Test Environment
Office Environment
Commitment and Motivation
Test Functions and Training
Scope of
Methodology Communication
Reporting
Defect Management
Testware Management
Test Process Management
Evaluation and Low-Level
Testing
30
Typical TPI Process Steps
1. Interview key people
2. Describe the ‘As-Is’ process including Strengths &
Weaknesses
3. Describe the ‘To-Be’ process
4. Document process improvement actions
5. Define expected benefits in terms of Objectives
6. Plan for Implementation of the Improvement Actions
(short-term and long-term)
31
TPI - Test Maturity Matrix
0
Key area / Scale
1
2
3
4
5
6
7
1
Test strategy
A
2
Life-cycle model
A
3
Moment of involvement
4
Estimating and planning
5
Test specification techniques
6
Static test techniques
7
Metrics
8
Test automation
A
B
9
Test environment
A
B
10 Office environment
9
1
0
1
2
B
C
D
B
C
D
A
1
3
A
B
B
A
B
A
B
C
D
C
C
A
Increasing Maturity
A
B
A
12 Test functions and training
B
A
14 Communication
15 Reporting
A
16 Defect management
A
A
B
B
C
C
B
C
B
A
17 Testware management
C
C
A
13 Scope of methodology
18 Test process management
1
1
B
A
11 Commitment and motivation
D
C
B
C
B
D
C
A
19 Evaluation
20 Low-level testing
8
A
B
B
C
32
Resources
• Web Sites
– Sogeti USA Web Site: www.us.sogeti.com
– TMap Web Site: http://eng.tmap.net/Home/
• TMap® certification (Netherlands)
– Foundation
– TMap ® professional Advanced/Expert
– TMap ® management Advanced/Expert
• Books
33
Sogeti SCT Services
• Managed Testing Services
(MTS)
• Quality Management
Services
• Test Automation Services
• Testing Professional Services
• SAP Testing Services
• 3000+ people worldwide in
SCT space
• 30+ yrs experience
34
Sogeti’s Geographical Presence
A workforce of over 18,000 staff spread over 200 locations
USA
Sweden
NL
Ireland
Belux
UK
Denmark
Germany
Switzerland
France
Spain
India
35
Thank you!
Chris Hampton, PMP
Practice Manager, Software Control & Testing
Sogeti USA
[email protected]
Cell: (214) 549-7613
Sogeti Office: (972) 892-3400
36