Transcript TIME BOXED TESTING

TIME BOXED TESTING
BCS SIGIST
13th July 1998
Graham Thomas - OSI Group
1
Abstract
There is great pressure upon developers today to improve productivity and
effectiveness. To achieve this there is a move away from the traditional
structured methodologies towards more dynamic, iterative and RAD
approaches.
This is being combined with Object and Component based techniques, and
delivered with a new generation of IDE’s, to produce thin client, web
based, voice and data products.
So how do we test it?
 Propose Time Boxed Testing
 Outline an approach
 Discuss the problems
 Give a flavour of the fun ?
2
Agenda
 Development
Lifecycle
 Testing Lifecycle
 Testing Helix
 Testing Stage by Stage
 Traceability
 Testing Process
 How to sell time boxed testing
 Lessons Learnt
3
The Project
 Telephony
based project
 2 types of proprietary hardware
 Client Server running Windows NT
 Interfaces to legacy systems
 Rational Rose case tool (using UML)
 Objectory process
 Visual Studio
 Was considering Java & thin clients
 Incremental/Iterative development
4
Incompatible Models?
Rqmts.
UAT
Sys Test
Analysis
Design
Breadth
Iterative (RAD) Model
Build
Int Test
Unit Test
Classic V Model
5
Difficult to Represent Test Stages
unit
Component
Integration
unit
Component
Integration
Depth
unit
Component
Integration
Breadth
System
Acceptance
6
Project Lifecycle


...
7
Testing Lifecycle


...
8
A dual spiral life cycle ?


B R
D A
Rqmts
Analysis
Design
Build
E P
C D
...
Plan
Design
Construct
Execute
9
Testing Lifecycle
 Unit Testing
 Component
Integration Testing
 System Testing
 Acceptance Testing
 Year 2000
Millenium Compliance
 Testing
Processes and Activities
 Tools & Traceability
 Resources
 Planning
1
Scope of Test Phases
Conical
Model
Component
Unit
Testing Integration System
Testing
Testing
Iterative
User
Acceptance
Testing
Release
1
Tuned Test Lifecycle
Testing Activities Brought Forward
Component
Unit
Testing Integration
Testing
Iterative
System
Testing
- Non functional
Requirements
- Performance
Analysis
User
Acceptance
Testing
- Scenario Testing
- Acceptance
Criteria
Release
1
Unit Testing
 Iterative
Time-Boxed
Activity
 Specific to delivered
components
 ‘White Box’ testing
techniques employed
 External Interfaces
tested
 Use of tools for
Static & Dynamic
analysis
Unit Testing
Sub Unit
Testing
Unit
Integration
Testing
External
Interface
Testing
Unit
Testing
Static
Analysis
Dynamic
Analysis
Quality
Assurance
1
Unit Testing
Sub
Unit
Unit
Package
Unit
Sub
Unit
Sub
Unit
Sub
Unit
Unit
Sub
Unit
Package
Unit
Sub
Unit
Unit
Sub
Unit
Sub
Unit
Sub
Unit
Sub Unit Testing
Unit
Unit Testing
Package
Unit Integration Testing
Previously tested & checked-in
Stubs, Drivers, Harnesses
1
Component Integration Testing
 Iterative
Time-Boxed
Activity
 Testing prioritised by
Business
 End to End process
threads where
possible
 Contains elements of
following test phases
 Build regression test
packs
Component
Integration
Package
Integration
Performance
Analysis
Non Functional
Requirements
Component
Integration
Testing
Functional
Requirements
Regression Test
Scenario
Regression
Test
1
Component Integration Testing
Step 1 - Package Integration
Component
Package
Package Integration
must precede
Component Integration
Package
Component
Package
Package
Component
1
Component Integration Testing
Step 2 - Component Integration
Component
Component
Component
Component
Component
Drop1
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Drop2
...
1
Component Integration Testing
Drop by Drop


Component
Component Component
Component
Component
Component
Component
Component
Component
Component
Component Component
Component
Component
Component
Component
Component
Component
Component
Component
ComponentComponent
Component
Component
Component
Component
Component
Component
Component
Component
...
Automated Regression Test Pack
1
System Testing
 One
off activity
 Structured testing
techniques
 End to End process
threads
 Ties together the
development
iterations
 Regression test
already constructed
Non Functional
Requirements
Functional
Requirements
Regression Test
System
Testing
Performance
Analysis
Design
Verification
1
Acceptance Testing
 One
off activity
 Test that the computer
system integrates with
business processes
 Structured testing
techniques
 End to End process
threads
 Regression test
already constructed
Scenario
Regression
Testing
Acceptance
Criteria
Testing
Business
Process
Testing
Acceptance
Testing
Data Take On
& Conversion
Testing
Usability
Testing
2
Y2000 - Millennium Compliance
6 Date Scenarios
This
Century
Across
Wholly
Century
next
Boundary Century
Jan 1st
Feb 29th
Dec 31st
 Separate
2 Business Tests
Financial
Year ?
Fiscal
Periods ?
test runs
 Carried out in
sequence
 Some tests can be
exercised post Y2000
2
Testing Lifecycle Traceability
High Level
Function
User
Requirements
Functional
Requirement
Non FR
Use Case
Development
Repository
Scenario
Test Script
Test Planning
Test Case
2
Testing Processes and Activities
Process
Quality
Interface
Problem
Mgmt
Conf. Mgmt.
Interface
Test Management
Activity
Entry
Test Execution
Test Reporting
Test Planning
Test Automation Monitor
Test Design
Regression Test
Measure
Test Preparation
Manage
Analysis of Change
Test Stage
Sign off
Exit
2
Processes
 Essential

Configuration Management

Impact Analysis using a Test Planning repository

Quality Management
 Good
practice

Fault Reporting - on the Intranet - Live

Gathering metric information
2
Tools

Full set of testing tools is required

Static analysis

Dynamic Analysis

Interactive debugging

Test Automation/Capture Replay essential

Test Planning Repository

Performance monitoring (stress/load)

Fault Reporting system
2
Monitoring
Cumulative Error Curve
140
120
100
80
60
40
20
0
1
2
3
4
5
6
7
8
9
10
11
12
Fault
13
14
15
16
17
2
Monitoring
Outstanding
50
45
40
35
30
25
20
15
10
5
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
2
Monitoring
Faults by Priority
140
120
100
80
A
B
C
60
40
20
0
1
2
3
4
5
6
7
8
9
10
11
12
13
2
Monitoring
Aged Defect Analysis by Priority
30
25
20
A
B
C
15
10
5
0
W1
W2
W3
W4
2
Test Resources
Unit
Testing
Component
Integration
Testing
System
Testing
User
Acceptance
Testing
TEST
TEAM
Developers
Business
Users
Developers
Business
Users
Testers
Secondment
3
Test Planning
Decision
Test Approach
Definition
Test Strategy
Testing Process
Development
Test drop 
Test drop
Test drop 
Non Functional Testing
Delivery
System Test
Acceptance Test
Y2000 Test
Time-boxed Testing
3
Selling the Approach
 Target
the following;

Test Team

Systems Architect

Development team

Standards & Methods

Business Representatives

Project Sponsor
 Via


Animated Presentation
Test Strategy document
3
So how easy was this?
3
So What went well?
 Systems Architect
 The
liked it 
presentation was well received 
3
What was difficult?
 Initial
cool response from other members of the
test team
 Development Team

Dev. Why test anyway, our code is wonderful and our
test harnesses will do all the testing?
 Business



were uneasy
Representatives were unhappy about;
Anything less than 100% testing
Bringing their effort forward in the development
lifecycle
The difference between the ‘Use Case’ modeling they
had done, and the test planning that they would need to
carry out
3
What was difficult? (2)
 Standards
& Methods
Saw nothing wrong with their approach, it was either;
• structured in which case their approach worked fine,
or
• it was incremental, in which case their approach
worked fine.
• They did not do iterative development, but if they
did their approach would still work.
This took several meetings to gain approval.
 Project

Sponsor
Cancelled the project. (Time Boxed Testing was not
cited as the reason !!!)
3
So what fun did we have ?
3
What did I learn from this?

Organisations change their development approach
more quickly than their testing approach

There is a reluctance to do anything less than 100%
testing

Testing is a key and integral part of any development
lifecycle

The more you think about it, the more sense Time
Boxed Testing makes
3
Contact Information
Graham Thomas
OSI
the management support company
e-mail: [email protected]
Phone: 0171 323 3353
Copies of slides including animated gifs at
www.osi.co.uk
3