TIME BOXED TESTING
Download
Report
Transcript TIME BOXED TESTING
TIME BOXED TESTING
BCS SIGIST
13th July 1998
Graham Thomas - OSI Group
1
Abstract
There is great pressure upon developers today to improve productivity and
effectiveness. To achieve this there is a move away from the traditional
structured methodologies towards more dynamic, iterative and RAD
approaches.
This is being combined with Object and Component based techniques, and
delivered with a new generation of IDE’s, to produce thin client, web
based, voice and data products.
So how do we test it?
Propose Time Boxed Testing
Outline an approach
Discuss the problems
Give a flavour of the fun ?
2
Agenda
Development
Lifecycle
Testing Lifecycle
Testing Helix
Testing Stage by Stage
Traceability
Testing Process
How to sell time boxed testing
Lessons Learnt
3
The Project
Telephony
based project
2 types of proprietary hardware
Client Server running Windows NT
Interfaces to legacy systems
Rational Rose case tool (using UML)
Objectory process
Visual Studio
Was considering Java & thin clients
Incremental/Iterative development
4
Incompatible Models?
Rqmts.
UAT
Sys Test
Analysis
Design
Breadth
Iterative (RAD) Model
Build
Int Test
Unit Test
Classic V Model
5
Difficult to Represent Test Stages
unit
Component
Integration
unit
Component
Integration
Depth
unit
Component
Integration
Breadth
System
Acceptance
6
Project Lifecycle
...
7
Testing Lifecycle
...
8
A dual spiral life cycle ?
B R
D A
Rqmts
Analysis
Design
Build
E P
C D
...
Plan
Design
Construct
Execute
9
Testing Lifecycle
Unit Testing
Component
Integration Testing
System Testing
Acceptance Testing
Year 2000
Millenium Compliance
Testing
Processes and Activities
Tools & Traceability
Resources
Planning
1
Scope of Test Phases
Conical
Model
Component
Unit
Testing Integration System
Testing
Testing
Iterative
User
Acceptance
Testing
Release
1
Tuned Test Lifecycle
Testing Activities Brought Forward
Component
Unit
Testing Integration
Testing
Iterative
System
Testing
- Non functional
Requirements
- Performance
Analysis
User
Acceptance
Testing
- Scenario Testing
- Acceptance
Criteria
Release
1
Unit Testing
Iterative
Time-Boxed
Activity
Specific to delivered
components
‘White Box’ testing
techniques employed
External Interfaces
tested
Use of tools for
Static & Dynamic
analysis
Unit Testing
Sub Unit
Testing
Unit
Integration
Testing
External
Interface
Testing
Unit
Testing
Static
Analysis
Dynamic
Analysis
Quality
Assurance
1
Unit Testing
Sub
Unit
Unit
Package
Unit
Sub
Unit
Sub
Unit
Sub
Unit
Unit
Sub
Unit
Package
Unit
Sub
Unit
Unit
Sub
Unit
Sub
Unit
Sub
Unit
Sub Unit Testing
Unit
Unit Testing
Package
Unit Integration Testing
Previously tested & checked-in
Stubs, Drivers, Harnesses
1
Component Integration Testing
Iterative
Time-Boxed
Activity
Testing prioritised by
Business
End to End process
threads where
possible
Contains elements of
following test phases
Build regression test
packs
Component
Integration
Package
Integration
Performance
Analysis
Non Functional
Requirements
Component
Integration
Testing
Functional
Requirements
Regression Test
Scenario
Regression
Test
1
Component Integration Testing
Step 1 - Package Integration
Component
Package
Package Integration
must precede
Component Integration
Package
Component
Package
Package
Component
1
Component Integration Testing
Step 2 - Component Integration
Component
Component
Component
Component
Component
Drop1
Component
Component
Component
Component
Component
Component
Component
Component
Component
Component
Drop2
...
1
Component Integration Testing
Drop by Drop
Component
Component Component
Component
Component
Component
Component
Component
Component
Component
Component Component
Component
Component
Component
Component
Component
Component
Component
Component
ComponentComponent
Component
Component
Component
Component
Component
Component
Component
Component
...
Automated Regression Test Pack
1
System Testing
One
off activity
Structured testing
techniques
End to End process
threads
Ties together the
development
iterations
Regression test
already constructed
Non Functional
Requirements
Functional
Requirements
Regression Test
System
Testing
Performance
Analysis
Design
Verification
1
Acceptance Testing
One
off activity
Test that the computer
system integrates with
business processes
Structured testing
techniques
End to End process
threads
Regression test
already constructed
Scenario
Regression
Testing
Acceptance
Criteria
Testing
Business
Process
Testing
Acceptance
Testing
Data Take On
& Conversion
Testing
Usability
Testing
2
Y2000 - Millennium Compliance
6 Date Scenarios
This
Century
Across
Wholly
Century
next
Boundary Century
Jan 1st
Feb 29th
Dec 31st
Separate
2 Business Tests
Financial
Year ?
Fiscal
Periods ?
test runs
Carried out in
sequence
Some tests can be
exercised post Y2000
2
Testing Lifecycle Traceability
High Level
Function
User
Requirements
Functional
Requirement
Non FR
Use Case
Development
Repository
Scenario
Test Script
Test Planning
Test Case
2
Testing Processes and Activities
Process
Quality
Interface
Problem
Mgmt
Conf. Mgmt.
Interface
Test Management
Activity
Entry
Test Execution
Test Reporting
Test Planning
Test Automation Monitor
Test Design
Regression Test
Measure
Test Preparation
Manage
Analysis of Change
Test Stage
Sign off
Exit
2
Processes
Essential
Configuration Management
Impact Analysis using a Test Planning repository
Quality Management
Good
practice
Fault Reporting - on the Intranet - Live
Gathering metric information
2
Tools
Full set of testing tools is required
Static analysis
Dynamic Analysis
Interactive debugging
Test Automation/Capture Replay essential
Test Planning Repository
Performance monitoring (stress/load)
Fault Reporting system
2
Monitoring
Cumulative Error Curve
140
120
100
80
60
40
20
0
1
2
3
4
5
6
7
8
9
10
11
12
Fault
13
14
15
16
17
2
Monitoring
Outstanding
50
45
40
35
30
25
20
15
10
5
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
2
Monitoring
Faults by Priority
140
120
100
80
A
B
C
60
40
20
0
1
2
3
4
5
6
7
8
9
10
11
12
13
2
Monitoring
Aged Defect Analysis by Priority
30
25
20
A
B
C
15
10
5
0
W1
W2
W3
W4
2
Test Resources
Unit
Testing
Component
Integration
Testing
System
Testing
User
Acceptance
Testing
TEST
TEAM
Developers
Business
Users
Developers
Business
Users
Testers
Secondment
3
Test Planning
Decision
Test Approach
Definition
Test Strategy
Testing Process
Development
Test drop
Test drop
Test drop
Non Functional Testing
Delivery
System Test
Acceptance Test
Y2000 Test
Time-boxed Testing
3
Selling the Approach
Target
the following;
Test Team
Systems Architect
Development team
Standards & Methods
Business Representatives
Project Sponsor
Via
Animated Presentation
Test Strategy document
3
So how easy was this?
3
So What went well?
Systems Architect
The
liked it
presentation was well received
3
What was difficult?
Initial
cool response from other members of the
test team
Development Team
Dev. Why test anyway, our code is wonderful and our
test harnesses will do all the testing?
Business
were uneasy
Representatives were unhappy about;
Anything less than 100% testing
Bringing their effort forward in the development
lifecycle
The difference between the ‘Use Case’ modeling they
had done, and the test planning that they would need to
carry out
3
What was difficult? (2)
Standards
& Methods
Saw nothing wrong with their approach, it was either;
• structured in which case their approach worked fine,
or
• it was incremental, in which case their approach
worked fine.
• They did not do iterative development, but if they
did their approach would still work.
This took several meetings to gain approval.
Project
Sponsor
Cancelled the project. (Time Boxed Testing was not
cited as the reason !!!)
3
So what fun did we have ?
3
What did I learn from this?
Organisations change their development approach
more quickly than their testing approach
There is a reluctance to do anything less than 100%
testing
Testing is a key and integral part of any development
lifecycle
The more you think about it, the more sense Time
Boxed Testing makes
3
Contact Information
Graham Thomas
OSI
the management support company
e-mail: [email protected]
Phone: 0171 323 3353
Copies of slides including animated gifs at
www.osi.co.uk
3