Transcript Make your test automation project a success story
Slide 1
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 2
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 3
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 4
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 5
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 6
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 7
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 8
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 9
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 10
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 11
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 12
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 13
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 14
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 15
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 16
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 17
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 18
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 19
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 20
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 21
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 22
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 23
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 24
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 25
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 26
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 27
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 28
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 29
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 30
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 31
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 32
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 33
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 34
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 2
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 3
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 4
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 5
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 6
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 7
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 8
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 9
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 10
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 11
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 12
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 13
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 14
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 15
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 16
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 17
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 18
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 19
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 20
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 21
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 22
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 23
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 24
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 25
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 26
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 27
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 28
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 29
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 30
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 31
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 32
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 33
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]
Slide 34
The difference that makes the difference
Make your test automation project
a success story
Magnus Nilsson
Lemontree
Agenda
The Test automation concept
– Success factors
– Components
– Test cases
– Framework
Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data
from the script
Central data storage
Development project
– strategy and
– center of
excellence
Functional
components
Project
Combine components to test cases by
actions as input data
In-/outdata
Test automation
Central data
storage
Concept
Test Automation
Test automation - building up the components
Functionality available in test
– Development of the functionality
as it comes available
Phase 1
Phase 2
Func. 3
Func. 4
Requirement
Design and
development
Test
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a
component (or a component
itself)
Concept
Test Automation
Test automation - building up the components
Function 1: Create customer using CRM
CRMCustomer
6
Test automation - building up the components
Function 1: Create customer using Webshop
WebCustomer
7
Test automation - building up the components
Function 1: Create customer using mobile application
AppCustomer
8
Test automation – test cases
Test case 1:
Login
Customer
Account Credit check Subscription
[new]
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
Billing
Concept
Test Automation
Test automation – test cases
Test case 1:
Login
Account Credit check Subscription
[new]
Customer
ver
1.0
2.0
[create]
Order
[new]
Inventory
Test case 2:
Login
Account [new]
Customer [open]
ver
ver
2.0
1.0
Billing
Account [new]
ver
2.0
Concept
Test Automation
Test automation - development of a framework
1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework
Test automation - development of a framework
ALM
HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table
Library definitions, template of standard functions
Structured data handling, input and output data
Lin
kEn
tity
Firs
_In
tNa
me
Fa
mil
yNa
Offi
me
cial
Na
B
me
u
s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D
I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t
O
Entity linked from
previous component
M
First name of
customer
M
Family name
M
Private or Business
O
Business ID will be
validated if
provided
The created CRM
customer ID
N
/
A
N
/
A
Status of execution
Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine
List object
Text object
Table object
Function
library A
Function
library B
Repository
objects
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices
Structured data handling, input and output data
Development of a
Framework
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions
Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab
Data &
Structure
Test
action script
Function
library A
Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me
FamilyN
ame
OfficialN
ame
B
us
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
I
n
I
n
I
n
LinkEnti
I
ty_In
n
FirstNa
me
I
n
FamilyN
ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out
O
M
M
I
nM
I
n
O
ILinkEnti
n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u
in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out
Entity linked from previous
component
First name of customer
Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t
/
A
N
/
A
Status of execution
Data
Data &
Structure
Master DB
Development of a
Framework
The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development
System test
Phase
0
The Business Case
Acceptance test
Plan next phase
Documentation
& Training
Week
Phase 2
Detailed design
Development
…
Project
Walkthrough
Test automation project - business case
Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.
What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.
How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.
Project
Walkthrough
Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests
Number of test cycles
Execution time per test, h
1000
4
0.5
Execution cost per hour
100
Annual cost for execution of regression test 200000
Cost for error
handling
AT
Error source
Number of errors found per year
1000
Handling time per error (report, reproduce,
administer, test data, close defect)
2
4
Number of FTE's for error handling per year
2
2
Error handling cost per hour
100
100
Annual error handling costs
External and business
costs for errors
Cost multiplier for Production Errors
Grand total
500
200000
200000
0
100000
Annual error correction costs for errors
Business costs for productions errors1)
1)Goodwill,
Production
2
400000
1100000
compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
Test automation project - Business case
Description
Savings - Manual Test
Labour
Savings - Error
Reduction
Implementation Cost
(Year 1)
Reduction
ratio
Total savings
Acc reduction
Year 1
60%
120000
120000
Year 2
70%
140000
260000
Year 3
80%
160000
420000
Year 1
10%
90000
90000
Year 2
20%
180000
270000
Year 3
30%
270000
540000
Phase 1
Phase 2
Phase 3
Operational costs
(running)
Annual
reduction/cost
Year 1 proj support, SME
100000
70000
70000
50000
Year 2 TAC OP
60000
Year 3 TAC OP
60000
Acc savings
Year 1
-80 000
-80 000
Year 2
260000
180000
Year 3
370000
550000
Project
Walkthrough
Test automation project - Business case
Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer
Maintenance
Create
Analyis&Design
Selection
Mixed
Yes
10
Development
Value
2
20
1
Reusability
Terminate
5%
10
1
10
Simple TC
45
1.5
61
2
New
20
3
15
Create
1
5
10
2
20
WS Customer
Complex TC
Create
Framework
Test data
App Customer
Integrations
Create
Project
Walkthrough
Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 2
Week
Detailed design
Development
Test
cases
Planned
Delivered
Week
Project
Walkthrough
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough
Analysis
Analysis
–
–
Disposition
–
–
Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus
Structure
–
–
–
–
Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template
Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests
Implementation
–
After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Project
Walkthrough
Analysis structure - disposition
Use case
what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?
Example
Data
condition 1
Customer
Data
condition 2
Web Self Care
» which data conditions
are covered by use cases?
New
Private
Project
Business
Walkthrough
Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area
Process
Description
Prio
Create
Web shop
1
1
1
3
3
Sales
Web self care
Sales
Customer
operations
CRM
1
Sales
Device
App
2
Sales
Device
Online
2
Product
Inventory
Product
configurations
Account
Terminate
1
New
1
Modify
Subscription
Cancel
New
3
1
10
3
1
15
1
1
3
1
1
1
1
Project
Walkthrough
Requirement and component coverage
Components
CRM
Requirements
Customer
Folder
area
Test
case
Requirement
Process
Sales
TC23
Sales
TC12
Sales
TC1
Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1
Sales
TC45
Sales
TC6
Customer
op R4.1.1
Customer
op R4.1.1
Data condition
Test case
Prio
Create
Customer
Create
Customer
Terminate
customer
Create bank
account
1
Create invoice
account
1
1
Create
Account
Terminate
New
Modify
Customer
Cancel
Create
X
X
2
2
Web Shop
X
X
X
X
Project
Walkthrough
Coverage analysis and status report; BC follow up
Status
Not covered
1
1
Customer
Customer
operations
1
New
Not completed
Terminate
Not covered
Web self
care
New
Passed
Mobile
device
New
Failed
1
No run
3
Not
completed
Failed
2
Passed
7
2
1
2
1
1
3
1
3
3
1
0
2
0
2
2
3
3
Coverage analysis and status report, HP ALM
Coverage in ALM
Test either
– manual,
– automated or
– business
component
Project
Walkthrough
Maintenance
X
Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester
Automation Engineer
Project
Walkthrough
Project organisation - Use roles to enforce competence
Automation stakeholder
Automation
stakeholder
Subject Matter Expert
Project A
Automation stakeholder
Subject Matter Expert
Subject Matter Expert
Project B
Automation stakeholder
Automation Project Lead
Automation Solution Architect
Automation Analyst
Automation Engineer
Test Automation Center
Subject Matter Expert
Project C
Project
Walkthrough
Project organisation - Test automation center
Development
Execution
Maintenence
Project
Walkthrough
Management
Test automation center
Test management
Release management
Project planning
Defect and Issue
Management
Change request
Resolution
Supplier
Operations
Communicating
Reporting
Planned activities
TAC Service Management
Project A
Project B
Project C
Request
Maintenance
ST
SIT
AT
Result
TAC
Execution
TAC
Maintenance
TAC
Development
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
•
Quality Management
•
eGovernment
•
Enterprise Solutions
Quality Management AQUA solutions
Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]