Make your test automation project a success story

Download Report

Transcript Make your test automation project a success story

Slide 1

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 2

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 3

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 4

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 5

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 6

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 7

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 8

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 9

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 10

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 11

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 12

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 13

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 14

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 15

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 16

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 17

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 18

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 19

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 20

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 21

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 22

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 23

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 24

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 25

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 26

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 27

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 28

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 29

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 30

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 31

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 32

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 33

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]


Slide 34

The difference that makes the difference

Make your test automation project
a success story
Magnus Nilsson
Lemontree

Agenda
 The Test automation concept
– Success factors
– Components
– Test cases
– Framework
 Walkthrough of a test automation project
– Analysis phase and the business case
– Maintenance of an automated solution

Test automation - success factors


Develop functional components



Decouple in- and output data
from the script



Central data storage



Development project
– strategy and
– center of
excellence



Functional
components

Project

Combine components to test cases by
actions as input data

In-/outdata
Test automation

Central data
storage
Concept

Test Automation

Test automation - building up the components
 Functionality available in test
– Development of the functionality
as it comes available
Phase 1

Phase 2

Func. 3
Func. 4

Requirement

Design and
development

Test

Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2

Each functionality is part of a
component (or a component
itself)
Concept

Test Automation

Test automation - building up the components
 Function 1: Create customer using CRM
CRMCustomer

6

Test automation - building up the components
 Function 1: Create customer using Webshop
WebCustomer

7

Test automation - building up the components
 Function 1: Create customer using mobile application
AppCustomer

8

Test automation – test cases
Test case 1:

Login

Customer

Account Credit check Subscription
[new]

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

Billing

Concept

Test Automation

Test automation – test cases
Test case 1:

Login

Account Credit check Subscription
[new]

Customer

ver
1.0
2.0

[create]

Order
[new]

Inventory

Test case 2:

Login

Account [new]

Customer [open]

ver
ver
2.0
1.0

Billing
Account [new]
ver
2.0

Concept

Test Automation

Test automation - development of a framework
 1st generation script
– Using automation tool for capture and replay only
– Hardcoded data
 2nd generation script
– Code refactoring for maintainable scripts
– Uses generic functions for iterations and error handling
 3rd generation script
– Data driven test (DDT) where dynamic data is taken from data table rows
– Keyword driven test (KDT) where action words controls the actions
– Use of driver scripts for executing multiple test cases
Development of a
– SME for test designs and requirement
– More complex scripting by developers
Framework

Test automation - development of a framework
ALM

 HP QuickTest Professional/Functional tester
– Framework itself with
• test management connection ALM,
Recovery
scenario
• keyword driven action scripts with
• data table,
• function libraries,
Test
action script
• Repositories,
• BPT,
• recovery scenario
– Generates code
Data table



Library definitions, template of standard functions
Structured data handling, input and output data

Lin
kEn
tity
Firs
_In
tNa

me
Fa

mil
yNa
Offi
me
cial
Na
B
me
u

s
Custi
n
omer
e
ID_O
Scrip
s
ut
tstat
s
e_Ou
I
t D

I
n
I
n
I
n
I
n
I
n
O
u
t
O
u
t

O

Entity linked from
previous component

M

First name of
customer

M

Family name

M

Private or Business

O

Business ID will be
validated if
provided
The created CRM
customer ID

N
/
A
N
/
A

Status of execution

Dashboard
Test resources
Test plan
Test lab
Application
GUI layer
Object
recognition
engine

List object
Text object
Table object

Function
library A
Function
library B

Repository
objects

Development of a

Framework

Test automation - development of a framework


HP QuickTest Professional/Functional tester

 Library definitions, template of standard functions
– Application specific libraries
– Standard actions in applications should be in application library
– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev
• Multiple checkout and merge
• Only driver scripts
– Framework libraries handling framework and best practices


Structured data handling, input and output data
Development of a

Framework

Test automation - development of a framework



HP QuickTest Professional/Functional tester
Library definitions, template of standard functions

 Structured data handling, input and output data
– Central data source
– Decouple structure and data
– Central structured data source
ALM
Dashboard
Test resources
Test plan
Test lab

Data &
Structure

Test
action script

Function
library A

Data table
Component A sheet
Component B sheet
CreateCustomer sheet
LinkEnti
ty_In
FirstNa
me

FamilyN
ame
OfficialN
ame
B
us

in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

I
n

I
n

I
n
LinkEnti
I
ty_In
n
FirstNa
me
I

n
FamilyN

ame
O
OfficialN
u
ame
t
O
Bu
us
t
in
es
Customer
sI
ID_Out
D
Scriptstat
e_Out

O

M

M

I
nM
I
n
O
ILinkEnti

n
ty_In
N
IFirstNa
/
nme
A
N
IFamilyN
/
name
A
OfficialN
O
ame
u
tB
Ous
u

in
tes
Customer
sI
ID_Out
D
Scriptstat
e_Out

Entity linked from previous
component
First name of customer

Family name
Entity linked from previous
O
Private or Business component
M
First name of customer
Business ID will be validated
if provided
Entity linked from previous
O
M
I
Family
name
component
n
The
ID
MI created CRM customer
Private
or Business First name of customer
M
n
Status
of execution Business ID will be validated
OI
M
Family name
if provided
n
M
I
Private or Business
The created CRM customer ID
nN
/
Business ID will be validated
O
I A
N
Status of execution if provided
n/
OA
N
The created CRM customer ID
u
t
O
u
t

/
A
N
/
A

Status of execution

Data

Data &
Structure

Master DB

Development of a

Framework

The test automation project
Phase 1
Analysis
Detailed design
Requirement
Development

System test
Phase
0

The Business Case

Acceptance test
Plan next phase

Documentation
& Training

Week

Phase 2
Detailed design
Development
…

Project

Walkthrough

Test automation project - business case
 Why do we need a business case?
– Test automation cost time and money
– The automated solution must at least ensure the time spent in development,
maintenance and execution is less than a manual solution would cost to accomplish the
same goal.

 What is a test automation business case?
– List of all costs and savings associated with test that possibly can be covered by test
automation.
– Not all drivers are possible to calculate directly as quality. But quality can be expressed
as an estimation of the number of testing hours to ensure the desired level of quality.
– Additional test automation drivers as repeated executions is calculated indirectly as
increased quality.

 How can a business case be used?
– Business case can be used to predict effectiveness as automation deliveries should
follow the calculated schedule.
– ROI should be calculated at regular audits based on the business case.

Project

Walkthrough

Test automation project - business case
Total test case executions per cycle
Cost for manual
regression tests

Number of test cycles
Execution time per test, h

1000
4
0.5

Execution cost per hour

100

Annual cost for execution of regression test 200000
Cost for error
handling

AT

Error source
Number of errors found per year

1000

Handling time per error (report, reproduce,
administer, test data, close defect)

2

4

Number of FTE's for error handling per year

2

2

Error handling cost per hour

100

100

Annual error handling costs
External and business
costs for errors

Cost multiplier for Production Errors

Grand total

500

200000

200000

0

100000

Annual error correction costs for errors

Business costs for productions errors1)

1)Goodwill,

Production

2
400000
1100000

compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest
industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

Test automation project - Business case
Description
Savings - Manual Test
Labour

Savings - Error
Reduction

Implementation Cost
(Year 1)

Reduction
ratio

Total savings

Acc reduction

Year 1

60%

120000

120000

Year 2

70%

140000

260000

Year 3

80%

160000

420000

Year 1

10%

90000

90000

Year 2

20%

180000

270000

Year 3

30%

270000

540000

Phase 1
Phase 2
Phase 3

Operational costs
(running)

Annual
reduction/cost

Year 1 proj support, SME

100000
70000

70000
50000

Year 2 TAC OP

60000

Year 3 TAC OP

60000

Acc savings

Year 1

-80 000

-80 000

Year 2

260000

180000

Year 3

370000

550000

Project

Walkthrough

Test automation project - Business case
 Implementation cost calculations
– Initial estimates
– Design estimates
Component/Functions
CRM
Technology
Customer

Maintenance
Create

Analyis&Design
Selection

Mixed
Yes
10

Development
Value

2

20
1

Reusability
Terminate

5%
10

1
10

Simple TC

45

1.5

61

2

New
20

3
15

Create

1

5
10

2
20

WS Customer
Complex TC
Create
Framework

Test data
App Customer
Integrations
Create

Project

Walkthrough

Business case – follow up short term
Phase 1
Analysis
Detailed design
Development
System test
Acceptance test

Plan next phase
Phase 2

Week

Detailed design
Development

Test
cases

Planned

Delivered

Week

Project

Walkthrough

Business case – follow up long term
”Test automation strategy”








Description of the standard approach on how to use the tools
Company specific set of rules on how automation should be done
Generic solution that is not dependent on individuals
Maintenance focused
Encapsulated by test automation centre
Enforced by framework
Ensured by the automation project
Project
Walkthrough

Analysis


Analysis
–
–



Disposition
–
–



Decision making: if we go live what does and does not work?
What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital
• Test and defect status itself often only aids prioritizing of resources/focus

Structure
–
–
–
–



Analysis based on existing documentation, test cases, use cases and requirements (static test review)
Provides the prioritization to an analysis structure template

Common and clear reporting on a level that enables correct business decisions
Structured test design and coverage analysis
Will control the test automation deliveries
Used in test management to create common priorities for both manual and automated tests

Implementation
–

After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM

Project

Walkthrough

Analysis structure - disposition
Use case

 what is the system used for?
– how is it done?
Process
Requirement • which are the requirements on process level?

Example

Data
condition 1

Customer

Data
condition 2

Web Self Care

» which data conditions
are covered by use cases?

New

Private

Project

Business
Walkthrough

Analysis structure - disposition
Use
cases/product
Requirements
Customer
Folder area

Process

Description

Prio

Create

Web shop

1

1
1
3
3

Sales

Web self care

Sales

Customer
operations

CRM

1

Sales

Device

App

2

Sales

Device

Online

2

Product

Inventory

Product
configurations

Account

Terminate

1

New

1

Modify

Subscription
Cancel

New

3

1

10

3

1

15

1
1

3
1

1

1

1

Project

Walkthrough

Requirement and component coverage
Components

CRM

Requirements
Customer
Folder
area

Test
case

Requirement
Process

Sales

TC23

Sales

TC12

Sales

TC1

Web self care
R1.1
Customer
op R2.3.4
Customer
op R2.4.1

Sales

TC45

Sales

TC6

Customer
op R4.1.1
Customer
op R4.1.1

Data condition
Test case

Prio

Create
Customer
Create
Customer
Terminate
customer
Create bank
account

1

Create invoice
account

1

1

Create

Account

Terminate

New

Modify

Customer
Cancel

Create

X
X

2
2

Web Shop

X
X

X
X

Project

Walkthrough

Coverage analysis and status report; BC follow up
Status

Not covered

1
1

Customer
Customer
operations

1
New

Not completed

Terminate

Not covered

Web self
care
New

Passed

Mobile
device

New

Failed

1

No run

3

Not
completed

Failed

2

Passed

7

2

1

2

1

1

3

1

3
3

1
0

2

0

2

2

3
3

Coverage analysis and status report, HP ALM

 Coverage in ALM
 Test either
– manual,
– automated or
– business
component

Project

Walkthrough

Maintenance




X

Test cases created
Test data prepared
Result reporting arranged
Press the button
Then what?
Functional tester

Automation Engineer

Project

Walkthrough

Project organisation - Use roles to enforce competence
Automation stakeholder

Automation
stakeholder
Subject Matter Expert
Project A

Automation stakeholder

Subject Matter Expert
Subject Matter Expert

Project B

Automation stakeholder

Automation Project Lead
Automation Solution Architect
Automation Analyst

Automation Engineer
Test Automation Center

Subject Matter Expert

Project C

Project

Walkthrough

Project organisation - Test automation center

Development
Execution
Maintenence

Project

Walkthrough

Management

Test automation center

Test management
Release management
Project planning

Defect and Issue
Management

Change request

Resolution

Supplier

Operations

Communicating
Reporting
Planned activities

TAC Service Management

Project A
Project B
Project C
Request

Maintenance

ST

SIT

AT

Result

TAC
Execution

TAC
Maintenance

TAC
Development

Short about Lemontree


Swedish company established 1999



Awarded Gasell and Superföretag 2008-2012



Turnover +98 MSEK 2011



AAA-rated



About 100 employees, growing intensive



Owners are actively working in the company, no external capital



Offices in Stockholm and Oslo



QM trainee program since 2012



Focus areas
•

Quality Management

•

eGovernment

•

Enterprise Solutions

Quality Management AQUA solutions

Thanks!
Magnus Nilsson
Lemontree Solution Manager
[email protected]