Transcript IBM Presentations: Blue Pearl DeLuxe template

Strategic Direction for
Functional Test Automation
SoftTest 2005
1
Susan Windsor
Insight Through Intelligence
WMHL Consulting Limited, MD
AGENDA
2

Future for Functional Testing

History of Functional Automation

Strategic Direction

Automation Frameworks in Action

Impact upon You?
Future for
Functional
Testing
3
Increased Demand………..?




4
Global development, with large projects having multi
site, multi geography and multi suppliers to contend
with.
Corporate and regulatory requirements growing all
the time
Business demands new applications, faster and
cheaper to obtain competitive advantage
Closer alignment of IT and business to repair lack of
business confidence
..Or, increased pressures?



5
Automation frameworks and greater business
knowledge requirements, mean reduced numbers of
traditional functional testers
Agile development methods mean developers
undertake more unit and component testing
Growth of outsourced testing to different geographies,
leading to greater competition for the roles
Is the role of the functional tester
(who is neither technical or business specialist) dead?
History of
Functional
Automation
6
Record and Playback





“all you need to do…….…!”
Easy to record the scripts
Extremely fragile
Expensive to maintain
On average, each test run requires at least 50% of
the scripts to be recorded again
Please tell me no one does this anymore!!!
7
Scripting






8
Use the scripting language to write scripts that do what
you want
Build in as much robustness as possible
But, you’re building an application to test an application,
which also needs testing!
Maintenance costs can be very high
Needs programmer skills
Cost of implementation not affordable at project level
Table Driven





9
Like scripting but more flexible and greater re-use
Remove items from the script that change, e.g. data
Reduces maintenance costs a bit
Implementation costs about the same
Still requires specialist skills to implement
Functional Test
Automation is Broken!





10
Focus on technology rather than business needs
80% of functional testing still manual
60% to 70% of automation tools used for nonfunctional testing
Typically, traditional functional automation stops at
100 scripts, regardless of test coverage requirement
Critical factors; cost of implementation and
maintenance, skills required and asset sharing over
different technologies
Strategic
Direction
11
Automation Frameworks

Wouldn’t it be good if……..
–
–
–
–
–
12
Tests could be documented in common format, regardless of
whether they are manual or automated
The format for the tests resulted in quicker preparation time than
traditional manual tests
Both developers and business testers could understand and use
the same test format
Script maintenance was no longer a requirement
Tests could use any of the test execution tools required by the
underlying technology
Faster planning, faster execution, and far less technical skill required
Business Analysts already using
them, and use will grow


Home grown frameworks built within organisations to
meet business demands
Niche suppliers providing frameworks; try a Google
search – 512,000 hits this week
–
–

13
Seen a handful that appear mature
Latest review by Paul Herzlich (OVUM analyst)
Market Leaders such as Mercury developing
Business Process Tester (BPT)
This is the industry direction now
Automation
Frameworks
in Action
14
The project






15
Lloyds TSB Offshore in Channel Islands
Migration from current banking platform IBIS to Temenos Globus G13
Tight deadlines due to EU Savings Tax Directive coming into force on
1st July 2005
Business processes required additional Globus modules
Multiple releases leading up to deadline required substantial testing
SQS-UK contacted to provide consultancy
The Team





16
SQS-UK recognised the requirement for test automation
Joint team from Odin Technology and SQS-UK assembled
Working alongside Lloyds TSB staff
Odin Technology provided testing technology
SQS-UK provided implementation resource and testing know-how
Project Approach
Split project roles for Automation Technicians and Testers
–
Three phase technical approach



–
Tool Evaluation & Technical Feasibility
Technical Implementation
Support
Test Analysis and Design throughout
Tool
Evaluation
Technical
Implementation
Support
Technician
Tester
Test Analysis & Design
Project Time
17
Project Phase 1:
Tool Evaluation and Technical
Feasibility
18
Tool Evaluation




Automation isn’t always straightforward!
GUI Tools don’t work “out of the box” for all environments
They will almost certainly require some custom functionality to
interact with the SUT
The Evaluation process:
–
–
–

19
Assess what custom functionality is required
Assess what involvement is required from an Automation
Technician to provide the custom functionality
Assess if there are any other efficiency improvements that can be
made through application of technology
This provides the scope of technical implementation phase
Project Phase 2: What would
normally happen...
20
Typical Automation Project

Take existing Manual Scripts
Rework by hand into automated scripts & function libraries

The bulk of the work…

Building the Object Map

–

Coding Test Scripts
–
21
Providing the Mapping between the business terms used to
describe the SUT Interface and the test tools requirements
Providing the instructions and associated data for the test tool to
interact with the SUT and perform testing
Object Mapping
Providing the Mapping between the business terms used to
describe the SUT Interface and the test tools’ requirements
Application
22
Example
Manual Tester
“The Login Window”
Test Tool
23
class: MSWDialog
label: “Login.*”
Example 2
Manual Tester
“Username”
Test Tool
24
class: HTMLEdit
id: Lgn_Uid
Object Mapping in a typical
Automation Tool

Usually implemented under different names in different tools
–
–
–

Mapping an Object
–
–
–
–
–
25
GUI Map
Object Repository
Object Map
Start the Object recognition tool
Point at the Object with the Mouse Pointer
Select the Object
Give the Object a meaningful business name
Make manual amendments to the properties if required
Test Script Creation
Providing the instructions and associated data for the test tool to
interact with the SUT and perform testing
Script 1.1
Script 1.1
AxeMainAPI.TestBegin "6"
rc=AxeMainAPI.CheckDependency("5")
If rc <> 0 Then
AxeMainAPI.TestBegin "6"
rc=AxeMainAPI.CheckDependency("5")
ExitRun(rc)
End If If rc <> 0 Then
AxeMainAPI.BasestateBegin "Home"
ExitRun(rc)
Browser("Browser").Basestate(AXEDIR
End If
& "samples\OdinPortal\html\index.html")
AxeMainAPI.BasestateBegin
AxeMainAPI.BasestateEnd
rc,"","" "Home"
if rc <> Browser("Browser").Basestate(AXEDIR
0 then
& "samples\OdinPortal\html\index.html")
AxeMainAPI.BasestateEnd rc,"",""
rc=AxeMainAPI.TestAbort
if rc <> 0 then
rc=AxeMainAPI.TestAbort
26
Scripting Approach
Use the Tools Scripting Language to construct business processes
Function Login(ByVal UID As String, ByVal PWD As
String)
Dim rc
rc = wait_window(“Login”,30)
if rc <> 0 Then
log_msg(“Window Not Found”, FALSE)
Return
End if
focus_window(“Login”)
click_on_text (“UserName”)
type(UID & “{Return}”)
click_on_text (“UserName”)
type(PWD & “{Return}”)
27
End Function
Script 1.1
Script 1.1
Script 1.1
1. Open Application
2. Enter Userid for Authorised User
Open Application
3. Enter1.Password
for User
Enter Userid for Authorised User
4. Click 2.
Login
Open Application
Enter1.Password
User
5. Select3. Create
new user for
from
menu
2.
Enter
Userid for Authorised User
Click Login
6. Enter4.postcode
and house number
Enter Password
User
5. Select3.user
Create
new user for
from
menu
7. Tick “Secure”
authority
Click Login
Enter4.password
postcode
and house number
8. Enter6.unique
5. Select user
Create
new user from menu
7. Tick “Secure”
authority
9. Hit Generate
userid
6.
Enter
postcode
and
house
number
8. Enter
unique
password
10. Validate
userid
is 8chars
long
7. Tick “Secure”
user authority
9. down
Hit Generate
userid
11. Write
userid for
later use
8. Enter
unique
password
10. Validate
userid
is 8chars
long
9. down
Hit Generate
userid
11. Write
userid for
later use
10. Validate userid is 8chars long
11. Write down userid for later use
Script 1.1
Script 1.1
AxeMainAPI.TestBegin "6"
rc=AxeMainAPI.CheckDependency("5")
If rc <> 0 Then
AxeMainAPI.TestBegin "6"
Run(rc) rc=AxeMainAPI.CheckDependency("5")
If rc <> 0 Then
End If
AxeMainAPI.BasestateBegin "Home"
Run(rc)
Browser("Browser").Basestate(AXEDIR
&
End If
"samples\OdinPortal\html\index.html")
AxeMainAPI.BasestateBegin
"Home"
AxeMainAPI.BasestateEnd
rc,"",""
Browser("Browser").Basestate(AXEDIR &
if rc <> 0 then
"samples\OdinPortal\html\index.html")
AxeMainAPI.BasestateEnd rc,"",""
AxeMainAPI.TestAbort
if rc <> 0 then
Application
Exit
Exit
rc=
rc=
AxeMainAPI.TestAbort
Reworking of Manual
Scripts
Test Automation Tool
System Under Test
28
Object
Maps
Project Phase 2: What was done
differently…
Taught an old dog some new tricks…
29
Test Design Model
A simple structured way of defining tests that is independent of the
interface and the means of execution
30
Features of the Test Design Model



Easy to design tests without technical knowledge
Well structured
Can be used to describe tests for any interface
–
–

Independent of Execution Mechanism
–
–


31
Graphical User Interfaces (GUI)
Non-UI Components (WebServices, APIs etc.)
Manual testing
Automated testing
Not theory - proven 7 year history of implementations
In use in UK and Europe at many organisations for defining
testing
Test Design Model
Sub-tests
Test steps
A single test step
Test
32
Object
Action
Data
Test Model – GUI Example
Step
Enter Username “jsmith”
Sub-test
Login as user 1
33
Test
Login – enter user
credentials
Product search – enter
product ID
Product details – validate
product info
Logout
A test step


Object + Action + Data
Action
–
–
–
34
SET
GET
VAL
enter data, navigation
retrieve data
compare expected/actual data
How tests are designed




Using Microsoft Excel
Business processes are modelled
Sequences of steps and data are built up in Sub-Test Tables
Sequences of Sub-Tests model business processes
[]Login
Username
Password
SignIn
action action data
action data
action
set
set
jsmith set
testing1 set
35
Interpreting the Model

There are products that can process the model
–
To generate Test Automation Scripts


–
36
For UI Testing using GUI Automation tools
For Non-UI Testing using Harnesses
To generate documentation for manual testing
Interpreting the Model
Manual Scripts
37
Non-UI Automated
Scripts
Automated
Scripts
Globus
A Self Descriptive Application
38
What is a Self-Descriptive
Application?
An application that, through an automated process, can provide a
description of its interfaces
Application
39
The Globus User Interface
How it works
Application
Globus
Desktop
Application
A description of
the interface for
an application is
stored in the
database
40
The Globus
Desktop interprets
the description and
dynamically
creates the user
interface
Windows and
objects are
presented to
the user
How Globus Describes its Interface
API
Globus
Desktop
Application
Custom
Interpreter
API can be invoked
to provide the
description without
presenting it
41
A Sample of Globus Information
APP: CUSTOMER_INST
42
FIELD
TYPE
LABEL
LENGTH
SEQ
CUST_SNAME
INPUT
Surname
20
1
FIELD
TYPE
LABEL
LENGTH
SEQ
CUST_GEN
OPTION
Gender
1
7
How this Information was used

The Description gives us 3 things:
–
–
–

This could provide us with two things automatically:
–
–
43
The business names used for the objects
The properties the test tool required for object identification
The sequence of the objects in the window
The Object Map
The sequence of steps for our TDM sub-tests
More on Self-Descriptive Apps

Requirement on development to “name” objects in
business terms

Microsoft .NET Applications
Windows Forms and Web Applications can provide
us with

–
–

44
Object Maps
Sub-tests for the Test Design Model
This can drastically reduce the cost of automation
Overall - The Test System Design
Microsoft Excel
Globus
Test Design
Model & Object Maps
Axe
Script 1.1
Script 1.1
Script 1.1
AxeMainAPI.TestBegin "6"
rc=AxeMainAPI.CheckDependency("5")
AxeMainAPI.TestBegin "6"
If rc <> 0 Then
rc=AxeMainAPI.CheckDependency("5")
AxeMainAPI.TestBegin "6"
If rc <> 0 Then
rc=AxeMainAPI.CheckDependency("5")
ExitRun(rc)
If rc <> 0 Then
End If
ExitRun(rc)
AxeMainAPI.BasestateBegin "Home"
End If
ExitRun(rc) &
Browser("Browser").Basestate(AXEDIR
AxeMainAPI.BasestateBegin
"Home"
End If
"samples\OdinPortal\html\index.html")
Browser("Browser").Basestate(AXEDIR
& "Home"
AxeMainAPI.BasestateBegin
AxeMainAPI.BasestateEnd
rc,"",""
"samples\OdinPortal\html\index.html")
Browser("Browser").Basestate(AXEDIR &
if rc <> 0 then
AxeMainAPI.BasestateEnd
rc,"",""
"samples\OdinPortal\html\index.html")
if rc <> 0 thenAxeMainAPI.BasestateEnd rc,"",""
rc=AxeMainAPI.TestAbort
if rc <> 0 then
rc=AxeMainAPI.TestAbort
45
rc=AxeMainAPI.TestAbort
Automation
Scripts
Project Phase 2:
Implementation
46
The Implementation

Technical Implementation
–

47
9 days of Test Tool Technician
Using Automatic Interface Description in Globus
– 1800+ Objects Mapped automatically
– 90 Sub-test components generated
 30-75 steps per Sub-test
– Time to Generate:
42 seconds
What was left for the Testers?

Test Design!
–
–
–
48
Designing scenarios for testing the application
Providing Test Data
Sequencing Sub-Tests to form business processes
Designing the Tests

4 Testers – Using the Test Design Model
No previous automation experience

40 Days Elapsed time

1012 Complex Banking Scenarios
Automated

49
Raw Statistics – Test Execution

Single Script
–
–
–

Automated
= 80secs
Manual
= 480secs
Automated is 6x Faster
Full Regression Pack
–
Automated
–
–
Manual
–
= 22 Hrs (1 Workstation)
= 5.5 Hrs (4 Workstations)
= 132 Hrs
= 22 Man/Days (using 6hrs productivity per
day)
50

22 Man/Days Testing in 5.5 Hrs.
Impact
Upon You
51
Recommendations


Find out your organisations direction
Decide upon your own future
1.
2.
3.

52
Increased management skills
Increased non-functional skills
Increased business skills
Ensure you understand how Automation
Frameworks support testing so you have a
view when asked, because you will be….!
Thank You
Susan Windsor
WMHL Consulting
Limited, MD
[email protected]
www.wmhl.co.uk
53
Case Study provided by Odin Technology Ltd
www.odin.co.uk