Transcript Secrets of Successful Automation Projects (Draft)

Chicago Quality Assurance Assocation
Secrets of Successful
Automation Projects
Presented by
Paul M. Grossman
Software Quality Assurance Analyst
QTP Automation Framework Designer



Portfolio of Custom Software

Software Development: .Net, Java, C#

eCommerce (Oracle ATG)

Mobile Applications: iOS, Android, Windows Mobile/WP7, Blackberry
Quality Assurance Testing

LoadRunner, QTP, HP ALMS

Custom Tool Development and Frameworks

Performance Testing

Mobile Testing: iOS, Android, Windows Mobile, Tablets
Developed Applications in the following sectors

Games & Entertainment

Healthcare

Social Networking
4tegroup.com
Paul M. Grossman
 11 years automation experience with HP Tools
 HP Quality Center 9.0 Certified Product Consultant
 HP Quick Test Professional 9.0 Certified Product
Specialist
 Beta Tester of QTP 9.5 – 11.5
 Speaker at Mercury/HP Events
 Became Automation Engineer due to Perfect Timing!
Session Goals




Key success factors in test automation today
Demonstrating test automation ROI
Effective framework design techniques
Successful script design for Agile
environments
Automation Challenges

Lack of communication of changes/details to testing group

Lack of understanding of should be automated

Lack of variation from plan in test scripts

Insufficient standards

Insufficient Analysis & Design

“Silver bullet syndrome” (automated testing is the “silver bullet”
that will end all testing problems on a project)

Lack of change control
Case Study
Automation Project Failure at a Medical Device Manufacturer
Year 1: QA Manager wants automated testing.
Buys WinRunner, accepts offer at new company.
Year 2: New QA Manager wants automated testing.
Hired Lead Developer (C++), given an Assistant (VBscript).
Goal for automation: 100% coverage.
Year 3: Lead Developer moves to new department.
Assistant told automation is high-priority, spare-time project.
Year 4: QA Manager moves to new division
Assistant retires.
Results: 0 Defects found. Negative Return On Investment.
Unsuspecting manual tester assigned to automation team.
New Division Manager expresses interest in
automation.
Top 5 Project Pitfalls
1. Lack of research &
planning
2. Lack of apparent
progress
3. Lack of experienced
resources
4. Not a full time
project
5. No buy-in
 100% automation goal resulted
in development in all product
areas
 20% development in all areas
with nothing to show
 Project handed over to
assistant with no product
training
 Misperception: FT not needed
because automation by it’s
definition is automatic, right?
 Testers & developers fueled
rumors of tool’s stability
Mitigation Strategies
1. Lack of research &
planning
2. Lack of apparent
progress
3. Lack of experienced
resources
4. Not a full time
project
5. No buy-in
 10% Automation goal
 POC Pilot Project
 Training
 Full-time commitment
 Repeat defects manually
on a non-automated
system
Tips And Tricks
Avoiding "100% Automation" Trap
 90% Automation can be
approached but it is generally
not cost effective
 New releases will have more
functionality with constantly
decreasing code coverage
 A percentage of the code could
be unreachable code, such as
uncalled functions
 Pick one: Functionality or Browsers
Do the math: You won't have time
for everything!
20 - 40% Automation makes a significant
impact on the testing process.
Case Study
The Pilot Project – Lessons Learned
 Automate the Acceptance Test (“Smoke Tests”)
— Short tests, but too wide of coverage
 Commitment to six weeks of full-time development
— Became four weeks, 10 days of manual testing
 Used “Record & Playback” to create the entire test script
— Overwhelming maintenance
 Demonstration to follow at the end of the project
— Management convinced they purchased the right tool
— Asked for training on Winrunner
Request to get training approved!
Case Study
Functional Decomposition

Crafted functions from Demo Script to create “TimeSaver” utilities

Wrote discrete state change functions: TRUE, Worked! FALSE, Uh-Oh!

Modular functions made maintenance manageable

Automated the common interfaces of two additional products

Proposed automating easy features first
– ShowStopper #1: Memory Leak. First 13-hours of endurance test.
– Developers stop blaming tool, now ask for ad-hoc scripts
Request to attend Mercury World Conference Approved!
Estimated Return On Investment
“Best Bang For The Buck”: Project TimeSavers
 Automate tedious manual tasks
— Create 64 Annotations - 30 Seconds
— Perform 128 End to End Procedures - 30 Minute Drive Space Test
— Register 128 New Patients - 30 Minute Boundary Test
 These tests were small, little maintenance
 Gets testers on your side, reduces their “pain”
 Scripts were easy to change into reusable functions
First Quantifiable Return On Investment: $40 (1 Hour)
Final estimated ROI for the company:
$2 Million
Estimated Return On Investment
Estimated Return On Investment
ROI Formula: Estimate Equivalent Cost of Automation
How much would it cost to hire someone to do this work manually?
Estimated hourly wage (Plus OT & Benefits): $40 Per Hour Average
Determine hours of runtime
10 x 5 = 50 Hours per system
50 x 3 = 150 Hours per week
- 10 hour runs, 5 nights a week
- on 3 systems
-----------------------
150 x $40 = $6,000
Extrapolated to the end of the year
X 52 Weeks
-----------------------
Four years of run time:
$312,000
$1,248,000
Estimated Return On Investment
What would some of the worst defects found by automation have
cost the company had they gotten to the field?
 Case Study Examples
— ShowStopper #2: Transient Boot Up Failure
– Replicated with automation and high end UPS
— ShowStopper #3: Data Transfer Problem
– Duplicated issue within five days.
– Reason: Related to Network Load limitation.
 We pegged the ROI of the worst detected defects by estimating the
cost of a single lost customer sale: $100,000
 Potential revenue loss from 7 ShowStoppers: $700,000
 Plus 1.2 Million not paid to a manual tester: 1.9 Million Est. ROI
Estimated Return On Investment
That was then…this is now
What would some of the worst defects found by automation have
cost your company had they gotten to the field?
— ShowStopper #1: Credit cards fail to authenticate?
– Store is closed world wide.
— ShowStopper #2: Search path leads to infinite loop?
– Customers frustrated with site jump to competitor.
— ShowStopper #3: 12-second+ response time?
– Customers abandon order
– Is customer support being effected?
– Slows down automation code development and debugging
– Slows down development programmers
– Slows down manual testers
Let's take a break!
When we come back…
 Tips And Tricks
 Dealing with Failure
 What is a Framework anyway?
What is a Framework?
Framework Options
Capture/replay
Exhaustive
Function-based
Random
Data-driven
Lexical Analyzer
Keyword-driven
Hybrid
What is a Framework?

Driver
 Loads & Executes Multiple Tests
 Facilitates Result Reporting

Object Repository or Identification
 Identifies objects by Class and Property Values
 Uses Regular Expressions for dynamic values

Data
 Pre-Defined
 Generated

Recovery Mechanism
 Browser closes
 Object invalid
 Error generated
A framework is a set of techniques:
•
Reduce Maintenance
•
Abstract Complexity
•
Extend Coverage
Tips And Tricks
Initial Project Design

Return To Home State Functionality
— Get back to the starting point from anywhere in the application, even if it is not running.

Navigation and Population
— Low level custom actions to Click buttons, Open menus, Select item from lists.

getValidObject (sClass, sName)
— A single function that returns a valid object reference from many close matches.
— Object exists (height, width & Abs_x > 0), is enabled, above underlay, closest to text.
— Uses Descriptive Programming or xPath to identify object

Tokens
— Stores reusable values, such as an Order Number
— Can return intrinsic values like Today's Date.

Assume Failure, Prove Success
— Use a global variable & Return Codes for overall success. This creates a single exit point
for script execution.

Modify, Recover, Repeat
Tips And Tricks
Modular Design

Functions… but No Subs
— Don’t waste time deciding which to use
— Subroutines have no built in functionality for return codes
— Sub FileExists (strFile, rc) ‘ Return Code Workaround

All Functions need Return Codes
— FileExists = TRUE
— IF FileExists(strFile) THEN ProcessFile (strFile)

Enter_CC "Declined"
Use Option Explicit in VBscript
— Reduces errors due to variable misspellings

Three Level Keyword Framework
— Components (Param1, Param2)
o Keywords (Param1, Param2)
All other supporting functions
CLICK "Ok"
getValidObject
Tips And Tricks
Global Variables and Dictionaries
 Minimize global variables:
— gloObject: Reference set by GetValidObject
 Use a Dictionary Object
— Global
— No Naming Convention
— Stores Object References
— Can be Persistent!
— Case Sensitive Key Names
dicFramework
Key
Vaue
"AlreadyFailed"
False
"NextMonth"
"June"
"ObjectClass"
"WebButton"
"gloObject"
[OK Button]
Tips And Tricks
Dealing with Failure
dicFramework("AlreadyFailed") = FALSE
Function CLICK (sButtonName)
If dicFramework("AlreadyFailed") = TRUE Then
Exit Function
End If
blnFound = getValidObject "WebButton", sButtonName
If blnFound = TRUE then
gloObject.Click
Else
dicFramework("AlreadyFailed") = TRUE
End If
End Function
'Sets gloObject
Tips And Tricks
Dealing with Failure
dicFramework("AlreadyFailed") = FALSE
Function CLICK (sButtonName)
…
If blnFound = TRUE then
On Error Resume Next
gloObject.Click
reportErr ("Click")
On Error Goto 0
Function reportErr (sFunc)
If err.Number <> 0 Then
Else
dicFramework("AlreadyFailed") = TRUE
End If
…
End Function
Report err.Description & " in " & sFunc
End If
End Function
Thinking Outside The Box
getValidObject (sClass, sText)
 getValidObject ("Link", "Login")
'Sets gloObject
 Three Attempts
— Descriptive Programming (Exact Match)
– oBrowser.oPage.Link("innerhtml:=(.*)?Login(.*)?")
– False? Add "Index:=0" - True if there multiple objects
— Search Object Collection (First Valid Match)
– oDesc("class") = "Link" : oDesc("Name") = ".*Login.*"
– Set oLinksCollecton = oBrowser.oPage.ChildObjects(oDesc)
— Switch Class
– oBrowser.oPage.WebButton("innertext:=Login")
– oBrowser.oPage.WebElement("innertext:=Login")
Thinking Outside The Box
getValidObject (sClass, sText)
 getValidObject ("Link", "Login") 'Sets gloObject
 Validation – Is this object Visible?
– Width > 0
– Height > 0
– Abs_x > 0
– Source_index: Appears over an underling object
 This is not the List object you are looking for
— List contains the string we want to select
Agile environments
 Version Control
— Not vulnerable to local system crash
— Code Change Comparison
— Merge Changes
— Revert Changes
Agile environments
 Continuous Integration with Jenkins
— Run scripts when changes are made or at Night
— Report overall Pass / Fail
— Replaces Driver
— Customizable
Secrets of Successful Automation Projects
 Successful Projects Require
— Modular Design, Demonstration, Low Maintenance
— Plan for Common Pitfalls
— Leverages Agile Processes
 Automated tests
— Keywords & KW-Based Components
— Supporting functions
—
Dynamic Reusable data
—
Object identification
—
Recovery & Cleanup
 Potential loss can be quantified as ROI
 Training Increases ROI
 Always think outside the box!
Resources




Training Classes
HP Certification classes
HP Discover
LinkedIn Users Group Meetings
On the Web





cqaa.org
stickyminds.com
sqaforums.com
advancedqtp.com
wilsonmar.com
Thank you!
Learn more at CQAA.org
Questions?
[email protected]
Additional Thanks to
Lee Barnes
Utopia Solutions
www.UtopiaSolutions.com