Transcript Tool Categories

Tips for Success when
Selecting Testing Tools
Marie Ackley
ackley @ promail.com
SASQAG panel discussion
October 18, 2001
Different Categories of
Testing Tools







Requirements Management
Configuration Management
Test Planning and Test Case Management
Test Evaluation - Coverage Analysis
Automated Testing Systems - Front End testing
Load Simulators and Performance Measurement Tools
Defect Tracking
others: Cyclomatic Complexity; Bounds Checkers;
Comparators - Files, Screens, Reports; Data Generators
- Combinatoric, Cause / Effect Node Graphing
2
Leading Cause of
SHELF WARE in Testing Tools
is Putting the CART before the HORSE
Don't invest in Tools without having good
QA Methods and Practices.
A good tool will compliment and augment
good processes but not even the greatest
tool can overcome process chaos.
3
Requirements Management

Supports multi levels of linking and tracing: Requirements to
Specifications to Test Cases, including hierarchies within those
groups

Provides traceability - both forward and backward

Provides coverage analysis

Provides multi relationships: one-to-many, many-to-one and
coverage is tracked for partials when relationship is one-to-many

Provides Change Management of requirements along with audit
history and maybe security privileges for modifications

Import and export features so stuff came go and come between the
tool and word processors or spreadsheets
4
Requirements Management
Links to tool comparisons
the "Volere" Requirements Method - by Atlantic Systems Guild
http://www.systemsguild.com/GuildSite/Robs/Template.html
Automating Requirements Management - by Karl E. Wiegers
Detailed feature comparisons of many requirements management tools
and guidance on selecting a requirements management tool
http://www.processimpact.com/articles/rm_tools.html
5
Configuration Management
"To change and to change for the better are two different things"
- German Proverb

Provides Identifications (indexing of artifacts)

Controls not only software code but documentation, hardware,
firmware, and system environment configuration

Provides audit trail and history tracking

Provides customizable reports

High ease of use for View Only functions

Sufficient security levels for the Promote functions

Accounting and reporting for different ”Status Levels" - WIP,
Waiting Room, Release/Production
6
Configuration Management
Links to tool comparisons
Configuration Management Vendor Links - by UCM Central
http://www.snuffybear.com/ucmcentral_new_vendorlinks.htm
config-mgmt Newsgroup
http://www.faqs.org/faqs/by-newsgroup/comp/comp.software.config-mgmt.html
7
Test Planning and
Test Case Management

Import test cases from other file formats such as word processing
or spreadsheet

Customizable reporting

Tracks history and status of test cases

Allows file attachments



Integration - can tool link to other tools such as your Requirements
tool and your Automated Testing tool
Provide a Test Execution Results repository
Maintains Master Set of test cases and then can build “subsets” for
project specific needs
8
Test Planning and
Test Case Management
Links for tool comparisons
Testing and Test Management Tools
http://www.methods-tools.com/tools/frames_testing.html
Information About Testing Tools http://www.cs.queensu.ca/home/shepard/testing.dir/under.construction/tool_list.html
9
Test Evaluation
- Coverage Analysis

Identifies what "unique" data combinations are needed

Uses an appropriate measurement technique
Decision; Condition/decision; Path; Predicate

Types of coverage included in the reporting
Branch; Loop; Multi; Operator; Operand

Integration - is it compatible with your other tools such as
Automated Test Execution tools

Implementation - is the tool "invasive" (must be compiled along
with the source code)
10
Test Evaluation
- Coverage Analysis
Links to tool comparisons
Experience With the Cost of Different Coverage Goals for Testing - by
Brian Marick
http://www.testing.com/writings/experience.pdf
Coverage Analysis - by Steve Cornett
http://www.bullseye.com/webCoverage.html
11
Automated Testing Systems Front End testing

Maintenance - object recognition when the application changes

Ease Of Use - complexity matches the programming talent of QA
persons

Tool language and syntax matches what company uses: Visual Basic
or C

Extensible Language - can use external subroutines or DLLs

Integration with other tools - Code Coverage, Test Case
Management, Defect Tracking

Compatibilty - platforms (Client/Server, Web, Mainframe) Operating
Systems(Windows, Mac, Unix, OS/2, Mainframe) and GUI
Development (PowerBuilder, C, Visual Basic, Galaxy
12
Automated Testing Systems Front End testing

Supports Data Driven inputs from flat files, spreadsheet files, and
database connections

For Evaluation/Proof of Concept - include automating a Level One test
Links to tool comparisons
Making the Right Choice - by Elisabeth Hendrickson
http://www.qualitytree.com/feature/mtrc.pdf
Short Sharp Advice about how to choose a testing tool - Mark Fewster
and Dorothy Graham
http://www.grove.co.uk/Tool_Information/Choosing_Tools.html
Tool Comparison between Rational, Mercury, Segue, and Compuware - by
Ray Robinson
http://www.qacity.com/testtype/AUTOMATION_TOOLS.doc 13
Automated Testing Systems
Links to articles
Lessons in Test Automation - by Elfriede Dustin
http://www.sqa-test.com/articles.html
The Truth About Automated Test Tools - by Linda Hayes
http://www.worksoft.com/resources/4-97.htm
http://www.worksoft.com/resources/5-97.htm
Test Automation - by Bret Pettichord
http://www.io.com/~wazmo/qa/#test_automation
Testers' Network - VeriTest website
http://www.veritest.com/testers'network
Test Automation Frameworks - by Carl J. Nagle
http://members.aol.com/sascanagl/FRAMESDataDrivenTestAutomationFrameworks.htm
14
Load Simulators and
Performance Measurement

Reporting includes graphical charts

Debugging features for script creation

Reporting includes graphical charts and logs of performance for
server speed and responsiveness

Quantity of simulated users

Variations in session-specific items such as cookies, usernames,
passwords, proxies and ASP-session IDs

Support for your middleware - SQLnet, DBLib or CBLib, SQL Server,
ODBC, Telnet

Challenges include SSL sessions, cookies, forms, proxies, password
authentication, dynamic HTML, full HTTP, SSL, XML, Javascript,
ActiveX, WAP, Java

Platforms and protocols
15
Load Simulators and
Performance Measurement
Link to tool comparisons
Load and Performance Test Tools - by Rick Hower
http://www.softwareqatest.com/qatweb1.html
Software Research - TestWorks
http://www.methods-tools.com/tools/testing.html
16
Defect Tracking

Determine how many current users you have and consider and
scaling for future users

Web based or client-server

Features: Customizable Fields; Customizable Reports; File
Attachments; Audit Trail; Email linking; Security Levels

Provides customizable rules for the workflows, security privileges
and notifications

Integration - does it link to your other tools
17
Defect Tracking
Links to tool comparisons
Tracking Down a Defect Management Tool - by Hung Quoc Nguyen
http://www.qacity.com/testgen/DefectTrackingNguyenSTQE701.pdf
The Problem with Problem Tracking - by Linda Hayes
http://www.worksoft.com/resources/3-99.htm
18
Websites for Tool Information
Vendor contact information
http://www.testingfaqs.org/tools.htm
http://www.ondaweb.com/sti/stivend.htm#L
http://www.dbmsmag.com/pctest.html#P0890
Bulletin board forums and articles
http://www.qaforums.com
http://community.sqatester.com
http://www.qualitytree.com/links/links.htm
http://www.stqemagazine.com
19