Transcript No Slide Title

Practical
Web/E-Commerce Testing
Part 1 – Functional Testing
Robert Sabourin
President
AmiBug.Com, Inc.
Montreal, Canada
[email protected]
July 20, 2015
© Robert Sabourin, 2001
Slide 1
AmiBug.Com, Inc.
Practical
Web/E-Commerce Testing
Part 1 – Functional Testing
Overview:
–
–
–
–
Quality Attributes & What can break?
Web Tier
Application Tier
Data Tier
July 20, 2015
© Robert Sabourin, 2001
Slide 2
AmiBug.Com, Inc.
Practical
Web/E-Commerce Testing
Part 1 – Functional Testing
• Robert Sabourin ,
Software Evangelist
• President
• AmiBug.Com Inc.
• Montreal, Quebec,
Canada
• [email protected]
July 20, 2015
© Robert Sabourin, 2001
Slide 3
AmiBug.Com, Inc.
AmiBug.Com, Inc.
• Software Development & SQA Consulting
• Services
– Training, Coaching and Professional
Development
– Light Effective Process
– Team Building and Organization
– We help people to get things done!
July 20, 2015
© Robert Sabourin, 2001
Slide 4
AmiBug.Com, Inc.
I am a Bug
Robert & Catherine Sabourin
ISBN: 0-9685774-0-7
www.amazon.com
In the style of a children's book.
Explains elements of software
development process in a fun easy
to read format.
July 20, 2015
© Robert Sabourin, 2001
Slide 5
AmiBug.Com, Inc.
Fundamental Question
• How do you know when you are finished?
July 20, 2015
© Robert Sabourin, 2001
Slide 6
AmiBug.Com, Inc.
Crosby on Quality
• “Quality is defined as conformance to
requirements”
• “Quality is not a measure of GOODNESS”
– Phil B. Crosby, Quality is Free
July 20, 2015
© Robert Sabourin, 2001
Slide 7
AmiBug.Com, Inc.
Deming Quality approach
(PDCA)
• Plan, Do, Check, and Act:
Plan what you want to implement.
Do the pilot implementation.
Check the results of the pilot.
Act on the results by tweaking the process before
the next project.
July 20, 2015
© Robert Sabourin, 2001
Slide 8
AmiBug.Com, Inc.
Edsger W. Dijkstra
• “Program testing can be used to show the
presence of bugs, but never to show their
absence”
July 20, 2015
© Robert Sabourin, 2001
Slide 9
AmiBug.Com, Inc.
"No! Try not, Do. Or do not. There is no
try."
July 20, 2015
© Robert Sabourin, 2001
Slide 10
AmiBug.Com, Inc.
Definition of a Bug
• To make our job more fun, whenever we
have a concern with software, we call it a
“bug”.
July 20, 2015
© Robert Sabourin, 2001
Slide 11
AmiBug.Com, Inc.
Testing Web and E-Commerce
Applications
• It’s all about people! (and the occasional
bug too)
July 20, 2015
© Robert Sabourin, 2001
Slide 12
AmiBug.Com, Inc.
Purpose
• What is the purpose of testing?
• Common definition of the purpose of testing
– Our purpose is to find bugs before our customers do!
• Broader definition
– The role of testing is to provide objective input to
facilitate business decisions (wise smart and good
decisions)
July 20, 2015
© Robert Sabourin, 2001
Slide 13
AmiBug.Com, Inc.
About Bugs
Bugs are not Good or Bad
July 20, 2015
© Robert Sabourin, 2001
Slide 14
AmiBug.Com, Inc.
About Bugs
Some bugs are important
and have a high priority!
July 20, 2015
© Robert Sabourin, 2001
Slide 15
AmiBug.Com, Inc.
About Bugs
Some bugs are dangerous
and have a high severity!
July 20, 2015
© Robert Sabourin, 2001
Slide 16
AmiBug.Com, Inc.
About Bugs
• Setting the priority and severity of a bug is a
business decision
July 20, 2015
© Robert Sabourin, 2001
Slide 17
AmiBug.Com, Inc.
About Bugs
• Changing business conditions impact the
priority and severity of a bug!
– Always review previous decisions in light of
changing business context
– Ensure staff assigning priority and severity are
aware of all relevant business drivers
July 20, 2015
© Robert Sabourin, 2001
Slide 18
AmiBug.Com, Inc.
Bug Quadrants
July 20, 2015
Urgent
Severe
Urgent
Not Severe
Not Urgent
Severe
Not Urgent
Not Severe
© Robert Sabourin, 2001
Slide 19
AmiBug.Com, Inc.
Business Decisions
• SQA Role:
– Objective input
• Development Role:
– Technical implementation/Risk
• Project/Product Management Role:
– Customer driven requirements/Impact
July 20, 2015
© Robert Sabourin, 2001
Slide 20
AmiBug.Com, Inc.
Quadrant Changing
• Same technical bug can be in a different
quadrant depending on the business context
• Monitor business drivers!
• Focus find and fix quadrant -1- bugs high
priority/high severity
July 20, 2015
© Robert Sabourin, 2001
Slide 21
AmiBug.Com, Inc.
Finished?
• How do you know you are finished?
July 20, 2015
© Robert Sabourin, 2001
Slide 22
AmiBug.Com, Inc.
You know you are
finished when …
• … the only bugs left are the ones that
Product Management and Development
agree are acceptable (based on objective
SQA input) ...
July 20, 2015
© Robert Sabourin, 2001
Slide 23
AmiBug.Com, Inc.
You know you are
finished when …
• … the only bugs left are the ones that
Product Management and Development
agree are acceptable (based on objective
SQA input) …
At least for now!
July 20, 2015
© Robert Sabourin, 2001
Slide 24
AmiBug.Com, Inc.
Just for Fun
July 20, 2015
© Robert Sabourin, 2001
Slide 25
AmiBug.Com, Inc.
Testing Web and E-Commerce
Applications
Quality Attributes
July 20, 2015
© Robert Sabourin, 2001
Slide 26
AmiBug.Com, Inc.
Quality Factors
• Definition
– Attributes of software
– If not implemented they pose a risk to the
success of the software
• business risk
– Factors that need to be considered when
deciding your testing strategy
July 20, 2015
© Robert Sabourin, 2001
Slide 27
AmiBug.Com, Inc.
Quality Factors
• Why?
– Purpose
• The purpose of testing can be viewed as being
activities done to identify issues of concern to the
commercial use or deployment of software
– Focus
• Quality factors help us to focus testing on the most
important characteristics of the software
July 20, 2015
© Robert Sabourin, 2001
Slide 28
AmiBug.Com, Inc.
Factor
Perceived Software Quality
Adaptability
Can functions be enhanced without major redesign?
Accessibility
Can all users access your application?
Auditibility
Can a log be presented showing all access&transactions?
Availability
Can functions be accessed when needed?
Continuity
Can system continue operation recovering from a failure?
Dependability
Can users trust results provided?
Expandability
Can functions be added without major redesign?
Functionality
Does software perform as advertised?
Integrity
Is all data stored and returned unaltered.
Interoperability
Can be used with other software?
July 20, 2015
© Robert Sabourin, 2001
Slide 29
AmiBug.Com, Inc.
Factor
Perceived Software Quality
Maintainability
Can software be updated to new release levels?
Operability
Can software be easily put into operation?
Portability
Can software be run in different environments?
Reliability
How long does the software operate without failure?
Re-usability
Can elements of software be reused in future?
Scalability
Can transaction volume increase with more resources?
Security
Can unauthorized users access systems and data?
Serviceability
Can bug fixes be integrated with minimum downtime?
Testability
Can all functions and features be tested?
Usability
Can the software be easily used by target user?
July 20, 2015
© Robert Sabourin, 2001
Slide 30
AmiBug.Com, Inc.
Ranking Quality Factors
• It is important to determine the relative
importance of each Quality Factor for your
project!
–
–
–
–
Testing resources are always severely limited
Focus on things important for our business
Use a systematic approach
Ranking will be used to ensure testing effort is
reasonably spread across Quality Factors
July 20, 2015
© Robert Sabourin, 2001
Slide 31
AmiBug.Com, Inc.
Ranking Quality Factors
Technical Risk
(1-5)
Business Risk
(1-5)
Exposure
T*B
Rank
Adaptability
Accessibility
Auditability
Availability
Continuity
Dependability
Expandability
Functionality
Integrity
Interoperability
Maintainability
Operability
Portability
Reliability
Re-usability
Scalability
Security
Serviceability
Testability
Usability
July 20, 2015
© Robert Sabourin, 2001
Slide 32
AmiBug.Com, Inc.
Example Ranking
Technical Risk
(1-5)
Adaptability
Accessibility
Auditability
Availability
Continuity
Dependability
Expandability
Functionality
Integrity
Interoperability
Maintainability
Operability
Portability
Reliability
Re-usability
Scalability
Security
Serviceability
Testability
Usability
July 20, 2015
Business Risk
(1-5)
2
1
1
3
5
3
1
4
3
1
1
1
4
3
1
3
5
3
3
5
Exposure
T*B
5
5
1
3
4
5
3
4
4
1
3
1
5
4
1
4
5
3
3
5
© Robert Sabourin, 2001
Rank
10
5
1
9
20
15
3
16
12
1
3
1
20
12
1
12
25
9
9
25
MEDIUM
MEDIUM
LOW
MEDIUM
HIGH
MEDIUM
MEDIUM
HIGH
MEDIUM
LOW
MEDIUM
LOW
HIGH
MEDIUM
LOW
MEDIUM
HIGH
MEDIUM
MEDIUM
HIGH
Slide 33
AmiBug.Com, Inc.
Testing Web and E-Commerce
Applications
What Can Break?
July 20, 2015
© Robert Sabourin, 2001
Slide 34
AmiBug.Com, Inc.
Overview
–
–
–
–
July 20, 2015
Types of Web Systems
Basic Web Architecture
Typical Transaction
Exercise
© Robert Sabourin, 2001
Slide 35
AmiBug.Com, Inc.
•
•
•
•
•
•
•
•
•
•
•
•
July 20, 2015
Informational
Delivery
Customized access
User-provided content
Interactive
File sharing
Transaction oriented
Application service provider
Database access
Document access
Workflow oriented
Automatic content generator
© Robert Sabourin, 2001
Any Combinations
Types of Web Systems
Slide 36
AmiBug.Com, Inc.
Adaptability
Accessibility
Auditability
Availability
Continuity
Dependability
Expandability
Functionality
Integrity
Interoperability
Maintainability
Operability
Portability
Reliability
Re-usability
Scalability
Security
Serviceability
Testability
Usability
Application service provider
Automatic content generator
Customized access
Database access
Delivery
Document access
File sharing
Informational
Interactive
Transaction oriented
User-provided content
Workflow oriented
High Focus
Medium Focus
Low Focus
July 20, 2015
Quality Factors Importance For
Different Web Application Types
© Robert Sabourin, 2001
Slide 37
AmiBug.Com, Inc.
Basic Web Architecture
Workstation
Application
Components
Workstation
Data
Services
Data
Server
Proxy Server
Workstation
Workstation
Process
HTTP
Requests
Application
Dependent
Code
Internet
Modem
Home PC
Internet Service Provider
July 20, 2015
Web Server
© Robert Sabourin, 2001
Application Server
Slide 38
AmiBug.Com, Inc.
Workstation Component
Workstation
Workstation
Workstation
Workstation
Modem
Home PC
WORKSTATIONS
Browser
Operating System
Java Virtual Machine
Proxy Server
User’s
Locale
Plug-Ins
Media Players
Viewers
Internet
Display depth
Display resolution
Resident applications
Internet Service Provider
July 20, 2015
Data
Server
Web Server
© Robert Sabourin, 2001
Application Server
Slide 39
AmiBug.Com, Inc.
Workstation Processing
Workstation
Workstation
Workstation
Workstation
Modem
Home PC
WORKSTATIONS
Render all pages based on input from Web Server.
Data
Server
Locally run objects such as FLASH animations
Execute active X components
Proxy Server
Execute JAVASCRIPT
Execute JAVA
Execute VB SCRIPT
Execute Viewers Internet
Initiate Email
Store local user files
Cookies, Cached objects
Internet Service Provider
July 20, 2015
Web Server
© Robert Sabourin, 2001
Application Server
Slide 40
AmiBug.Com, Inc.
Internet Connections
Internet
Workstation
Workstation
Proxy Server
Workstation
Workstation
-
Network
Servers
Switches
Networks
Routers
Data
Server
Internet
Modem
Home PC
Internet Service Provider
July 20, 2015
Web Server
© Robert Sabourin, 2001
Application Server
Slide 41
AmiBug.Com, Inc.
Internet Connections
Web Server
- All HTTP Requests are handled by a Web Server
Workstation
- To
serviceWorkstation
the request:
- Find page requested and send it to requestor
- or Generate another request to another server
- or Execute a local script
or program
Proxy Server
Workstation
Workstation
requestor
- or Execute a local active server page script
which generates a page to send to the requestor
- or Execute
a local CGI script and send
result to
Internet
Modem
requestor
Home PC
- or Return an error message indicating that the
request could notInternet
beService
filled
presently
Provider
July 20, 2015
© Robert Sabourin, 2001
Data
Server
Web Server
Application Server
Slide 42
AmiBug.Com, Inc.
Internet Connections
Application Server
Workstation
Workstation
Workstation
- Web Application Business Logic is executed on
the application server
- Services and objects processed are totally
Proxy Server
Workstation
dependent on the Web Application Purpose
- Sometimes developed using third party application
infrastructures such as IBM WebSphere
Data
Server
Internet
Modem
Home PC
Internet Service Provider
July 20, 2015
Web Server
© Robert Sabourin, 2001
Application Server
Slide 43
AmiBug.Com, Inc.
Internet Connections
Workstation
Workstation
Data Server
Proxy Server
Data
layer
Workstation
May contain application specific data
May contain security data
Implemented with one or more physical servers
Internet
Modem
May
include several Databases
Communication to Application Server and Web
Server via standard interfaces (SQL)
Service Provider
- May includeInternet
stored
procedures and other data Web Server
handling related application code
July 20, 2015
© Robert Sabourin, 2001
- May be shared with other applications
Data
Server
Home PC
Workstation
Application Server
Slide 44
AmiBug.Com, Inc.
Firewalls
- Software or hardware combined
- Restricts traffic
- Source
- Destination
- Type
of data
Workstation
Workstation
- Authentication
- Authorization
Firewalls
Data
Server
Proxy Server
Workstation
Workstation
Internet
Modem
Home PC
Internet Service Provider
July 20, 2015
Web Server
© Robert Sabourin, 2001
Application Server
Slide 45
AmiBug.Com, Inc.
What can break?
Time
Attention span on Web is limited!
If the time to get a response from
any Web or E-Commerce
application is longer than 8-12
seconds most users will surf to
some another site!
July 20, 2015
© Robert Sabourin, 2001
Slide 46
AmiBug.Com, Inc.
What can break?
Time
Quick fixes and bandage
solutions often lead to
performance degradation!
Operation with full databases,
nearly 100% consumed system
resources.
Data
Server
Stuff outside and within your
control impacts response time!
July 20, 2015
© Robert Sabourin, 2001
Slide 47
AmiBug.Com, Inc.
Areas of control
Workstation
Workstation
Proxy Server
Workstation
Workstation
Outside of
Web
Application
Developers
Direct
Control
Within
Applications
Developers
Direct
Control
Data
Server
Internet
Modem
Home PC
Internet Service Provider
July 20, 2015
Web Server
© Robert Sabourin, 2001
Application Server
Slide 48
AmiBug.Com, Inc.
Workstation Bugs!
Displays and interactivity dependent
on Browser Type, Version & O/S
User
Workstation
Workstation
Missing or wrong version of Third
Party Plug-In or Object
Application
Data
Server
Incompatible or Missing JAVA
Runtime environment and engine
Proxy Server
Workstation
Workstation
Incompatible or Missing
JAVASCRIPT interpreter
Modem
Home PC
Incompatible or Missing Viewer
Internet
(PDF, 3rd party formats)
Locale not supported including
Language,
Fonts other resources
Internet Service Provider
July 20, 2015 Screen resolution ©
Robert
Sabourin,
2001
and
color
support
not match application
Web Server
Application Server
Slide 49
AmiBug.Com, Inc.
Web Server Bugs!
Defective HTLM constructs!
User interface design errors unusable!
Missing, incorrect or broken links
Runtime errors in Active Server
Page (ASP) scripts.
Application
User
Workstation
Missing, incorrect images,
Workstation
incorrectly identified objects
Resources restricted - not enough
memory, files system, capacity
Runtime errors in JAVASCRIPT or
VBSCRIPT code
Data
Server
Logic, design or programming logic
in code implementing user interface
Proxy Server
Workstation
Workstation
Missing or wrong version of web
pages!
Denial of service attacks!
Modem
Home PC
Too many users for capacity of
HTTP server.
Internet
Pirates, Vandals, Hackers,
Snoopers,Internet
Bad
Guys redirections
Service Provider
Web Server
Application Server
Too much traffic for capacity
ofSabourin,
Web 2001Incorrect exception handling
in code
July 20, 2015
© Robert
Slide 50
server.
implementing user interface.
AmiBug.Com, Inc.
Web Server Bugs too!
HTTP Server Failure
Error handling errors in
communicating with other servers
Java Server Failure
Connection time out due to
communication delays
Application
User
Workstation
O/S Failure - O/S Related problem
Workstation
moving between Unix and Windows
Logic error in detecting and
processing time out condition
Data
Server
Third Party Component Failure
Scripting error in CGI script
Proxy Server
Workstation
Workstation
Incorrect processing of cookie data
in determining session status.
Server Hardware failure
Memory Leaks - resource
Modem
management
Home PC
Internet
Access failure communicating with
Application
Server
Internet Service Provider
Web Server
Application Server
Access failure communicating
July 20, 2015
© Robert with
Sabourin, 2001Logic error in implementation
Slide 51of CGI
Date Server.
scripts.
AmiBug.Com, Inc.
Application
Server Bugs!
Missing, required functionality
Application
User
Incorrectly implemented business
Workstation
Workstation
logic
Data
Server
Runtime errors in scripted code
Proxy Server
Workstation
Workstation
Incorrect version of code or objects
Modem
Server, O/S or
third party software
Internet
failure - middle-ware failure
Home PC
Server hardware failure
Internet Service Provider
July 20, 2015
Web Server
© Robert
Sabourin,
2001
Cannot
handle
transaction
volume
Application Server
Slide 52
AmiBug.Com, Inc.
Application Server Bugs too!
Communication failure with Data
server
Application
User
Communication failure with Web
Workstation
Workstation
Server
Logic error in implementation of
error handling code
Data
Server
Proxy Server
Workstation
Workstation
Memory leaks - resource usage
Modem
Internet
Failure of third
part components
Home PC
B2B link to other site failed
Internet Service Provider
July 20, 2015
Web Server
Design
errors
in implementation
of
© Robert
Sabourin,
2001
business logic
Application Server
Slide 53
AmiBug.Com, Inc.
Data Server Bugs!
Performance delays due to amount
of data or system load
Communication failure with
Application server
Application
User
Workstation
Workstation
Communication failure with Web
Server
Data
Server
Logic error in implementation of
error handling code
Proxy Server
Workstation
Workstation
Memory leaks - resource usage
Modem
Home PC
Runtime errors executing stored
Internet
procedures
Incorrect schema referenced by
different
servers - configurations
Internet Service Provider
July 20, 2015
Design errors in implementation
of
© Robert Sabourin, 2001
database logic
Web Server
Application Server
Slide 54
AmiBug.Com, Inc.
Testing Web and E-Commerce
Applications
Web Tier
July 20, 2015
© Robert Sabourin, 2001
Slide 55
AmiBug.Com, Inc.
Basic Web Architecture
Workstation
Workstation
Web Tier
Data
Server
Proxy Server
Workstation
Workstation
Internet
Modem
Home PC
Internet Service Provider
July 20, 2015
Web Server
© Robert Sabourin, 2001
Application Server
Slide 56
AmiBug.Com, Inc.
Web Tier Content
Testing
• Content Testing Correctness
– Validation and Verification of content is a
critical part of testing any Web or E-Commerce
application
• Validate that all information presented is factual
• Verify that the correct information is presented
July 20, 2015
© Robert Sabourin, 2001
Slide 57
AmiBug.Com, Inc.
Platform Coverage
• What is a Platform?
– In testing we consider a “Platform” to be the
“Hardware” and “Software” environment that
an application under test is run on
– Typically required to operate on several
different platforms
– Platform can also include other concurrently
running applications
July 20, 2015
© Robert Sabourin, 2001
Slide 58
AmiBug.Com, Inc.
Browser Compatibility
Testing
• Web application should:
– Operate on all popular platforms (Requirements
based)
– Operate with all popular versions of plug-ins
and third party components or viewers required
– Up to corporate policy how far to take this!
www.browserwatch.com
Pareto principal
July 20, 2015
© Robert Sabourin, 2001
Slide 59
AmiBug.Com, Inc.
Browser Compatibility
Testing
• Standards?
– Largely overridden by browsers
– Differences
• user input
• display
• processing
– Differences in Java virtual machines
– Differences in Plug-Ins
July 20, 2015
© Robert Sabourin, 2001
Slide 60
AmiBug.Com, Inc.
Browser Compatibility
Testing
• Which Browser?
–
–
–
–
–
–
–
Microsoft Internet Explorer
Netscape Navigator
AOL Browser
Lotus Notes Browser
Web TV Browser
Mosaic
Opera
July 20, 2015
© Robert Sabourin, 2001
Slide 61
AmiBug.Com, Inc.
Browser Compatibility
Testing
• Which Version of Which Browser
– Battle of the browsers led to unprecedented
number of incompatible versions of supposedly
compatible products and technologies
– Beta releases used as if they were commercial
gold releases
– Non-standard features implemented in different
manners on each browser
– Patches and service packs
July 20, 2015
© Robert Sabourin, 2001
Slide 62
AmiBug.Com, Inc.
Browser Compatibility
Testing
July 20, 2015
© Robert Sabourin, 2001
Slide 63
AmiBug.Com, Inc.
Platform Coverage
• Which operating system (O/S)?
–
–
–
–
–
–
–
Windows 2000, XP
Windows 98 Original, 2nd Edition
Windows 95 Original, Upgrade A, OSR2 B
Window NT 4.0, SP3, SP4, SP5, SP6
Linux
Solaris
MAC OS
July 20, 2015
© Robert Sabourin, 2001
Slide 64
AmiBug.Com, Inc.
Platform Coverage
•
Which Version of Co-dependent Software?
–
•
Which Display Resolution?
–
–
–
–
–
•
640 x 480
800 x 600
1024 x 768
1280 x 1024
Custom
Which System Font Size?
–
–
–
•
Multiple versions possible for each object used or depended upon!
Small
Large
Custom
Which Color Depth?
–
–
–
–
–
–
Monochrome
4-bit
8-bit
16-bit
24-bit
32-bit
July 20, 2015
© Robert Sabourin, 2001
Slide 65
AmiBug.Com, Inc.
Platform Coverage
• Number of possible combinations are
tumultuous
• Cannot possible try each combination even
for a short period of time each
– Even if we restrict out testing to the most
commonly used variations over the past two
years or so!
July 20, 2015
© Robert Sabourin, 2001
Slide 66
AmiBug.Com, Inc.
Platform Coverage
• We cannot possibly try all combinations
• Instead
– vary platform used under testing in a manner
consistent with Technical Risk and Business
Risk associated with each platform component
– define a grid which indicates which platform
will be used at which time
– vary platforms with tests
July 20, 2015
© Robert Sabourin, 2001
Slide 67
AmiBug.Com, Inc.
Platform Coverage
Pattern
Example - Multi-Layered Pattern
– For each relevant platform component a pattern
is defined, in our example we care about the
following three characteristics of the
workstation:
• Pattern A - Browser
• Pattern B - Operating System
• Pattern C - Locale
July 20, 2015
© Robert Sabourin, 2001
Slide 68
AmiBug.Com, Inc.
Platform Coverage
Pattern
Example - Multi-Layered Pattern
– Two testing workstations are made available to
testers and are prepared well in advance
according to a predefined pattern, identified as
W001 and W002
– Four testing sessions are run per day in
different time slots know as TS01, TS02, TS03
and TS04
July 20, 2015
© Robert Sabourin, 2001
Slide 69
AmiBug.Com, Inc.
Platform Coverage
Pattern
Example - Multi-Layered Pattern
– Testing is organized into 17 different testing
sessions identified as T001, T002, … , T017
Time
Slot
TS01
TS01
TS02
TS02
TS03
TS03
TS04
TS04
Work
Station
WS001
WS002
WS001
WS002
WS001
WS002
WS001
WS002
Day 1
T001
T002
T003
T004
T005
T006
T007
T008
Day 2
T009
T010
T011
T012
T013
T014
T015
T016
Day 3
T017
T001
T002
T003
T004
T005
T006
T007
Day 4
T008
T009
T010
T011
T012
T013
T014
T015
Testing Session
Day 5
Day 6
T016
T007
T017
T008
T001
T009
T002
T010
T003
T011
T004
T012
T005
T013
T006
T014
Day 7
T015
T016
T017
T001
T002
T003
T004
T005
Day 8
T006
T007
T008
T009
T010
T011
T012
T013
Day 9
T014
T015
T016
T017
T001
T002
T003
T004
Day 10
T005
T006
T007
T008
T009
T010
T011
T012
– 80 sessions over 10 days of testing
July 20, 2015
© Robert Sabourin, 2001
Slide 70
AmiBug.Com, Inc.
Platform Coverage
Pattern
Example - Multi-Layered Pattern
– Pattern A - Browser
– Technical and Market Risk Assessment
Colour
Code
July 20, 2015
Risk
Number of
Browser Technical Business Exposure Sessions
IE 5.5
0.2
0.2
0.04
13
IE 4.0
0.2
0.4
0.08
27
NE 6
0.3
0.1
0.03
10
NE 4
0.3
0.3
0.09
30
Total
0.24
80
© Robert Sabourin, 2001
Slide 71
AmiBug.Com, Inc.
Platform Coverage
Pattern
Example - Multi-Layered Pattern
– Pattern A - Browser
– Resulting Pattern - Uniform Random
Time
Slot
TS01
TS01
TS02
TS02
TS03
TS03
TS04
TS04
Work
Station
WS001
WS002
WS001
WS002
WS001
WS002
WS001
WS002
July 20, 2015
Day 1
Day 2
Day 3
Day 4
Browser Pattern
Day 5
Day 6
© Robert Sabourin, 2001
Day 7
Day 8
Day 9
Day 10
Slide 72
AmiBug.Com, Inc.
Platform Coverage
Pattern
Example - Multi-Layered Pattern
– Pattern B - Operating System
– Technical and Market Risk Assessment
Colour
Code
July 20, 2015
Risk
Number of
O/S
Technical Business Exposure Sessions
Win 2000
0.1
0.1
0.01
4
Win 98SE
0.2
0.4
0.08
32
Win 98Or
0.3
0.2
0.06
24
Win NT4
0.1
0.2
0.02
8
Mac OS
0.3
0.1
0.03
12
Total
0.2
80
© Robert Sabourin, 2001
Slide 73
AmiBug.Com, Inc.
Platform Coverage
Pattern
Example - Multi-Layered Pattern
– Pattern B - Operating System
– Resulting Pattern - Uniform Random
Time
Slot
TS01
TS01
TS02
TS02
TS03
TS03
TS04
TS04
Work
Station
WS001
WS002
WS001
WS002
WS001
WS002
WS001
WS002
July 20, 2015
Day 1
Day 2
Day 3
Day 4
OS Pattern
Day 5
Day 6
© Robert Sabourin, 2001
Day 7
Day 8
Day 9
Day 10
Slide 74
AmiBug.Com, Inc.
Platform Coverage
Pattern
Example - Multi-Layered Pattern
– Pattern C - Locale
– Technical and Market Risk Assessment
Colour
Code
Risk
Number of
Locale Technical Business Exposure Sessions
US Eng
0.2
0.3
0.06
20
Can Eng
0.2
0.2
0.04
13
Can Fren
0.2
0.3
0.06
20
Chinese
0.4
0.2
0.08
27
Total
0.24
80
July 20, 2015
© Robert Sabourin, 2001
Slide 75
AmiBug.Com, Inc.
Platform Coverage
Pattern
Example - Multi-Layered Pattern
– Pattern C - Locale
– Resulting Pattern - Uniform Random
Time
Slot
TS01
TS01
TS02
TS02
TS03
TS03
TS04
TS04
Work
Station
WS001
WS002
WS001
WS002
WS001
WS002
WS001
WS002
July 20, 2015
Day 1
Day 2
Day 3
Day 4
Locale Pattern
Day 5
Day 6
© Robert Sabourin, 2001
Day 7
Day 8
Day 9
Day 10
Slide 76
AmiBug.Com, Inc.
Web Testing Checklists
• Checklists
– Your team can develop a series of one page
checklists as an aid to testing the user interface
of your web and e-commerce applications
– All team members are trained in their use
– During assigned testing activities, whenever a
checklist violation is observed, it should be
reported as a bug
July 20, 2015
© Robert Sabourin, 2001
Slide 77
AmiBug.Com, Inc.
Checklists
• Start with Checklists for:
–
–
–
–
Usability Testing Checklist
GUI Testing Checklist
Application Style Consistency Checklist
Design Mistakes Checklist
• Choose check list items from examples, standards,
guidelines and relevant past experience
July 20, 2015
© Robert Sabourin, 2001
Slide 78
AmiBug.Com, Inc.
Checklists
• For each checklist:
– One page per checklist
– Related, Rational, Relevant
– Train testing team
• Example of conformance
• Example of violation
– Same list used in design & development
July 20, 2015
© Robert Sabourin, 2001
Slide 79
AmiBug.Com, Inc.
Link Testing
• Hyperlinks
– Broken
• Point to nothing
• Tools can help find broken links
– Missing
• Link should be there but it is not
– Wrong
• Point to the wrong thing
• Take user to the wrong page
July 20, 2015
© Robert Sabourin, 2001
Slide 80
AmiBug.Com, Inc.
Link Testing
• Links
–
–
–
–
Are all links from a page valid?
Are all internal links relative?
Are all external links absolute?
External links should be periodically verified as
valid after the site is live.
– Can key pages be reached with minimal links,
for example under three links?
July 20, 2015
© Robert Sabourin, 2001
Slide 81
AmiBug.Com, Inc.
Link Testing
• Links
– Use site mapping tools to review the structure
of the Web Application
•
•
•
•
•
July 20, 2015
Any recursive links?
Any bi-directional links?
Are image maps correctly assigned links?
Are frame-sets managed correctly?
Can each page be accessed?
© Robert Sabourin, 2001
Slide 82
AmiBug.Com, Inc.
Link Testing
Example with eValid
July 20, 2015
© Robert Sabourin, 2001
Slide 83
AmiBug.Com, Inc.
Link Testing
Example with eValid
July 20, 2015
© Robert Sabourin, 2001
Slide 84
AmiBug.Com, Inc.
Link Testing
• Links
– Can a search engine of web application find
each page?
– Can you navigate to each page?
• Other
– Is Web Page Description correct and relevant
when bookmarked?
July 20, 2015
© Robert Sabourin, 2001
Slide 85
AmiBug.Com, Inc.
Link Testing
• Re-direction
– Are HTTP error codes managed by appropriate
Web Application pages
•
•
•
•
•
•
July 20, 2015
404 - Not Found
400 - Bad Request
401 - Unauthorized
403 - Forbidden
500 - Internal error
502 - Temporary overload
© Robert Sabourin, 2001
Slide 86
AmiBug.Com, Inc.
Testing Web and E-Commerce
Applications
Application Tier Testing
July 20, 2015
© Robert Sabourin, 2001
Slide 87
AmiBug.Com, Inc.
Basic Web Architecture
Workstation
Application
Tier
Workstation
Data
Server
Business
Logic
Proxy Server
Workstation
Workstation
Web Application
Functionality
Internet
Modem
Home PC
Internet Service Provider
July 20, 2015
Web Server
© Robert Sabourin, 2001
Application Server
Slide 88
AmiBug.Com, Inc.
Functional Testing
• Identify Functions
– Primary
– Contributing
• Prioritize
• Identify Test Cases
– Per function
– Explicit
July 20, 2015
© Robert Sabourin, 2001
Slide 89
AmiBug.Com, Inc.
Functional Testing
• So what is a function!
– “Anything the software is supposed to do”
• Results in a display
• Changes data
• Effects the environment
• Basically two classifications
– Primary
• FAILURE RENDERS PRODUCT USELESS FOR TYPICAL USER
– Contributing
• PRODUCT COULD STILL PERFORM USEFUL FUNCTIONS FOR
TYPICAL USER IF FAILURE
July 20, 2015
© Robert Sabourin, 2001
Slide 90
AmiBug.Com, Inc.
Test Case Design
• Characteristics of Good Test Case
–
–
–
–
–
–
Reasonable probability of catching a bug
Can be repeated
Not redundant
Best of breed
Neither too simple nor too complex
Based on reality
July 20, 2015
© Robert Sabourin, 2001
Slide 91
AmiBug.Com, Inc.
Test Case Design
• Test Case Characteristics
–
–
–
–
Consider Input
Consider Outcome
Consider Initial State
Consider End State
Input/Outcome
Input/Outcome
July 20, 2015
© Robert Sabourin, 2001
Slide 92
AmiBug.Com, Inc.
Modeling
• Test Case Design
– All state transitions
– All paths through the system
– Rational input
– Unexpected input
– Can we get to all states?
July 20, 2015
© Robert Sabourin, 2001
Slide 93
AmiBug.Com, Inc.
State Transition
Model of Web Site
• Consider each page of the site as a State
• Consider the users action as the input
• Consider the variable output as the outcome
July 20, 2015
© Robert Sabourin, 2001
Slide 94
AmiBug.Com, Inc.
State Transition
Model of Web Site
• Example mail.yahoo.ca!
July 20, 2015
© Robert Sabourin, 2001
Starting Page
Slide 95
AmiBug.Com, Inc.
State Transition
Model of Web Site
• Example mail.yahoo.ca!
July 20, 2015
© Robert Sabourin, 2001
Valid ID, Password
Slide 96
AmiBug.Com, Inc.
State Transition
Model of Web Site
• Example mail.yahoo.ca!
July 20, 2015
© Robert Sabourin, 2001
Mail Session Page
Slide 97
AmiBug.Com, Inc.
State Transition
Model of Web Site
• Example mail.yahoo.ca!
July 20, 2015
© Robert Sabourin, 2001
Invalid ID, Password
Slide 98
AmiBug.Com, Inc.
State Transition
Model of Web Site
• Example mail.yahoo.ca!
July 20, 2015
© Robert Sabourin, 2001
ID Error Page
Slide 99
AmiBug.Com, Inc.
State Transition
Model of Web Site
• Example mail.yahoo.ca!
Acknowledged
Valid ID
Mail Session
Start
Completed Session
July 20, 2015
© Robert Sabourin, 2001
ID Error
Invalid ID
Slide 100
AmiBug.Com, Inc.
State Transition
Model of Web Site
• Example mail.yahoo.ca!
TestId
tTid01
tTid02
tTid03
tTid04
July 20, 2015
Start State Input
Outcome End State
START
ValidID
Mail Sess.
START InvalidID ErrorMsg ErrorID
Mail Sess. Complete MailProc START
ErrorID
Ack.
START
© Robert Sabourin, 2001
Slide 101
AmiBug.Com, Inc.
Test Case Design
• Equivalence Classes
– Sets of input events which are similar and exercise the application
in the same way!
• Boundary Values
– Sets of input events on or very close to the boundary of accepted
valid input
• Test cases should include a mix of valid and
invalid input for all equivalence classes and
boundary conditions
July 20, 2015
© Robert Sabourin, 2001
Slide 102
AmiBug.Com, Inc.
Test Case Design
• Outcomes
– Positive
• have at least one test case to generate each possible
outcome
– Negative
• have at least one test case which does not generate
each possible outcome
July 20, 2015
© Robert Sabourin, 2001
Slide 103
AmiBug.Com, Inc.
Test Case Design
• Input
– Invalid inputs
• at least one test with each possible type of invalid input for
each type of field on a dialogue
• use gray box methods to determine common types of fields
– Pairs
• at least one test for each possible pair of invalid inputs
July 20, 2015
© Robert Sabourin, 2001
Slide 104
AmiBug.Com, Inc.
Test Case Design
• Error Conditions
– Error conditions
• at least one test for each possible error condition
July 20, 2015
© Robert Sabourin, 2001
Slide 105
AmiBug.Com, Inc.
Test Case Design
• Input Text
• literal numbers
– one, “1”, “i”, “l”, “I”
•
•
•
•
July 20, 2015
broken HTML, XML
cut and paste
control characters
navigation to field!
© Robert Sabourin, 2001
Slide 106
AmiBug.Com, Inc.
Real Data
• Real test data
– Logs from field are excellent source of real data
which is a gold mine for testing since it defines
the real usage scenarios from the field!
– Scramble to protect confidential information
July 20, 2015
© Robert Sabourin, 2001
Slide 107
AmiBug.Com, Inc.
Wisdom
• Repeated testing
– Make sure test cases can be repeated
• allows for regression
• allows for data driven test automation
– Test descriptions should be explicit
July 20, 2015
© Robert Sabourin, 2001
Slide 108
AmiBug.Com, Inc.
Web Tier Testing
Scenario Based Testing
July 20, 2015
© Robert Sabourin, 2001
Slide 109
AmiBug.Com, Inc.
Modeling
• Story Boards
– E-Commerce
• user experience
• what can be done
• paths through the system
July 20, 2015
© Robert Sabourin, 2001
Slide 110
AmiBug.Com, Inc.
Modeling
• User Experience
Login
Read message
Check Out specials
Buys some stuff
Goes to affiliate
July 20, 2015
© Robert Sabourin, 2001
Slide 111
AmiBug.Com, Inc.
Modeling
• User Experience Test Case
–
–
–
–
Parameterize experience
Walk through scenario from start to end
Use pre selected input for each case
Always run every test as if it were a user
experience
July 20, 2015
© Robert Sabourin, 2001
Slide 112
AmiBug.Com, Inc.
Data Parameterization
• Separate test procedure from test data
– Procedure
• Instructions to follow in order to run the test
• Navigation instructions
• Invariant part of a test
July 20, 2015
© Robert Sabourin, 2001
Slide 113
AmiBug.Com, Inc.
Data Parameterization
– Test Data
• Any data typed into a form or field by the end user
• Variable part of a test
– Potential other data
• Any other field
• Any decision
July 20, 2015
© Robert Sabourin, 2001
Slide 114
AmiBug.Com, Inc.
Data Parameterization
• Purchase a book from Amazon.com
• Procedure
–
–
–
–
–
–
–
–
go to URL www.amazon.com
wait for home page to load
in search field type in ISBN number of book
add book to shopping cart
proceed to checkout
log in with user name and password
ship to this address
place order
July 20, 2015
© Robert Sabourin, 2001
Slide 115
AmiBug.Com, Inc.
Data Parameterization
July 20, 2015
© Robert Sabourin, 2001
Slide 116
AmiBug.Com, Inc.
Data
ISBN Number
Data Parameterization
July 20, 2015
© Robert Sabourin, 2001
Slide 117
AmiBug.Com, Inc.
Data Parameterization
July 20, 2015
© Robert Sabourin, 2001
Slide 118
AmiBug.Com, Inc.
Data Parameterization
July 20, 2015
© Robert Sabourin, 2001
Slide 119
AmiBug.Com, Inc.
Data Parameterization
Data
User Name
Data
Password
July 20, 2015
© Robert Sabourin, 2001
Slide 120
AmiBug.Com, Inc.
Data Parameterization
July 20, 2015
© Robert Sabourin, 2001
Slide 121
AmiBug.Com, Inc.
Data Parameterization
July 20, 2015
© Robert Sabourin, 2001
Slide 122
AmiBug.Com, Inc.
Data Parameterization
• Data
–
–
–
–
URL
ISBN
USER NAME
PASSWORD
July 20, 2015
© Robert Sabourin, 2001
Slide 123
AmiBug.Com, Inc.
Data Parameterization
• Advantages
– Separates test code from data
– Leverage well defined test procedure
– Same procedure can be used for many different
purposes depending on data selected
– Critical first step in design of scenario based
test automation scripts
– Forces decisions to be deliberate - not random
or ad-hoc!
July 20, 2015
© Robert Sabourin, 2001
Slide 124
AmiBug.Com, Inc.
Exploratory Testing
July 20, 2015
© Robert Sabourin, 2001
Slide 125
AmiBug.Com, Inc.
Exploratory Testing
• An approach to “test design” that employs
general-systems heuristics and specific-systems
expertise to enable rapid development of test
cases.
• Approach formalized by James Bach
(www.satisfice.com)
• Used in General Functionality and Stability Test
Procedure for Windows 2000 Application
Certification
July 20, 2015
© Robert Sabourin, 2001
Slide 126
AmiBug.Com, Inc.
Exploratory Testing
• Test cases
– Not known in advance
– Defined & executed “on the fly” while you learn about the product
• Testers need to “hone up” their skills in
making maps!
– Consistent note taking style
– Practice
• We need to have a standard template
– Uniform way to capture details
July 20, 2015
© Robert Sabourin, 2001
Slide 127
AmiBug.Com, Inc.
Exploratory Testing
• During test we must capture
–
–
–
–
–
–
–
–
Function, options or sub-functions being explored
Test cases attempted
Comments, notes, images or attachments
Hints, reminders and observations which may be useful to future
testers
Date, Platform, Build or Configuration under test
Name of person running test
Oracles, “strategy to assess correctness”
Other relevant details
July 20, 2015
© Robert Sabourin, 2001
Slide 128
AmiBug.Com, Inc.
Confirm Test Objective
Ensure context known
Kick Off
Example
Exploration
Process
Ensure HW and SW OK
All tools available
Chunk of 90 to 120 min
Test, Plan, Discover
Prepare
Wrap up
Collect all notes data
Run
Review results with
Test Lead
Complete
Review
Reassess goals
Piece together map
July 20, 2015
© Robert Sabourin, 2001
Follow Up
Slide 129
AmiBug.Com, Inc.
Testing Web and E-Commerce
Applications
Data Tier Testing
July 20, 2015
© Robert Sabourin, 2001
Slide 130
AmiBug.Com, Inc.
Basic Web Architecture
Workstation
Data
Tier
Workstation
Data
Server
Application
Data
Proxy Server
Workstation
Workstation
Various
Repositories
Internet
Modem
Home PC
Internet Service Provider
July 20, 2015
Web Server
© Robert Sabourin, 2001
Application Server
Slide 131
AmiBug.Com, Inc.
Data Operations
•
•
•
•
Search for a record
Insert a new record
Delete an existing record
Change an existing record
July 20, 2015
© Robert Sabourin, 2001
Slide 132
AmiBug.Com, Inc.
Data Operations
• Search for a record
– Different Sizes of Database
• Small
• Typical
• Large
July 20, 2015
© Robert Sabourin, 2001
Slide 133
AmiBug.Com, Inc.
Data Operations
• Search for a record
– DBMS Cache impact
•
•
•
•
•
•
July 20, 2015
Random order
Inverse order
In order
Worst case
Best case
Repeated
© Robert Sabourin, 2001
Slide 134
AmiBug.Com, Inc.
Data Operations
• Search for a record
–
–
–
–
Rules and syntax
Data should not be found
Data should be found
Multiple selections found
July 20, 2015
© Robert Sabourin, 2001
Slide 135
AmiBug.Com, Inc.
Data Operations
• Search for a record
–
–
–
–
Multilingual database searches
Many concurrent queries in different languages
Watch out for search being correct
Observe performance and timing!
July 20, 2015
© Robert Sabourin, 2001
Slide 136
AmiBug.Com, Inc.
Data Operations
• Insert a new record
–
–
–
–
–
–
Before first record
After last record
Random points in table
Same data repeatedly
Invalid data
Valid data
July 20, 2015
© Robert Sabourin, 2001
Slide 137
AmiBug.Com, Inc.
Data Operations
• Delete a record
–
–
–
–
–
First record
Last record
Random points in table
Delete all data
Delete data which is pointed to!
• Insert / Delete
– Sequence repeatedly and randomly
July 20, 2015
© Robert Sabourin, 2001
Slide 138
AmiBug.Com, Inc.
Data Operations
• Change an existing record
– Build record progressively by a series of
changes
• Change name, then change address, then change
phone number ...
– Change to records being pointed to
July 20, 2015
© Robert Sabourin, 2001
Slide 139
AmiBug.Com, Inc.
Test Lab DBMS
Folk Wisdom
• Access
– Ensure that database use is Restricted
• accessed only by Test Team
• accessed by DBA when requested by Test Lead
• Privileges
– Ensure user privileges are consistent with typical usage environment
– execute tests as a NORMAL USER not as a ROOT USER!
• Tuning
– Ensure DBMS is configured with recommended or typical customer target
parameters for all tunable characteristics such as CACHE size
• State
– Ensure you can reset DBMS content by a restore or copy function the
Database to a known state
July 20, 2015
© Robert Sabourin, 2001
Slide 140
AmiBug.Com, Inc.
Real World Test Data
– Use test data which is derived from or inspired
by real world data
– Protect real names and identification by
carefully scrambling data
– Ensure security of any
» commercial
» medical
» personal identification
July 20, 2015
© Robert Sabourin, 2001
Slide 141
AmiBug.Com, Inc.
Self Checking Data
Approach
• Test Data Set
– Each record has at least one field which can be
used to confirm the integrity of the entire
record
– Useful to see if a database or a record of a
database has been corrupted as a result of
insert, delete and change operations.
July 20, 2015
© Robert Sabourin, 2001
Slide 142
AmiBug.Com, Inc.
Self Checking Data
Approach
• Consistency Checking
– Rules based
• Valid postal/zip code?
• Valid ID?
– Checksums
• Computed from data
– Side by Side Reviews
• Does data match across applications?
– Visible fields dependent of state of application
July 20, 2015
© Robert Sabourin, 2001
Slide 143
AmiBug.Com, Inc.
References
•
•
•
•
•
•
•
•
•
“Software Testing Methods and Tools Course Notes”, AmiBug.com, 2001,
Robert Sabourin.
“Stress Testing Web Applications Workship Notes”, AmiBug.com, 2001,
Robert Sabourin.
“Testing Applications on the Web”, Wiley 2000, Hung Nguyen.
“Testing Process”, MSDN, 2000, Biraj Rath, Raj Nath, Mukesh Agarwal, Jas
Lamba
www.satisfice.com
“General Functionality and Stability Test Procedure for Windows 2000
Application Certification”, Microsoft, 2000, James Bach.
www.riceconsulting.com
www.useit.com
www.csst-technologies.com
July 20, 2015
© Robert Sabourin, 2001
Slide 144
AmiBug.Com, Inc.
References
•
•
•
•
•
•
“Effective Methods for Software Testing”, Second Edition, Wiley 2000,
William Perry.
“The Engineering of Software”, Addison Wesley, 2001, Hamlet, Maybee.
“Configuration Management, The Missing Link in Web Engineering”, Artech
House, 2000, Susan Dart.
“Lessons Learned in Software Testing”, Wiley, 2002, Kaner, Bach, Pettichord
“Managing the Testing Process”, Microsoft Press, 1999, Rex Black
“I am a Bug”, AmiBug.Com, 2000, Robert Sabourin
July 20, 2015
© Robert Sabourin, 2001
Slide 145
AmiBug.Com, Inc.
Thank You
• Questions?
July 20, 2015
© Robert Sabourin, 2001
Slide 146
AmiBug.Com, Inc.