Transcript Testing Computer Software 2nd Edition (1999)

Testing Computer Software
2nd Edition
(1999)
Cem Kaner
Jack Falk
Hung Quoc Nguyen
Testing Computer Software
• Objectives and Limits of Testing
– Realistic test planning is dominated by the need to
select a few test cases from a huge set of
possibilities.
– You cannot fully/completely test a program
– The prime benefit of testing is improved quality
Testing Computer Software
• Software Errors
–
–
–
–
–
–
–
User Interface Errors
Error Handling (one of the least tested areas)
Boundary-Related Errors
Calculation Errors
Initial and Later Stages (happens only the first time)
Control Flow Errors (menus to submenus)
Errors in Handling and Interpreting Data (between
modules)
– Race Conditions
Testing Computer Software
• Software Errors (continued)
– Load Conditions
– Hardware
– Source and Version Control
– Documentation (manual, book)
– Testing Errors (code, procedures, user error)
Testing Computer Software
• Reporting and Analyzing Bugs
– The point is to get bugs fixed
– A good report is written, simple, understandable,
reproducible, legible, and non-judgmental
• The Bug Tracking System
– The system introduces project accountability
– Users of the system (Testers, Project Manager,
Programmer, Product Manager, Technical Support,
Writer, Test Manager, Senior Executives, Lawyers)
Testing Computer Software
• The Bug Tracking System (continued)
– The authors strongly discourage providing
personal performance data
– Generate many useful reports (weekly status,
deferred, patches)
Testing Computer Software
• Test Types and Test Case Design
– Pre-testing stage
• Requirements review
• Design review
– Coding stage
• Whitebox/Glassbox Testing
– Testing stage
• Blackbox Testing
–
–
–
–
Functional Testing
System Testing
Load Testing
Configuration Testing
» Test only one device per group to eliminate the errors common to that
group.
• User Manuals
Testing Computer Software
• Test Types and Test Case Design (continued)
– Equivalence Classes (same result from two
different tests)
• Functional Equivalence Testing (automation, random
input)
– Boundary Analysis
– Testing state transitions (ex. Menus)
– Race Conditions
Testing Computer Software
• Test Types and Test Case Design (continued)
– Regression (bug fixes, library of tests)
• Testing Tools
– Basic Tools
• File compare Utility, File Viewers
– Automated Regression Tests
– Standards compliance-checking programs
– Tools for code coverage, assertion checks, memory
Testing Computer Software
• Test Planning and Documentation
– Test Plan as a product and Test Plan as tool
– Goal of Test Plans and Docs
• Facilitate technical tasks of testing (study program,
more complete coverage)
• Describe testing tasks and processes (size, depth of
testing job)
• Organize testing project (improve individual
accountability, schedule)
Testing Computer Software
• Test Planning and Documentation (continued)
– White box vs Black box
• They are complementary. They find different types of
bugs.
– Suggest evolutionary method for developing test
plan
• Start small and build and add new sections as you go
along. Avoid huge test plan at the start of testing.
Testing Computer Software
• Test Planning and Documentation (continued)
•
•
•
•
•
Test Plan (defined by IEEE Standard 829)
Acceptance Test Criteria
Test Design spec
Test Summary Report (complete a cycle of testing)
Some Minor docs
–
–
–
–
Test case spec and Test procedure spec
Test item transmittal report
Test script
Test log
Testing Computer Software
• Software Development Model implications for
testing
– Waterfall
•
•
•
•
You must review the UI early
Start writing test plan as early as possible (risk assess)
You cannot start testing until late in the project
When you start testing, you are on the critical path
Testing Computer Software
• Software Development Model implications for
testing
– Evolutionary
• Start reliability testing as soon as first level of
functionality – plan to staff early
• Plan waves of usability tests as project grows more
complex
• Write test plan as you go
• Do your most powerful testing as soon as possible
because project might be stopped and shipped at any
time
Testing Computer Software
• Managing a testing group
– Management is the real quality control group in
any company
– Provide predictability to the project manager
– Different Types of Testing groups
•
•
•
•
QC – enforces standards
QA – senior people involves at all stages
Testing services – find bugs
Development Services – debug, initial tech support