Transcript Software Testing - University of Illinois at Urbana

cs498dm
Software Testing
Darko Marinov
January 22, 2008
Course Overview
• Introduction to software testing
– Systematic, organized approaches to testing
– Based on models and coverage criteria
– Testing is not (only) about finding “bugs”
– Improve your testing (and development) skills
• Teaching staff
– Insructor: Darko <[email protected]>
– TA: Vilas <[email protected]>
Administrative Info
• FAQ: Deliverables?
– No exams: no final, no midterm
– Five problem sets (5*15%) and a project (25%)
• Project: proposal, two reports, hopefully bug reports
• Project is on testing a piece of refactoring engine
• Undergrads must be registered for 3 hours
– For more, you can do an independent study
– If interested in research on testing, contact me
• Juniors: ask for permission if you didn’t
“Assignment”
• Did you sign up on Wiki
https://agora.cs.uiuc.edu/display/cs498dmsp08/Home
• Did you receive emails on the list
[email protected]
• Did you try out Eclipse/NetBeans?
– Which refactorings did you try?
“Prizes”
• Potential categories
– The “buggiest” bug found in the course
• Hardest to find, most important, realistic case study
– Most bugs found or tests generated
• Not the best measures of testing effort
• Occasional candies to encourage your
participation and discussion in lectures
– Used to be a part of grade but not any more
Several Warm-Up Topics
• Your opinion about the guest lecture?
– Do you want to have more of them?
• My report from the Workshop on Teaching
Software Testing (WTST 7)
– No technique is perfect
– Abstraction/modeling is important
• Think of various ways that software could break
– Testing for testers vs. testing for developers
• Do you look at source code while testing?
This Lecture: Introduction (Cont’d)
•
•
•
•
Why look for bugs?
What are bugs?
Where they come from?
How to detect them?
• Topics that will be covered in the course
• Related topics that will not be covered
“Bugs” in IEEE 610.12-1990
• Fault
– Incorrect lines of code
• Error
– Faults cause incorrect (unobserved) state
• Failure
– Errors cause incorrect (observed) behavior
• Not used consistently in literature!
Correctness and Quality
• Common (partial) properties
– Segfaults, uncaught exceptions
– Resource leaks
– Data races, deadlocks
– Statistics based
• Specific properties
– Requirements
– Specification
Traditional Waterfall Model
Requirements
Analysis
Design
Checking
Implementation
Unit Testing
Integration
System Testing
We will look at general
techniques, applicable in
several phases of testing
Maintenance
Regression Testing
Phases (1)
• Requirements
– Specify what the software should do
– Analysis: eliminate/reduce ambiguities,
inconsistencies, and incompleteness
• Design
– Specify how the software should work
– Split software into modules, write specifications
– Checking: check conformance to requirements,
using for example conformance testing
Phases (2)
• Implementation
– Specify how the modules work
– Unit testing: test each module in isolation
• Integration
– Specify how the modules interact
– Integration testing: test module interactions
– System testing: test the entire system
• Maintenance
– Evolve software as requirements change
– Regression testing: test changes
Testing Effort
• Reported to be >50% of development cost
[e.g., Beizer 1990]
• Microsoft: 75% time spent testing
– 50% testers who spend all time testing
– 50% developers who spend half time testing
When to Test
• The later a bug is found, the higher the cost
– Orders of magnitude increase in later phases
– Also the smaller chance of a proper fix
• Old saying: test often, test early
• New methodology: test-driven development
(write tests even before writing code)
Software is Complex
•
•
•
•
•
•
Malleable
Intangible
Abstract
Solves complex problems
Interacts with other software and hardware
Not continuous
Software Still Buggy
• Folklore: 1-10 (residual) faults per 1000
nbnc lines of code (after testing)
• Consensus: total correctness impossible
to achieve for complex software
– Risk-driven finding/elimination of faults
– Focus on specific correctness properties
Approaches to Detecting Bugs
•
•
•
•
Software testing
Model checking
(Static) program analysis
…
Software Testing
• Dynamic approach
• Run code for some inputs, check outputs
• Checks correctness for some executions
• Main questions
– Test-suite adequacy (coverage criteria)
– Test-input generation
– Test oracles
Other Testing Questions
•
•
•
•
•
•
•
•
Selection
Minimization
Prioritization
Augmentation
Evaluation
Fault Characterization
…
Testing is not (only) about finding faults!
Current Status
• Testing remains the most widely used
approach to finding bugs
– Validation: are we building the right system?
– Verification: are we building the system right?
• Testing is gaining importance with test-first
development and increased reliability needs
• A lot of research on testing (part of mine too)
– We’ll look mostly at techniques used in practice
“Schools” of Software Testing
• Bret Pettichord described four schools
– Analytic (a branch of CS/Mathematics)
– Factory (a managed process)
– Quality (a branch of quality assurance)
– Context-Driven (a branch of development)
• This course focuses on artifacts, not process
• Do you want a guest speaker from industry?
Topics Related to “Finding Bugs”
• How to “eliminate bugs” (localize faults)?
– Debugging
• How to “prevent bugs”?
– Programming language design
– Software development processes
• How to “show absence of bugs”?
– Theorem proving
– Model checking, program analysis
Testing Topics to Cover
• Test coverage and adequacy criteria
– Graph, logic, input domains, syntax-based
•
•
•
•
•
•
Test-input generation
Test oracles
Model-based testing
Testing software with structural inputs
Test automation
Testing in your domain of interest?
Your Interests
•
•
•
•
•
•
•
•
•
Testing methods and methodology (*2)
Writing good/correct test cases (*3)
Better/more effective testing (*5)
Working with testers/developers (*2)
Relationship of testing and development cycle
Improve development (*4)
Software verification and (test) automation (*2)
Job skills (*3) (short-term or long-term?)
Please update me as we go through the course
Summary of the Introduction
• Eliminate bugs to save lives and money
• “Bugs” may mean faults, errors, failures
• Several approaches for detection: software
testing, model checking, static analysis…
• Software testing is the most widely used
approach for validation and verification
– We will cover systematic approaches to testing,
based on coverage criteria for various models
– Testing is not (only) about revealing faults
Next Lecture
• Thursday, January 24, 3:30pm, 1304 SC
– Example Interactive Testing Session
•
•
•
•
Test some refactoring
It should be fun
We can learn from mistakes as we go
We will learn some testing terminology
• Assignments
– Really sign up on Wiki
– Really try out Eclipse and/or NetBeans