Transcript Context Driven Testing

See...Think…Respond
Context Driven Testing
Adam White
Founder
Manager Test Engineering and Escalations
Agenda – Context Driven Testing
• Why did this topic come up?
• What exactly is context driven testing?
• Why should you care about context driven testing
• How can you apply it at your organization?
Background
• Who started this topic
• When was it started?
• Why talk about it now?
Across the portfolio –
How would we test these products?
ScriptLogic
Desktop Authority
Comprehensively manages the Windows desktop throughout the desktop lifecycle, reducing
the total cost of desktop and application ownership by enabling administrators to proactively
manage, inventory, secure and support desktops from a central location.
Awareness
Security Explorer
A powerful, graphical solution for real-time, interactive management of access controls and
Windows NT/2000/XP/2003/Vista security on Windows file servers and workstations across
the enterprise
Acronis
True Image Enterprise
Server
Optimization
Allows you to 1) create an exact server disk image, including the operating system,
applications, and configurations and 2) backup mission critical databases.
Migration
Eyeblaster
eb.campaign manager
An all encompassing ad server that is designed specifically for the generation and delivery
of campaigns/creative units from rich media and standard banners to search bid
management and more.
Feedback From the Portfolio
• Every major feature should have, at a minimum, a couple bugs
found and assigned in the defect tracking system.
– No bugs almost always means the feature has not been well tested.
• A tester’s primary role is to find bugs and not to simply pass test
cases.
• Limiting the testing cycle, by extending development time, results
in a product with a high number of defects.
• All features presented by the product should function as intended.
Feedback From the Portfolio - continued
• A quality product depends on adequate and accurate
requirements.
– Requirements are the basis for adequate test coverage
planning.
• Automated test scripts should be maintained for core
product features.
– Automated test scripts allows QA maximize test
coverage and maintain staff size
– The number of automated scripts should increase from
release to release in order to extend the amount of
functionality covered by automated testing.
Across the portfolio –
How would we test these products?
ScriptLogic
Desktop Authority
Comprehensively manages the Windows desktop throughout the desktop lifecycle, reducing
the total cost of desktop and application ownership by enabling administrators to proactively
manage, inventory, secure and support desktops from a central location.
Awareness
Security Explorer
A powerful, graphical solution for real-time, interactive management of access controls and
Windows NT/2000/XP/2003/Vista security on Windows file servers and workstations across
the enterprise
Acronis
True Image Enterprise
Server
Optimization
Allows you to 1) create an exact server disk image, including the operating system,
applications, and configurations and 2) backup mission critical databases.
Migration
Eyeblaster
eb.campaign manager
An all encompassing ad server that is designed specifically for the generation and delivery
of campaigns/creative units from rich media and standard banners to search bid
management and more.
The Seven Basic Principles of Context
Driven Testing
1) The value of any practice depends on its context.
2) There are good practices in context, but there are no best
practices.
3) People, working together, are the most important part of any
project's context.
4) Projects unfold over time in ways that are often not predictable.
The Seven Basic Principles - continued
5) The product is a solution. If the problem isn't solved, the product
doesn't work.
6) Good software testing is a challenging intellectual process.
7) Only through judgment and skill, exercised cooperatively
throughout the entire project, are we able to do the right things at
the right times to effectively test our products.
What people tend to forget about testing
• Testing groups exist to provide testing-related services. They do not run
the development project; they serve the project.
• Testing is done on behalf of stakeholders in the service of developing,
qualifying, debugging, investigating, or selling a product. Entirely
different testing strategies could be appropriate for these different
objectives.
• It is entirely proper for different test groups to have different missions. A
core practice in the service of one mission might be irrelevant or counterproductive in the service of another.
• Metrics that are not valid are dangerous.
• The essential value of any test case lies in its ability to provide
information (i.e. to reduce uncertainty).
What people tend to forget about testing
• All oracles are fallible. Even if the product appears to pass your test, it might
well have failed it in ways that you (or the automated test program) were not
monitoring.
• Automated testing is not automatic manual testing: it's nonsensical to talk
about automated tests as if they were automated human testing.
• Different types of defects will be revealed by different types of test
• Tests should become more challenging or should focus on different risks as
the program becomes more stable.
• Test artifacts are worthwhile to the degree that they satisfy their stakeholders'
relevant requirements.
Across the portfolio –
How would we test these products?
ScriptLogic
Desktop Authority
Comprehensively manages the Windows desktop throughout the desktop lifecycle, reducing
the total cost of desktop and application ownership by enabling administrators to proactively
manage, inventory, secure and support desktops from a central location.
Awareness
Security Explorer
A powerful, graphical solution for real-time, interactive management of access controls and
Windows NT/2000/XP/2003/Vista security on Windows file servers and workstations across
the enterprise
Acronis
True Image Enterprise
Server
Optimization
Allows you to 1) create an exact server disk image, including the operating system,
applications, and configurations and 2) backup mission critical databases.
Migration
Eyeblaster
eb.campaign manager
An all encompassing ad server that is designed specifically for the generation and delivery
of campaigns/creative units from rich media and standard banners to search bid
management and more.
What people tend to forget about testing
•
Article
Aligning Testing with its Context.
http://www.stpmag.com/retrieve/stp-0612.htm.
•
Resources
www.context-driven-testing.com
www.developsense.com
www.satisfice.com
www.testingeducation.org
•
Training
Rapid Software Testing by Michael Bolton and/or James Bach
Highly Recommended!
Questions?
What else is important to you?