Transcript Acceptance Testing - University of Wisconsin

Acceptance Testing
Sascha Wener, Paul Lovy, Sean Morgan, Jacob Brunette, Nathan Blaubach
History
1. 1996-
Automated testing is used in XP process.
2. 2002Ward Cunningham, one of the founders of XP, “publishes Fit,
a tool for acceptance testing based on a tabular, Excel-like notation”.
(http://fit.c2.com/)
3. 2003Bob Martin uses Fit to create FitNesse.
(http://www.fitnesse.org/)
4. 2006-
Modern day Acceptance testing is realized
2
Why Acceptance Testing?
“The goal of User Acceptance Testing is to assess if the system can support day-to-day
business and user scenarios and ensure the system is sufficient and correct for business
usage.”(www.develop.com)
Saves Money, Time, and Reputation
The cost of fixing an error increases exponentially compared to how long it lives in the
software lifecycle.
Acceptance testing can find these errors before they ever surface.
3
Boehm’s law in the Software Lifecycle
4
Relative Costs to Fix
Software Defects
5
Types of Tests
• User acceptance testing
• Operational acceptance testing
• Contract and regulation acceptance testing
• Alpha testing
• Beta testing
6
User acceptance testing
•
Design an UAT strategy as part of planning
•
Design real-world Test Cases to cover all functional scenarios
•
Customize UATs if needed, it is not a "one-size fits all" activity.
•
Select members of the testing team which represents real world end-
users
•
Execute Test-Cases and compose testing document
•
Fix bugs found during testing
•
Sign-off, acceptance of software application to meet user requirements
7
Operational Acceptance Testing (OAT)
• Takes place after User Acceptance Testing
• Final version before a system is released
• Verification testing of procedures
• Testing housekeeping procedures
8
Operational Acceptance Testing (OAT)
The OAT process ensures that a system has minimized the frequency and the
magnitude of the effect of these following events:
Backup & Recovery
Shutdown & Resumption
Change Implementation
Operational Support & Procedures
Change Back-out
Alerts
Component Failure
9
Contract and regulation acceptance
testing
•
The constraints of the testing environment defined within contracts
acceptance criteria.
•
Solution must work within contract constraints.
•
System is tested against specified criteria
•
Validation of conformation with regulations set by government or local
authorities usually regarding law and safety.
10
Alpha Testing
Alpha Test: Software is near its development and is internally tested for bugs/issues
How long they last
Usually 3 - 5 times the length of the beta test with many iterations.
Who participates (tests)
•
•
•
Engineers
Employees
sometimes “friends and family”.
Focuses on testing that would emulate ~80% of the customers.
11
Alpha Testing
What defects testers should expect
•
•
•
Bugs
Crashes
Missing docs and features
How are defects addressed
•
•
Most known critical issues are fixed
Features may change or be added
What Alpha testing achieves
•
•
Sets well-defined benchmarks
Measures a product against those benchmarks
12
Beta Testing
What Beta testing does
•
•
•
Improve the quality of the product
Integrate customer input on the complete product
Ensure release readiness.
When Beta testing happen
Just prior to launch, sometimes ending within weeks or even days of final release.
How long Beta testing lasts
Usually only a few weeks (sometimes up to a couple of months) with few major iterations.
Who cares about Beta testing
Usually involves product marketing, support, docs, quality and engineering
Who participates in Beta testing
“Real customers”
13
Beta testing
What defects testers should expect
•
•
•
•
Some bugs
Fewer crashes
Most docs
Feature complete.
How are defects addressed
Much of the feedback collected is considered for and/or implemented in future versions of the product.
Only important/critical changes are made.
What Beta testing achieves
About chaos, reality, and imagination. Beta tests explore the limits of a product by allowing customers to
explore every element of the product in their native environments.
When Beta testing is over
Developers have a good idea of:
•
What your customer thinks about the product?
•
what s/he is likely to experience when they purchase it?
What happens next?
14
Beta testing
Release Party!
15
Alpha vs. Beta Testing
Alpha Testing
Beta Testing
Alpha testing performed by Testers who are usually internal
employees of the organization
Beta testing is performed by Clients or End Users who are not
employees of the organization
Alpha Testing performed at developer's site
Beta testing is performed at client location or end user of the product
Reliability and security testing are not performed in-depth Alpha
Testing
Reliability, Security, Robustness are checked during Beta Testing
Alpha testing involves both the white box and black box techniques
Beta Testing typically uses black box testing
Alpha testing requires lab environment or testing environment
Beta testing doesn't require any lab environment or testing environment.
Software is made available to the public and is said to be real time
environment
Long execution cycle may be required for Alpha testing
Only few weeks of execution are required for Beta testing
Critical issues or fixes can be addressed by developers immediately
in Alpha testing
Most of the issues or feedback is collected from Beta testing will be
implemented in future versions of the product
Alpha testing is to ensure the quality of the product before moving to
Beta testing
Beta testing also concentrates on quality of the product, but gathers
users input on the product and ensures that the product is ready for real
time users.
16
Alpha Beta Test Schedule
17
Developing test case scenarios
Documents use to Develop Test Scenarios:
1.
Business Use Case, Business Process Flows,
2.
Project Charter, Context Diagram,
3.
Business Requirements,
4.
Document (BRD),
5.
System Requirements Specification (SRS),
6.
Testing Guidelines and Techniques,
7.
Other Vendor’s Deliverables
A document can then be generated to ensure that the solution meets
the business requirements.
18
Practice Examples
•
Video: http://www.huffingtonpost.com/2011/10/28/woman-receives-massive-parkingticket_n_1063780.html
•
http://www.aviationtoday.com/av/commercial/Rohde-and-Schwarz-Completes-SiteAcceptance-Testing-of-German-ANSP-RadiocommunicationsSystem_83476.html#.VID23hA4LmM
•
http://www.thebatt.com/news/view.php/857508/Space-the-Aggie-frontier
19
Sources
https://www.develop.com/useracceptancetests
http://www.guru99.com/alpha-beta-testing-demystified.html
http://www.techopedia.com/definition/3887/user-acceptance-testing-uat
http://guide.agilealliance.org/guide/acceptance.html
http://riceconsulting.com/home/index.php/User-AcceptanceTesting/keys-to-successful-user-acceptance-testing.html
20
Sources (Continued)
http://www.centercode.com/blog/2011/01/alpha-vs-beta-testing/
http://www.khannur.com/stb2.10.htm
http://www.testingperformance.org/definitions/what-is-operationalacceptance-testing
http://qainfotesting.blogspot.com/2010/04/operational-acceptancetesting-oat_27.html
https://briankegels.wordpress.com/2012/07/24/the-purpose-of-analysis/
http://www.isixsigma.com/industries/software-it/defect-preventionreducing-costs-and-enhancing-quality/
21