Program Objectives
Download
Report
Transcript Program Objectives
Test Plan: Introduction
Primary focus: developer testing
– Implementation phase
– Release testing
– Maintenance and enhancement
Secondary focus: formal system verification
– Addressed within current test plan via examples
– Assumed to be primarily independent
Tests to be Performed
Bottom-up integration testing
– Build a module, build a test to simulate how it is used
Black-box
– Based on specification and ‘educated guess’ stress tests
White-box
– Based on code inspection
Platform testing
– Establish baseline capability of hardware/OS, detect differences
Performance
– Per module, and peer to peer distributed costs
Test coverage
– Coverage level dependent on resources, time, and cost of failure
Allocation of Testing Responsibilities
Assumption: development team performs
–
–
–
–
Internal (module-level) performance
Sample system performance (limited example federation)
Full white-box
Limited black-box (basic system functionality)
Assumption: external group performs
– Independent, detailed system verification (Spec compliant)
– Standardized performance tests
Test coverage
– Coverage will be measured, analyzed for relative effectiveness
– Coverage levels TBD
Testing Philosophy
Testing is not just pre-release! Continual process within
development phase
Catch defects as early as possible
Make sure defects stay fixed
Track cause of defects: repair problem, do not keep repatching the tire
Need support at both design level and implementation
level to accomplish these goals
Continual Testing Process
Tests created during development
Central code repository: modules, test suites tied together
– Tests are treated as live code to be maintained, not ‘once-offs’
– Test documentation: how to run, what is tested, and why
Revision control on both modules and tests
Modular development
– System broken down into hierarchies of testable components
Automated, incremental integration testing
– As code is developed, it is tested standalone, then incrementally
within the confines of the system
Continual feedback into development, maintenance cycle
– Weekly postings of performance results, current defect status
Design Support
Standard testing and debugging methods on each module /
class (peek, dump, exercise, performance)
Self-checking code (pre and post parameter asserts, valid
state calls)
Debug levels, controlled via runtime flags
Centralized logging mechanisms to collect distributed
traces
Logs used in both developer testing and sample user traces
from the field
Development Tool Support
Shadow development trees for automated and individual
testing (ClearCase ‘views’)
Common testing tools, testing approach to simplify testing
process, and to allow automated testing
Examples:
–
–
–
–
–
–
Standard test harnesses, method of invocation
Standard testing directories, makefile commands per module
Standard set of test-record-evaluate tools
Central I/O mechanisms to collect distributed test results
Standard system-level drivers
Sequential test harnesses and emulated distributed harnesses to
emulate determinism during development
Levels of Testing per Module
Basic: used in initial development, later in porting
– Minimal level of functionality to establish module operation
– Simplicity is critical during development, porting
Detailed: used to verify module correctness before
checking into current baseline
– Does module meet interface contract
– Tests for common coding errors
Regression: replicates previous use which caused failure
– Used to verify defect has been corrected
– Used to ensure defect does not re-occur over time
Performance: tracks the CPU usage per module and peer to
peer performance across distributed components
Complex Functional Areas: Testing
Examples
Memory: continual test for memory leaks and over-writes
– Automated use of Purify within weekly test suites across all
modules, all levels of the system
Threads: platform-specific performance and
implementation variances must be established per platform
– Standard set of tests which mimic system use of threads
Causality: complex areas of the system (such as zerolookahead ordering) are difficult to establish correctness
across all use cases (dropped packets, simultaneous time
stamps with variable arrival times, cancelled events, …)
– Detailed test scripts, executed in deterministic test harnesses with
varying error conditions
Periodic Testing Activities
Code walkthroughs: encompass both system code and
associated test suites. Testing focus:
– Do the test suites sufficiently stress the module
– Do the test suites still accurately represent the expected use of the
module
– Has the underlying platform changed, and has performance
changed accordingly
Change Control Board: formal tracking process and tools
to establish, record and monitor status of functional change
requests, defect priority and status data