Transcript 20 Golden Rules for Quality Testing

20 Golden Rules for Quality
Testing
Andy Redwood
© Andy Redwood 2003-2006
1
What’s In?
A selection of these 







Risk Assessed Testing
How to save money
Capability Maturity
Testing Requirements
Test Estimating
Quick wins with
Strategies
Marrying documentation
and traceability
Philosophy and
psychology of testing







2
Test only what you need
to
Can you learn from your
defects?
Reusing Test Assets
Test Environments
Measures and metrics
Use of Tools
Importance of sign off
Golden Rule Number 1
Testing is a Risk Management exercise.

Is anybody worried?


why expend valuable resources searching for defects
that are not important or viable to find?
Ensure all the Requirements are risk-assessed.



to the best method that works for you or your
organisation.
prioritise all the testing to target the most risk with the
least test effort.
when there are only minor risks outstanding, an informed
decision can be made on when to stop testing.
3
Golden Rule Number 2
Training in Testing reduces long term costs.

Budget for Test Training.


Training in Best Practice.




less than 1% of an organisations training budget
dedicated to training in testing is not unusual.
formal training in Test Management, Test Design, Static
and Dynamic techniques.
ISEB Foundation and Practitioners Certification.
Spread the Message.
Integrate testing with other disciplines.
4
Golden Rule Number 3
The Testing Culture in an organisation can
only mature at the rate of its capability.
Don’t try too much too soon.
 Establish all the perceived problems.
 Create a balanced economy of scale.
 What can be achieved will vary from one
organisation to another.

5
Golden Rule Number 4
The greater the number of late changes to
Requirements, the more rework is required.

Firm Requirements?


requirements are not just changing but new ones are
being created, even when you’re in Test Execution?
Adopt the “Bus to Bus Stop” rule.



requirements get on the Bus.
nothing else gets on and nothing gets off (ring-fenced
scope). requirements may change seats (change
control).
new requirements get on the next Bus coming along
behind (good release management).
6
Golden Rule Number 5
Testing estimates should be based on
Business Risk.

Do you guess?






collect testing measures and metrics.
use previous project information to guide you.
weight the risk with a ‘time taken to test’.
improve with each proceeding project.
predict with ever increasing accuracy.
Warning.



‘similar’ is not the ‘same as’.
you will need to evaluate any new ‘hotspots’.
please don’t estimate by habit!
7
Golden Rule Number 6
Drafting a Test Strategy from a facilitated
workshop is a Quick Win.

Strategy Workshops.




confirm the status known testing requirements very early
in the life cycle.
unknown activities surface early, creating contingency
within the programme.
my last 100 Test Strategy workshops have raised an
average 36 previously unknown high risk issues.
some of these would not otherwise have been spotted
until system test execution.
8
Golden Rule Number 7
Linking Development documents to Test Documents improves
understanding.




demonstrate the
relationship – ‘what’ vs
‘how’.
documentation is what
we as testers must test
back to.
in many cases this
relationship is implied
and not explicit.
it’s not in the
methodologies.
Feasibility
Study
Test Strategy
Project
Definition
User Acc.
Test Plan
Requirements
Definition
Ops. Accept.
Test Plan
Operational
Requirements
System
Test Plan
Integration
Test Plan
System Design
Document
Unit
Test Plan
Detailed Design
Document
KEY
Development
Deliverable
Testing
Deliverable
User Acc.
Test Spec.
9
Ops. Accept.
Test Spec.
System
Test Spec.
Integration
Test Spec.
Unit
Test Spec.
Golden Rule Number 8
All test conditions should have a parent
requirement.

Traceability.




between Requirements and Test Conditions within a Test
Repository.
all Requirements are tested.
aids Test Progress reporting.
Eye of a Bird.


scan back and forth looking for requirements that have
no conditions, also look for test conditions that do not
map to a requirement.
you may identify missing business processes or find an
item that requires a Business decision.
10
Golden Rule Number 9
Always test the Test Environment.

A few common phrases.




“The log-ons were all wrong”.
“The interface link was down”.
“We were pointing at the wrong files”.
Why is this?



lack of ‘pipe-cleaning’ of the test environment prior to
starting the real testing.
insufficient check-lists.
not reusing previous knowledge.
11
Golden Rule Number 10
Tests should be executed to prove that the
item under test doesn’t work, rather than
that it does work.

Is this the mindset?





still a radical concept in some quarters?
psychological conflict with reporting a high number of
errors and making good progress to plan.
high number of errors found in test must remain an
important test objective.
ensures a high quality and robust systems in Production.
tests should be designed to challenge high-risk attributes
at the earliest opportunity.
12
Golden Rule Number 11
It is better to test the errors out in early
Project phases as opposed to having a
failure in Production.

Origination.


evidence suggests that 50% of all error found in
system testing originate before coding (I.e. in
requirements definition or design).
Cost.

it can cost hundreds of times more to fix a problem in
production than in the Requirements phase.
13
Golden Rule Number 11
It is better to test the errors out in early
Project phases as opposed to having a
failure in Production.

Focus.


on static testing techniques, for example,
walkthroughs, reviews, workshops and formal
inspections will raise early incidents.
Warning.

it is still radical for testing to be perceived as a full lifecycle activity.
14
Golden Rule Number 12
Each test phase should attempt to test
something different with as little duplication
as possible.




duplication costs. Some duplication is unavoidable.
testing is more efficient and effective if each test phase
is defined with minimal duplication, within the Test
Strategy or Approach document.
details for each test phase will appear in phase specific
test plan documents.
question if a particular test phase is actually required.
 base this decision on risk and viability.
 prime objective for undertaking the phase, in some
cases, has already been proven in previous phases.
15
Golden Rule Number 13
An incident is not a error or a fault, but is
anything that was not expected.

The Nature of Incidents.




incidents can occur in any Programme phase.
incidents need not be testing specific.
an incident that results from human error or a component
fault will be logged as a problem.
Evaluation of incidents.



categorise them under strategically defined incident
types.
allow sufficient time to conduct root cause analysis.
gather meaningful measures, establish trends, calculate
metrics.
16
Golden Rule Number 14
All incidents should be traceable to a Test
Condition and a Requirement.

This is fundamental if you are to 




mitigate all the risks (against requirements).
know you can stop testing.
evaluate data on types of incidents raised against
requirements (or test conditions).
use data to make judgement decisions on likely stability
and appropriate amounts of testing required.
point at the high-risk areas in the production system to
monitor in the first few weeks post implementation.
17
Golden Rule Number 15
Test Assets and Project Histories must
be considered for Re-use.

Project Histories.

previous documentation can





reduce analysis, design and build costs.
allow better understanding of the existing functionality.
allow more accurate project estimates.
enable earlier test preparation .
testware is worth an average 40% of every
project budget.
18
Golden Rule Number 15
Test Assets and Project Histories
must be considered for Re-use.

Re-use of Assets.
 test packs run as regression tests –



provide the status of changes since the previous
release.
can be constructed at all test levels to enable maximum
flexibility for testing large or small projects.
form a baseline from which to commence preparation of
new tests.
19
Golden Rule Number 16
If you can’t measure, you will not know if
what you achieve is good or bad.

Why collect?





measures lead to metrics lead to trends.
metrics form Management Information.
appropriate MI allow informed decisions.
measures make perceptions factual.
Does it matter what you collect?


objective of the measure may vary.
only consistent measures will allow meaningful metrics.
20
Golden Rule Number 17
Measures point to root causes.
Fix the root cause and you improve the
quality .


fix to ensure the error will not recur will reduce costs and
improve quality of delivery.
Warning.



care is required, as it is easy to mistake analysis for the
root cause for a witch-hunt.
the root cause could be anywhere; it is unlikely to be in
the phase in which the incident was identified.
the important thing is to fix it and learn from it, not
apportion any blame.
21
Golden Rule Number 18
Tools can only automate existing process;
they require support.

Are you ready?


A good way to do it!


run tests manually follow by independent automated test
runs - less risk, more control.
The Cost…


Existing process needs to be in a state that allows it to
be automated.
“the cost of the tool is not the cost of the tool”.
Help, I need somebody, help…


user groups, limited (sometimes free) on-site vendor
support.
help from the many consultancies.
22
Golden Rule Number 19
If you don’t ask a 3rd Party Supplier the
question, you may never be told the answer.

Some of the most common questions.






how did you test this package before you handed it over
to us?
do you have any test documentation I could look at?
what errors do you know to be outstanding that are in
this release?
does this release include changes for any other client
and if so, how does it affect us?
what’s the turnaround time for fixing our priority one
problems?
Want more?

71 weighted questions [email protected]
23
Golden Rule Number 20
Sign off Criteria should never be a surprise
to Management.

Review Points.



when criteria are agreed in advance and reviewed at set
points in the life cycle, sign off into Production is the
norm, not the exception.
nothing escapes into production with high-risk caveats or
with insufficient testing.
Who Signs?



list all interfacing managers that need to always be
informed.
recommend if sign off is ‘ok to go’ or ‘what the deviation
is’ from the standard approach.
managers will make proactive decisions rather than have
to wait until it has finished to react.
24
More Questions?
[email protected]
© Andy Redwood 2003-2006
25