Transcript CTG Presentation Template

True stories about
testing based on
experiences
University of Antwerp
Patrice Willemot – Pre sales test consultant
Petra Haldermans - Test Consultant / Test Manager
27/04/2011
CTG - Company overview
Corporate
Headquarters
Buffalo, NY
Founded in 1966
© 2011 CTG, Inc.
2
CTG - Company overview
European
Headquarters
Diegem, Belgium
Founded in 1976
Bertrange, Luxembourg
Reading, United Kingdom
© 2011 CTG, Inc.
3
CTG - Company overview
 Over 3.550 employees worldwide
 350 employees in Belgium
 Quality certification
 ISO 9001, Tick IT
 Beste Werkgever (2003, 2005, 2007, 2009 and 2011)
 Investor in People
© 2011 CTG, Inc.
4
CTG Belgium
© 2011 CTG, Inc.
5
5
Testing unit @ CTG
Market Leader in Belgium
Tought Leadership
• Early Adopter in Software
Testing since 1999
• Hosts Belgium’s largest
Software Test Seminar
• 150 Test Professionals
• 3 times winner of Eurostar
‘Best Paper’ Award
• International speakers
Intellectual Property
• 10+ years of experience
consolidated in IP Tools
• STBoX
We Commit
Strong & Active
Partnerships
• HP Platinum Partner
• 3 times HP Partner of the
Year in 5 years
• Microsoft Gold Partner
Focus on Innovation &
Training
• Prove savings
• CTGLabs
• Search for Improvements
• STAcademy
• Report on the value
• FASTBoX
• Invest in market best practices
• PTBoX
• Share risks & rewards
• Centers of Excellence
• Always finish what we start
© 2011 CTG, Inc.
6
Testing Services @ CTG
Plan
Drive IT
CTG as
Strategic
Partner
Design
Transition
Execute
Optimizing the software and service lifecycle
Test
Maturity
Assessment
Test Practices Improvement
Testing
within reach
Performance Testing
Test
Governance
Test Automation Implementation
Excel within the Lifecycle
© 2011 CTG, Inc.
Training &
Coaching
Do IT
CTG as
Operation
Partner
7
Knowledge Center
STBoX
Essentials
STBoX
Prime
STBoX
Agile
Workflow
Templates
Workflow
Templates
Workflow
Templates
Instructions
Examples
Instructions
Examples
Instructions
Examples
FASTBoX
Best
Practices
Functional
Libraries
PTBoX
Utilities
Best
Practices
Templates
Instructions
Instructions
Examples
Technologies
ST Academy
Training
Knowledge Base
Wiki
© 2011 CTG, Inc.
Center
Centers of
Excellence
Knowledge
sharing
CTG Labs
Annual
Convention
White Papers
8
© 2011 CTG, Inc.
9
© 2011 CTG, Inc.
10
© 2011 CTG, Inc.
11
© 2011 CTG, Inc.
12
© 2011 CTG, Inc.
13
© 2011 CTG, Inc.
14
© 2011 CTG, Inc.
15
© 2011 CTG, Inc.
16
© 2011 CTG, Inc.
17
© 2011 CTG, Inc.
18
© 2011 CTG, Inc.
19
© 2011 CTG, Inc.
20
© 2011 CTG, Inc.
21
© 2011 CTG, Inc.
22
True stories about testing based on
experiences
• 3 Dilemmas
– V-model
– Test Plan and risks
– Test Design and Execution
• 1 How to
– Implementation of a test process
© 2011 CTG, Inc.
23
Customer Case Company A: introduction
• Assignment:
– UAT Testing + Test Management
– Test Governance
• Difficulties:
– Different parties:
• Requirements definition: “Company A”
• Development & Testing (except UAT): Company B
• UAT Testing: “Company A” & CTG
– Meetings via Skype (companies in Belgium and the Netherlands)
© 2011 CTG, Inc.
24
Customer Case Company B: introduction
• Assignment
– Test Management
– System Testing, Acceptance Testing
– Test Automation
• Difficulties
– Distributed test teams (Trier, Paris, Gent, Bordeaux, Vienna)
– Different Test approaches
– Different development approaches (Scrum, V-model, Waterfall model)
– Different level of test maturity (very low, low, medium, high)
© 2011 CTG, Inc.
25
Dilemma 1: V-model
• No business users available for Acceptance
Testing
• A company not familiar with the V-model
• No distinction between for example
Component Testing and Acceptance Testing
© 2011 CTG, Inc.
26
Dilemma 1: The theory
Responsibility:
Business
User
Requirements
System
Requirements
Acceptance
Testing
Responsibility:
IT Project Team
Global Design
Detailed Design
System
Integration
Testing
System
Testing
Component
Integration
Testing
Component
Testing
Code/Build
© 2011 CTG, Inc.
27
Dilemma 1: The theory
Test Basis for
User
Requirements
Acceptance
Testing
System
Integration
Testing
System
Requirements
System
Testing
1
1’
2
Global Design
Detailed Design
Component
Integration
Testing
Component
Testing
Code/Build
© 2011 CTG, Inc.
28
Dilemma 1: The solution
• Goal:
– Increase business knowledge of the tester
– Change the mindset of the end user
• How:
– Combination of end users and testers
– Organize guided user acceptance test sessions
– Split test tasks between testers and end users
• Test Design on business level by IT Testers
• Review test cases by business
• Test Execution by business
– Promote early moment of involvement of testing
• Conclusion:
– Think about possible risks and workarounds during test planning!
© 2011 CTG, Inc.
29
Dilemma 1: The solution
Responsibility:
Business
User
Requirements
System
Requirements
Acceptance
Testing
Responsibility:
IT Project Team
Global Design
Detailed Design
System
Integration
Testing
System
Testing
Component
Integration
Testing
Component
Testing
Code/Build
© 2011 CTG, Inc.
30
Customer Case: “Company A”
• Testing done in NL:
– Unit Testing
– Unit Integration Testing
– System Testing
• Testing done in BE:
– User Acceptance Testing, by both CTG and “Company A” (business
users)
• Test Design by CTG
• Review by “Company A” (business users)
• Execution by CTG and “Company A”
© 2011 CTG, Inc.
31
Customer Case: “Company A”
Responsibility:
Business
User
Requirements
System
Requirements
CTG
Responsibility:
IT Project Team
Acceptance
Testing Company
A
System
Integration
Testing
System
Testing
Component
Integration
Testing
Global Design
Detailed Design
Component
Testing
Company B
(Netherlands)
Code/Build
© 2011 CTG, Inc.
32
Customer Case: “Company B”
• Phase 1:
– CTG responsible for System Integration and Acceptance Testing
• Test Design and Test Execution
• Phase 2:
– CTG responsible for System Integration Testing
• Test Design and Test Execution
– “Company B” responsible for Acceptance Testing
• Test Execution of test cases created by CTG
© 2011 CTG, Inc.
33
Customer Case: “Company B”
• Phase 3:
– CTG responsible for testing during a Scrum Sprint
• Test Design and Test Execution
– “Company B” responsible for Acceptance Testing
• Test Execution of test cases created by CTG
• Phase 4:
– CTG responsible for testing during a Scrum Sprint
• Test Design and Test Execution
– CTG responsible for testing the integration of different Scrum Sprints
(Scrum of Scrum)
– “Company B” responsible for Acceptance Testing
• Test Execution of test cases created by CTG
© 2011 CTG, Inc.
34
Customer Case: “Company B”
Phase 1
Phase 2
© 2011 CTG, Inc.
35
Customer Case: “Company B”
Phase 3
Acceptance
Testing
© 2011 CTG, Inc.
36
Customer Case: “Company B”
Phase 4
Acceptance
Testing
© 2011 CTG, Inc.
37
Dilemma 2: Test Plans and Risks
• The client doesn’t want to have a complete
Test Plan
• A Test Plan: what do I need to add?
• How to deal with risks? We never thought
about them
• We have a Test Plan, but nobody reads it
© 2011 CTG, Inc.
38
Dilemma 2: The theory
Important things to consider (Risks & Requirements Based Testing):
– product risks & requirements  Features To Test
– project risks
– test levels & quality attributes test types
– priorities
– constraints:
• available resources
– people & expertise
– infrastructure
– tools & techniques
• time & budget
© 2011 CTG, Inc.
39
Dilemma 2: The theory
© 2011 CTG, Inc.
40
Dilemma 2: The theory
IEEE 829 - Template
• content – 16 points
– test plan information
– introduction
– test Items
– features to be tested
– features not to be tested
– approach
– item pass / fail criteria
– suspension and resumption criteria
© 2011 CTG, Inc.
–
–
–
–
–
–
–
–
test deliverables
testing tasks
environmental needs
responsibilities
staffing and training needs
schedule
risks and contingencies
approvals
41
Dilemma 2: The solution
• Goal:
– Create awareness & pay attention to the importance of a Test Plan
• How:
– Organize a workshop about the content of the Test Plan (needs)
– Pay attention to the consequences in case no Test Plan is agreed upon
– Check consistency with Project Plan (e.g. escalations procedures)
– Rely on the Test Policy
– Think about the type of project (Agile / classic development)
• Conclusion:
– Convince the client that a Test Plan has a lot of advantages, even when a
company is used to work in the same way for each project
– Reach consensus with a list of known risks
© 2011 CTG, Inc.
42
Customer Case: “Company A”
• A full version Test Plan with extra attention on:
– Suspension & Resumption criteria
– Test Project Risks
– Objectives per Test Level (and thus per company)
– Entry & Exit criteria between UAT Testing and the other Test Levels
– Description of the UAT process, with pre-defined milestones / quality
gates
• Strengths and must haves:
– Penalties for the Risks
– Formal agreement of the document
© 2011 CTG, Inc.
43
Customer Case: “Company B”
– A full version of a test plan has been created based on the “Company B” template
•
•
•
•
•
Test Items
Scope
Test Approach
Organization
Test Deliverables
•
•
•
•
•
Environmental needs
Staffing and training needs
Risks and contingencies
Agreements
Glossary
– Test plan has never been reviewed or read by stakeholders
– Test plan has been ignored during the test project
– New template of the test plan has been created
• Process approach
• Roles and Responsibilities
• Environmental needs
© 2011 CTG, Inc.
44
Dilemma 3: Test Design & Execution
• We only want to do Exploratory Testing
• My team has a lot of knowledge about Test
Design Techniques
• I am used to work with business users, but my
new test team is not able to execute the
existing test cases… what should be my level
of detail?
© 2011 CTG, Inc.
45
Dilemma 3: The theory
Input
Process
Expected Result
- High-level test case describes the input data and the predicted result on an
abstract level. It can contains one or more low level test cases.
- Example: A man age > 35 with >2 children
- Low-level test case, the abstract values that were assigned to the input an
predicted output of the high level test case, are replaced by concrete values.
- Example: Dirk Honda, 70 years, 4 children
© 2011 CTG, Inc.
46
Dilemma 3: The theory
• Test Design Techniques:
– Line coverage
– Equivalence Partitioning
– Statement coverage
– Boundary Value Analysis
– Decision coverage (= Branch coverage) – Algorithm Test
– Condition coverage
– Decision Table Test
– Decision / Condition coverage
– Syntactic Test
– Multiple Condition coverage
– Semantic Test
– Condition Determination coverage
– Elementary Comparison Test (ECT)
– Real-life Test
– Data Cycle Test / Entity Life Cycle Test
– Error Guessing
– Process Cycle Test
– Random Test
– Program Interface Test
– Idiot Proofing
© 2011 CTG, Inc.
47
Dilemma 3: The theory
Exploratory testing is simultaneous
learning, test design, and test execution
• An interactive test process
• Using the information gained while testing to design new and better tests
• A formal process defined
– tasks, objectives and deliverables  Test Sessions and charts
• Testers have the skills to listen, read, think and report rigorously and effectively
© 2011 CTG, Inc.
48
Dilemma 3: The solution
• Goal:
– Define the “right” fit for the company
• How:
– Learn from experiences; what went OK, what went wrong in the past?
– Define advantages & disadvantages of high / low level test cases
– Think about which and whether you want to oblige Test Design
Techniques
– Find out the knowledge of your test team
• Conclusion:
– Define the Test Strategy based on the needs and people within your
test team
© 2011 CTG, Inc.
49
Customer Case: “Company A”
• Unit Testing: no written test cases
• Unit Integration Testing: no written test cases
• System Testing: written test cases in Excel; titles imported in Microsoft
Team Foundation Server 2010
• User Acceptance Testing: high level written test cases in TFS
– Written (Use Case Testing) by CTG and reviewed by the business
users of Company A
– A list of needed test data created and finally defined by Company A
– Extra: error guessing done (not structured planned)
• Pitfall: all testing based on the same Use Case Specification Document,
with defined objectives per test level
© 2011 CTG, Inc.
50
Customer Case: “Company B”
At the start
• System-integration test cases based on requirements on the back of a
napkin
• High level test cases with high level information (no click on button, …)
• Acceptance test cases based on requirements on the back of a napkin
• High level test cases with high level information (no click on button, …)
Evolved to
• Acceptance test cases based on requirements on the back of a napkin
• High level test cases with high level information used by the end users
• Scrum test cases based user stories
• Only a title, no steps
• Uncontrolled exploratory testing
• Test Automation based on user stories
© 2011 CTG, Inc.
51
True stories about testing based on
experiences
• 3 Dilemmas
– V-model
– Test Plan and risks
– Test Design and Execution
• 1 How to
– Implementation of a test process
© 2011 CTG, Inc.
52
Implementation of a test process: A best
practices approach
• 2 main phases
– Test Maturity Assessment
Test Maturity
Assessment
Test Practices
Improvement
– Test Practices Improvement
Test
Maturity
Assessment
Test Maturity
Assessment
© 2011 CTG, Inc.
Test Practices
Improvement
Test Practices
Improvement
53
Test Maturity Assessment
Identification
Assessment
AS IS
TO BE
•
•
•
•
Interview
Metrics
Documentation
Determine SWOT
People
Process
•
•
•
•
Workshop
Based on STBoXTM
Increase S and O
Eleminate W and T
Technology
Improvements
Business cases
Project cards
Roadmap
Improvements roadmap
© 2011 CTG, Inc.
54
From improvement goals to roadmap
Identification
TO BE
Improvement
goal 1
Improvement
goal 2
Improvement
goal 3
© 2011 CTG, Inc.
Improvements
Improvement action 1
Improvement action 2
Improvement action 3
Improvement action 4
Improvement action 5
Improvement action 6
Improvement action 7
Improvement action 8
Improvements
Improvement
project 1
Improvement
project 2
Quick Win
55
Test Maturity Assessment
SMALL
MEDIUM
LARGE
+/- 6 activities
Informal
1 formal deliverable
+/- 12 activities
Formal
+/- 10 formal
deliverables
+/- 22 activities
Formal
+/- 14 formal
deliverables
Quickscan
Regular
Profound
Size of the assessment depends on
# departments
# languages
# test types
# interviews
# test levels
Size of organization
© 2011 CTG, Inc.
Scope
• People
• Process
• Technology
56
Test Practices Improvement
• Outcome of the assessment
Quality Roadmap
• Business cases
• Project Cards
• Reworked roadmap
• Re prioritization
• Life long learning
• Continuous improvement
Implementation
• Prioritised roadmap
• People
• Process
• Technology
Retrospective
• Test Maturity (Re-)Assessment
• Self Assessment
© 2011 CTG, Inc.
57
[[email protected]]
[[email protected]]
QUESTIONS AND ANSWERS
http://www.ctg.com/
© 2011 CTG, Inc.
http://jobs.ctg.eu/
58