Transcript Role of the Tester in Agile Teams

How Agile Ended Our Defect
Report-Fix-Check-Rework Cycle
Richard Leavitt
Rally Software Development
[email protected]
Copyright 2003-2005, Rally Software Development Corp
Objectives for Today
Observation:
∙ Traditional workflows and project metrics evolved to
manage large defect inflows during long testing
phases
Question:
∙ How relevant are these management processes in
dramatically shorter development cycles?
Agenda:
∙ Defect Workflow & Metrics in Test-Last
∙ Our Agile Transition
∙ New Dev & Test Processes, Workflows and Reports
∙ Benefits and Bruises
Slide 2
Copyright 2003-2005, Rally Software Development Corp
Agile Doesn’t Change Our Goals
∙ Customers judge our software as high value
and high quality
∙ Our business judges our efforts as timely and
providing high ROI
∙ We like our jobs
Slide 3
Copyright 2003-2005, Rally Software Development Corp
Agile Doesn’t Change Our Questions
We desire:
∙ A small set of useful and valid measures,
provided by
∙ Standard reports that everyone anticipates
and understands, with
∙ The ability to drill down and roll up
Readiness
When can
we release?
Slide 4
Progress
What’s
blocking?
Copyright 2003-2005, Rally Software Development Corp
Status
How’s feature X
doing?
Before: Phased Development
Process
Advantages:
• Requirements and design
“up front”
• Logical
• Prescriptive
• Plan based
Disadvantages:
• Requirements “freeze” paradigm
• Dreaded system integration phase
• High defect counts and “surprises”
• Hard to adapt to changing requirements
• Plan based
Slide 5
Copyright 2003-2005, Rally Software Development Corp
One of My Test-Last Environments
∙ Release size: 16 FTEs x 14 Months
∙ Detailed work breakdown with task complete
dates
∙ Unit testing? What’s that?
∙ QA drops with high bug counts (100’s)
∙ Islands of information
– Project plan
– Requirement docs
(MRD->PRD->SRS)
– Test cases in Excel
– Build results
Slide 6
– Test effort & results
– Traceability matrix
– Defect/Issue/Request
lists and metrics
Copyright 2003-2005, Rally Software Development Corp
High Defect Counts Add Weight
∙ “Push” workflow
systems with
complex rule engines
for routing & signoffs
∙ Fine-grained
permissions to
control ownership
and changes
∙ Lengthy bug reviews
& management
burdens
∙ DELAY!
Slide 7
Copyright 2003-2005, Rally Software Development Corp
But Defects are Easy to Count &
Graph!
Defect Submit & Close Rates
700
600
Defect Count
Statistical gymnastics
∙ Defect rates
∙ Densities, cycle times
∙ By phase
∙ By owner, type,
status, severity,
module, how found…
∙ In general, lots of
totals, percentages,
averages and ratios
500
400
300
200
100
0
0
10
20
30
40
50
60
70
80
Days Since Test Start
Slide 8
Copyright 2003-2005, Rally Software Development Corp
90
100 110
And how else can we answer our
questions?
After all, our typical experience goes like this:
“I’ve never heard of any direct measure of
project status, program readiness for release,
or progress toward meeting milestones, etc.
Instead, we’ve used surrogate measures and
metrics.”
The Darker Side of Metrics
Douglas Hoffman, Software Quality Methods, LLC
Slide 9
Copyright 2003-2005, Rally Software Development Corp
Problems with Defect Metrics
Readiness
When can
we release?
Defect Submit & Close Rates
700
QA Drops
Defect Count
600
500
400
300
200
100
0
0
10
20
30
40
50
60
70
80
Days Since Test Start
Slide 10
Copyright 2003-2005, Rally Software Development Corp
90
100 110
Problems with Defect Metrics
Readiness
When can
we release?
Defect Submit & Close Rates
700
Defect Count
600
Bug Review
Meetings
500
400
Defects “closed”
as
300
• Deferred 200
• Duplicate 100
• Low priority0
• Can’t Reproduce
0
10 20 30
• Enhancement Request
Slide 11
Close Rate2
Close Rate1
40
50
60
70
80
Days Since Test Start
Copyright 2003-2005, Rally Software Development Corp
90
100 110
So, Phased Development Processes
Encourage a Defect-Driven Approach
And its accompanying problems:
∙ Large defect counts with heavy process
burdens, high management effort and delay
∙ Defect-based metrics that don’t relate well to
the real questions we’re trying to answer
₋ All metrics programs cause unintended side
effects on our behaviors, but can we do better?
Slide 12
Copyright 2003-2005, Rally Software Development Corp
Why Agile Development?
∙ Agile promotes rapid delivery and feedback
over elaborate upfront planning
Short iterations give all stakeholders timely and regular
visibility into readiness, status and progress
∙ Shared commitments simplify workflows…
but everyone needs real-time status information!
∙ Agile handles change better
₋ Accepts late-binding requirements
₋ Continuous integration and early testing means less
“integration hell” and associated defects & surprises
In Short
Minimum Process, Maximum Value
Slide 13
Copyright 2003-2005, Rally Software Development Corp
The Agile Paradigm Shift
Waterfall
Agile
Cost
Requirements
Schedule
Constraints
Value /Vision
Driven
Plan
Driven
Estimates
Cost
Schedule
The Plan drives cost & schedule
Slide 14
Copyright 2003-2005, Rally Software Development Corp
Features
The Vision drives features
Moving to Iterative & Incremental
Agile Development
Waterfall
Iterative
Iterative and
Incremental
Critical Path
through
Phases
Critical
Drop
Schedule
Requirements
& Design
Freeze &
Signoff
Manage
Scope
Creep
Detailed
Design
Inside
Iteration
Development
Process
All Features
in Parallel
Multiple
Drops to
QA
Increment
at a time
Test & Fix
Process
Last Phase
Only
“Test
What’s
Working”
Acceptance
Test Inside
the Iteration
Project
Mgmt
Slide 15
Time Boxes
Copyright 2003-2005, Rally Software Development Corp
Parallel
Acceptance
Test Driven
Agile Team Practices
∙ Time box everything
∙ Two levels of planning: rough for releases,
fine-grained for iterations
∙ Just-in-time requirements elaboration
∙ Focus on the highest priorities
∙ Early and continuous testing
∙ Try to remove roadblocks and defects
immediately
Slide
Slide16
2
Copyright 2003-2005, Rally Software Development Corp
The New Planning Process
∙ Release cycle: 8 or 10 weeks (4 or 5 iterations)
₋ Prod Mgmt defines themes & features/stories
₋ Dev provides rough “planning” estimates
₋ Release Backlog is then negotiated & prioritized
₋ “Hardening” iteration at the end tackles technical
debt
∙ Iteration cycle: 2 weeks with one ½ day
planning meeting
₋ Recap previous iteration
₋ PM presents new and elaborated stories & use cases
₋ Dev details tasks & estimates just-in-time
₋ Priorities are set and the team commits
Slide 17
Copyright 2003-2005, Rally Software Development Corp
Story Acceptance is Common Goal
∙ One schedule, one budget, one team,
but independent dev & test people and tasks
∙ Features are squeezed, NOT testing and
schedule
∙ Dev codes highest priority requirements all
the way to acceptance
∙ QA elaborates & automates acceptance tests
in parallel with coding
₋ Dev-QA collaboration versus separation improves
₋
Slide 18
code robustness and testability right away.
Quality no longer “tested in”
Copyright 2003-2005, Rally Software Development Corp
Early & Continuous Testing Drives
Down Defect Counts
∙ Unit tests run on code check-in
∙ Nightly build runs automated acceptance
tests
∙ Team tries to resolve failures immediately
In Short,
We Stay Close To Shippable Code
∙ When combined with short iterations, we get
∙ Tiny defect counts (dozens, max) with little
management burden and team overhead
Slide 19
Copyright 2003-2005, Rally Software Development Corp
Iteration Decisions & Metrics
Meeting iteration intent and schedule means
responding to changing realities
∙ Should we cut or adjust any features?
₋ What progress have we made on each story?
₋ How much is there left todo?
₋ What’s our remaining budget?
∙ What’s standing in our way of acceptance?
₋ Blocks?
₋ Which story card tests are failing?
₋ Are any defects open against our iteration stories?
Slide 20
Copyright 2003-2005, Rally Software Development Corp
New Iteration Status Views
Readiness
When can
we release?
4
Status
How’s feature X
doing?
Progress
What’s
blocking?
Slide 21
Copyright 2003-2005, Rally Software Development Corp
2
Tie Acceptance Test Results To
Story
Progress
What’s
blocking?
Slide 22
Copyright 2003-2005, Rally Software Development Corp
So, Failing Tests Replace Many Defects for
Simple, Pull-based Workflow
∙ Dev and Test now communicate via failing
tests instead of routing defects
Test-driven Advantages
₋ Failures are easy to reproduce
₋ Failures are transient,
₋
₋
Slide 23
less management overhead
We’re done when the tests pass
Lighter process with less delay
Copyright 2003-2005, Rally Software Development Corp
Summary:
Instead of Focus on Defects
Defect Submit & Close Rates
700
Defect Count
600
500
400
300
200
100
0
0
10
20
30
40
50
60
70
80
Days Since Test Start
Slide 24
Copyright 2003-2005, Rally Software Development Corp
90 100 110
We focus on Acceptance
Defect Submit & Close Rates
700
Defect Count
600
500
400
300
200
100
0
0
10
20
30
40
50
60
70
80
Days Since Test Start
Slide 25
Copyright 2003-2005, Rally Software Development Corp
90 100 110
Summary: Instead of Complex,
Push-based Workflow
Slide 26
Copyright 2003-2005, Rally Software Development Corp
We use Simpler, Pull-based
Workflows
Slide 27
Copyright 2003-2005, Rally Software Development Corp
Benefits and Bruises
∙ Agile has dramatically changed our project planning,
tracking and workflow. We are more responsive to
new requests and late changes.
>Everyone needed education on paradigm shift
∙ Everyone focused on incremental, meaningful
functionality, not defect find/assign/fix/regress
>Demands high collaboration and real-time status.
>Upstream analysts must provide steady flow of
business needs and priorities
Slide 28
Copyright 2003-2005, Rally Software Development Corp
More Benefits and Bruises
∙ Time boxes mean we always know when
we’re going to release
>Though the scope may be slightly different!
∙ QA plays large role in requirement
elaboration and defect prevention.
>Early resistance to TDD –
required a change agent (Mike Cohn)
>Struggles to automate tests inside iteration
>New skills required in test design and automation.
Slide 29
Copyright 2003-2005, Rally Software Development Corp
More Benefits and Bruises
∙ Tiny bug lists (dozens) with little
management overhead
>Most “bugs” found during exploratory testing in
hardening iterations.
>Re-factoring tests is critical to keep up velocity.
∙ Simple, pull-based workflow with team
members actively grabbing the next most
important item off the stack
>Need culture of shared ownership
that “takes responsibility”
Slide 30
Copyright 2003-2005, Rally Software Development Corp
Suggestions for Getting Ready
∙ Suggested Process
₋
₋
₋
₋
Brown bag discussions
Guest lectures & webinars
Local Agile/XP user groups
Agile Conferences
∙ Agile2005 – Denver, July
05
∙ Where to start on the
web
₋
₋
Slide 31
Agile Alliance Web Site
www.agilealliance.org
Rally Ecosystem Pages
www.rallydev.com/ecosyst
em
∙ Suggested Reading
₋
₋
₋
₋
₋
Agile Project Management
w/ SCRUM
www.controlchaos.com/
Agile & Iterative
Development
www.craiglarman.com
Lean Software
Development
www.poppendieck.com/
Agile Project Management
www.jimhighsmith.com/
Tactical Management of
Iterative Development
www.rallydev.com
Copyright 2003-2005, Rally Software Development Corp
Questions?
Slide 32
Copyright 2003-2005, Rally Software Development Corp
Acknowledgments
∙ The Darker Side of Metrics, Douglas Hoffman,
PNSQC 2000
∙ It’s Not About the Bugs, Harry Robinson,
Stickyminds 2003
∙ Managing Software Requirements: A Use Case
Approach, Dean Leffingwell, 2003
∙ The Test Manager at the Project Status
Meeting, Brian Marick, Steve Stukenborg,
1997
Slide 33
Copyright 2003-2005, Rally Software Development Corp