Transcript Lightning Talks - Excelon Development

Lightning Talks
Presented at STAREast 2005
By Matt Heusser … and the gang
[email protected]
Pamela Wagner
CUNA Mutual Group
Risk Analysis in five minutes
[email protected]
Simple RA Form
Requirement or issue
Likelihood of
failure
Impact of failure
I need to walk down the hall
without tripping.
High
Medium
Think about your experiences, use old bug reports, trust your
gut!
CONTEXT CHANGES EVERYTHING!
Instead of a hall- the red carpet
at the Oscars where I am
nominated for Best Actress,
Director, Producer and
Screenwriter.
High
HIGHEST
POSSIBLE!
Jennifer Brendlinger
Concurrent Technologies Corporation
CMMI for testing: A critical look
[email protected]
What is CMMI?

Capability Maturity Model (CMM) Integration




Disciplines – Bodies of knowledge



Serve as a guide for improvement of organizational processes
http://www.sei.cmu.edu/cmmi/
March 2003 – Concurrent Technologies Corporation (CTC) was
appraised at Maturity Level 3 of the CMMI-SE/SW, Version 1.1,
Staged Representation – http://www.ctc.com
Systems Engineering – Development of total systems
Software Engineering – Development of software systems
Staged Representation

Proven sequence of improvements, each serving as a
foundation for the next
Maturity Levels

Maturity Level 1: Initial


Maturity Level 2: Managed


Processes are well defined and understood and are described in
standards, procedures, tools and methods.
Maturity Level 4: Quantitatively Managed


Requirements are managed, and processes are planned, performed,
measured and controlled.
Maturity Level 3: Defined


Processes are ad hoc and chaotic. Success depends on heroics of the
people, not on the use of proven processes.
Selected subprocesses are controlled using statistical and other
quantitative techniques.
Maturity Level 5: Optimizing

Processes are continually improved based on a quantitative understanding
of the common causes of variation inherent in processes.
Organizational Processes

Process Areas

Related practices that satisfy a set of goals



Specific Goals (SG)


Describe what must be implemented to satisfy the
process area
Specific Practices (SP)


Verification – Ensure that selected work products meet their specified
requirements
Validation – Demonstrate that a product or product component fulfills
its intended use when placed in its intended environment
Activities performed to achieve a specific goal
Generic Goals (GG)

Enable the organization to institutionalize best practices
Verification & Validation

Verification




Validation



SG1 – Preparation for verification is conducted.
SG2 – Peer reviews are performed on selected work products.
SG3 – Selected work products are verified against their
specified requirements.
SG1 – Preparation for validation is conducted.
SG2 – The product or product components are validated to
ensure that they are suitable for use in their intended operating
environment.
GG3 – The process is institutionalized as a managed
process.

Company processes and procedures
Work Products Used to Enhance
Testing and Comply with CMMI











Verification and Validation (V&V) Plan
Operations & Maintenance (O&M) Manual
System Test Procedures
Requirements Traceability Matrix
Discrepancy Reports
Change Requests
Test Run Logs
Peer Review Reports
Meeting Minutes
Trip Reports
Metrics
Bottom Line - Benefits of CMMI

For the Testing Process



For the Project


Established plans to follow
“Enforced” cooperation, dialog, reporting and follow-up
Quality work product that meets its requirements and
fulfills its intended use
For the Organization



Everyone buys in to the processes and procedures
No grassroots efforts
No chaos
Questions
Jennifer C. Brendlinger
Software Test Lead
Concurrent Technologies Corporation
[email protected]
Reversing the
Outsource/Offshore Trend
with Technology
Rick Vatcher
CEO, AccordSQA
The Promise – The Truth
• SQA Outsourcing Saves Money
– Outsourcing Yields Poor Results (May 4, 2005
USATODAY)
– 70% had a bad experience and 44% didn’t
save money.
• SQA skilled workforce more plentiful in
developing countries
– India experiencing skilled labor crisis (Ostfold SW)
Take a Look at the SQA Industry
• SQA Tools developed by SQA
Professionals
• Automated, Easy to use, Easy to learn,
LESS TEDIOUS
• Knowledge based tools replacing Manual
Repetition
• Business Process Part of Process –
Elevating Roles
• Integral Part of the Development Process
Next-Generation SQA Accelerate the
Process – Eliminates Offshore Gains
3500
3000
MAN-HOURS
Next
Generation
Technology
2nd
Generation
Tool
Test
Development *
480
3000
Maintenance
Cycles
50
150
Costs @
$40/Hour
$21.2K
$126.0K
2500
2000
1500
1000
500
0
Next
Generation
Test Maintenance
Test Creation
2nd
Generation
•1500 test cases/scripts (2 hours per script) equivalent to
45 SmarteScript test scenarios (10.5 hours per scenario)
83% Less Cost! - 83% Less Time!
Software Tool Advancements Driving
Best Practices in Resource Allocation
•Next-Gen Tools Close the Gap
–Save money
–Speed Time-to-Market
–Elevate Resources
…Ultimately Reversing Outsourcing
Trend
Sheryl Stockless
The Mathworks, Inc.
Moving Quality Engineering Up-Front
[email protected]
Verification
New bug
(Customer, internal)
Team Triage
Need more info
Not a bug
Close
(Eng, QE, Doc, Support)
(QE)
Submit bug fix
(Eng)
Pass
Verify bug fix
Fail
(QE)
Add a test
(QE)
Close
(QE)
Sheryl Stockless, The MathWorks, Inc.
Star East 2005
Validation and
Verification
New bug
(Customer, internal)
Need more info
Validate bug
(QE)
Team Triage
Not a bug
Close
(Eng, QE, Doc, Support)
(QE)
Submit bug fix
(Eng)
Pass
Verify bug fix
Fail
(QE)
Close
(QE)
Sheryl Stockless, The MathWorks, Inc.
Star East 2005
Jennifer Bonine & Marie George
Express Scripts
Sarbanes-Oxley in the testing shop
[email protected]
What Any QA department Need to
meet SOX Requirements
• Defined Control Points
–
–
–
–
–
Requirements
Test Plan
Test Cases
Go/No Go document
Implementation Plan
• Processes around the Control Points
– Don’t over process
• Accountability and Traceability
– Sign off and acceptance from required stakeholders
– Traceability of Control Points, acceptance to projects/releases
*Applicable for anyone who is testing applications to get involved and understand audit
requirements
Automated Testing using
Distributed Computing
Darshan Domah
Senior Software Test Engineer
Vital Images Inc.
5/19/2005
STAREAST 2005
Distributed Test Environment
DTE
tclient/tserver
Tcl
•Set up Test
•Start Test
•Monitor Test
Python
•Results Reporting
WinRunner
•Drive GUI of Application
•Uses Analog Mode
Overview of tclient/tserver
•
•
•
•
2 Tier Client/Server
TCP/IP and Sockets
Multiple servers, dedicated ports
Used to:
–
–
–
–
–
reset box
bring SW
execute queue of tests
monitor tasks
get results
• Some Commands
– Start application remotely
• tclsh83 tclient.tcl –c –bgnd –h HOST1 vitrea.exe
– Start a winrunner test remotely
• tclsh83 tclient.tcl –c –wrun –n e:/Orientation –i 2 –t 3600 –h HOST5 –o
z:/orientation/testresults
Overview of Application
• 3D Visualization for medical imaging-Vitrea
• Uses: diagnosis, surgery planning, post operation analysis
CT and MRI data
from scanners
Vitrea
interprets
data
Recreates 2D
images and 3D
volumes of anatomy
Medical personnel (Radiologists & Physicians)
use software capabilities to analyze rendered images,
make diagnosis and create reports
– Testing
• Patient Orientation
• Reliability Uptime
Communicating the Value of
Exploratory Testing
David Gilbert
President
Sirius Software Quality Associates
Exploratory Testing IS High Value
• Due to lack of co-dependencies, it can begin
much earlier.
–
–
–
–
No requirements needed
Full functionality not needed
Can work around instabilities
Formal Stubs and Harnesses not always needed
Exploratory Testing IS High Value
• Due to flexibility, it should have the highest
defect to time testing ratio of any other testing
process
– Not business process constrained
– Looks for the cause and classification of defects, not
just the repeatable specific case
But Value is Relative…
• …and based on PERCEPTION!
– A painting from your child is very valuable to you.
– The Mona Lisa has value universally.
• What is the difference?
• Quality of Presentation!
Why is Automation Held in Such
High Esteem?
• Even though we all know many automation
efforts do not successfully test their projects
well, they are perceived as
–
–
–
–
–
–
Organized
Structured
Professional
Repeatable
Reportable
Manageable
We Must Present Our Results
Professionally
• To be taken seriously and given respect, we
must meet the same level of professional
management and presentation.
• This can be VERY labor intensive, using
–
–
–
–
–
–
ScreenGrabbers
Spreadsheets
HomeGrown Databases
Word Processors
Lots of cut and paste
Reams of paper for printing results
We Must Present Our Results
Professionally
• Or use a tool, with features such as …
–
–
–
–
–
–
Keylogging
Screenshots
Video
Organized Test Cases
Integrated Issue Management
Graphs and Reports
Anibal Sousa
Microsoft
Too many bugs to fix, too little time …
[email protected]
Motivation
• 5000+ bugs during product development life cycle
• Subjective and not deterministic process for triaging
bugs
• Testers were confused about what kind of bugs were still
being accepted – they wanted to be able to focus on the
right set of problems to investigate
• Bad for morale and team engagement
• Bugs will occur through out the whole project
• Quality should go up and not down
• Risk of regression gets higher with time
• Not all features get ready at the same time
• Aspects to be considered: cost, risk, impact to teams
(test, UA, etc.)
Proposal
Priority
Name
Definition
1
Wish
These might not be even considered bugs.
2
Improvement
This bug is hard to find, not noticeable, causes minor problems (or none)
and can be ignored.
3
Bug
This is a bug, but might be obscure, rare or have small impact. Normally it
has easy workaround.
4
KB
This bug might be very visible and affect functionality significantly. In case it
has a workaround, it is not obvious or simple.
5
PSS
This bug will cause Support calls. Publishing KB article is not enough, since
workaround may not exist or be too complicated.
6
QFE
Serious bug. If a customer finds this bug, we will have to issue a QFE.
Ship Stopper
Reasonable region of a feature is not working as expected because of the
bug, and there is no workaround. We can not ship the product with this bug
active.
7
•
•
•
•
•
Rules of game clear and public to all team members
Bug is opened, field is set in the bug database by tester, might be updated by
developer, feature team decide on it, escalating to Management team if necessary
Bug bar considers: feature status, moment in project schedule and bug details, like
scope, risk, cost, etc.
Triage process and sets right priority for all accepted bugs accordingly
Clear communication of results
Example
7
Wish
6
Improvement
X
5
Bug
X
4
KB
3
PSS
2
QFE
X
X
1
Ship Stopper
X
Areas
5
Bug
4
KB
3
PSS
2
QFE
1
Ship Stopper
Areas
3
PSS
2
QFE
1
Ship Stopper
Areas
Reports
X
Reports
X
Reports
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Import
Export
Forms
X
X
Import
Export
Forms
X
X
Import
Export
Forms
Performance
SBA
Integration
PDA
User
Assistance
X
X
X
X
X
X
X
X
X
X
X
X
X
Performance
X
Performance
SBA
Integration
PDA
PDA
X
X
X
X
SBA
Integration
User
Assistance
X
User
Assistance
After …
• Team engagement and commitment – high
morale
• Number of regressions (metrics) went
down significantly
• Number of opened bugs did not go down,
but accepted bugs did
• Product was shipped in timely manner
Questions?
• Anibal Sousa – [email protected] or
[email protected]
In the Backroom,
No One Can Hear You Scream
Harry Robinson, Google
[email protected]
Does Your Boss’s Boss Know Your Value?
Sure, Finding Bugs is Important
But, I bet your work could help others.
And helping them will help you!
Publicizing Your Contribution is Important
And the pen is mightier than the bug report.
So Start. Write Now.
• Start a blog.
• Post a paper to Stickyminds.
• Give brown bag presentations to your team.
• Offer to speak at local QA group meetings.
• Send in an abstract for STAR West 2005!
Here’s A Poem To Inspire You!
The codfish lays ten thousand eggs,
The homely hen lays one.
The codfish never cackles
To tell you what she's done.
And so we scorn the codfish,
While the humble hen we prize,
Which only goes to show you
That it pays to advertise.
- Anonymous
Lightning talks will be at the better software
conference.
Call for presentations!
http://www.sqe.com/lightningtalks.asp