A Day in the Life of a Tester

Irinel Crivat SDET Technical Lead ASP.NET

http://www.asp.net

1

Overview



The purpose of testing a program is to find problems in it



What a tester does?

  

Design test case (automated & manual) Find problems is the core of a tester work Write problem reports 2

What are we testing?

Writing dynamic, high performance Web applications has never been easier ASP.NET helps you deliver real world Web applications in record time.

ASP.NET combines developer productivity with performance, reliability, and deployment.

3

What are we testing?

Improved Performance and Scalability

  

Compiled execution Rich output caching Web-Farm Session State Enhanced Reliability



Memory Leak, DeadLock and Crash Protection

  

Developer Productivity Easy Programming Model Flexible Language Options Great Tool Support



Rich Class Framework

  

Easy Deployment "No touch" application deployment Dynamic update of running application Easy Migration Path

 

New Application Models XML Web Services Mobile Web Device Support 4

Feature Crews

 Feature Crew

“A small self-organizing, cross-discipline group that works together to completely deliver a discrete Feature”.

   

Crew has flexibility to decide how to work, how to meet quality bar Feature: Independently testable functionality taking 1-6 weeks of elapsed time to complete Crew: 1PM, 1-5 Dev, 1-5 Test, other disciplines as appropriate No remaining work is left to do on a feature when it gets checked in



interim builds of the product are higher quality



Tasks are done earlier in the development cycle 5

Feature Crew Tenets

    

Small, agile work units – the smaller the better Dedicated resources to reduce context switching Autonomy and decision making pushed to lowest levels – you decide how to work, who does what All members “in it together” – begin and end together Feature Crew continues until all work is “done” - feature complete, no bugs, tests in place, etc.

6

Key Concepts

   

Quality Gates A set of quality metrics and deliverables that must accompany a feature when it is first submitted to the product . Feature A discrete piece of independently testable functionality that integrates at the same time into the PU Branch .

Feature Branch A light-weight source branch that is dedicated to the development of a single feature. Goal is to allow Feature Crew to fully stabilize in isolation before integrating .

SCRUM A project management methodology that involves regular check-in meetings to map progress .

7

  

What do Feature Crews offer?

Increased Efficiency     

More parallel involvement of all disciplines (better Dev and Test integration) Less context switching in the core phase of building a product Design issues and feature bugs are found earlier when they are cheaper to fix More effective and regular communication with cross discipline counterparts Fewer intra-version breaking changes

Enhanced Agility and Predictability  

More honest schedules More consistent definition of “done” with quality gates

 

More confidence in release dates Spend more time building and less time stabilizing

Improved Quality 

Higher quality new features



More stable builds to allow for better customer feedback 8

Feature Crew Life Cycle

9

Discipline Activities

Dev

Feature design, dev doc, prototyping

Test

Feature design, test plans,

PM

Feature design, functional spec

UE

Feature design, content plans Checkpoint 1 Design Review Coding, bug fixing Checkpoint 2 Status Review Feature complete, ready to integrate Unit tests, bug fixing Finalizing quality gates, testing, bug fixing Test libraries, automation Threat model, bug triage, Feature DCRs, Bug triage, writing samples, quickstarts Testing Finalizing quality gates, testing Finalizing quality gates, testing Writing content Finalizing quality gates 10

Responsibility Areas

PM Dev Test UE Architects



Lead the feature crew



Status tracking/reporting



FC schedule



Functional specs



Bug Triage



Design docs



Code features, F1



Code unit tests



Test plan



Automation tests



Content plan



Write documentation



Write samples, quickstarts



Review FC plans



Review dev designs



Review FC plans, specs Loc, UX, Fundamentals, Partners Leads, Mgmt



Review FC plans, specs, status



Write samples, quickstarts



App Building



Review UE docs



Test features (scenario testing)



Fix bugs



Test features



Review UE docs



Test features



Review UE docs



Review FC plans, designs



Review UI strings



Test features (scenario testing)



Review code



Test features as appropriate



Manage change, people 11

SCRUM

   

Agile programming methodology



Small, dedicated groups Daily sync up meetings among crew members



Anyone can attend, but only crew members can talk



Led by a Scrum Master who tracks progress and drives problem solving Ideally – short ~10 minute meeting where crew members answer only three questions:

  

What did you do since we last met?

What are you doing next?

Are you blocked? Yes/No Problem solving and discussion happens after the SCRUM meeting 12

Phase-by-Phase Breakdown

      

FC Kickoff Design Phase Checkpoint #1 – Design Review Implementation Phase Checkpoint #2 – Progress Review Integration Phase Feature Complete/Integration Verification 13

   Design Phase

FC Design Phase

Design specs Functional specs Activities

– agreeing what to do, how to do it



Create functional specs, dev design plans, test plans, UE plans

   

Enumerate Quality Gates, Dependencies, Fundamentals



Build into FC plans, decide when/who/how



All disciplines document plans from their perspective in parallel Identify risks, issues with any



Schedule Quality Gate requirements Updated costing of features, updates to schedule if needed Define Checkpoint process for FC (depends on feature complexity)

Deliverables    

functional spec, dev design, test plans, UE plan FC Schedule solidified, Work items identified, Feature list updated with appropriate info Feature Branch created, builds setup Tools in place

How it works   

FC works together to define designs, all participate in all aspects of design, iterative brainstorming meetings Interactive, iterative discussions between PM, Dev, Test, UE Design phase completion triggers Checkpoint #1



FC decides when/how to schedule checkpoint with mgmt team 14

Testing - Design Phase



Characteristics of a good test

    

It has a reasonable probability of catching an error It is not redundant It’s the best of its breed It’s neither too simple nor too complex It makes program failures obvious 15

Testing - Design Phase



Techniques to come up with powerful test cases

    

Equivalence class analysis Boundary analysis Testing state transitions Testing race conditions and other time dependencies Doing error guessing 16

Test case design Equivalence class analysis

 

If you expect the same result from two tests, you consider them equivalent A group of tests forms an equivalence class when

   

They involve the same input variables They result in similar operations in the program They affect the same output variables None force the program to do error handling or all of them do 17

Test case design Boundary analysis



The boundary values are the biggest, smallest, soonest, shortest, loudest, fastest, ugliest members of the class, i.e. the most extreme values



Many testers include a mid-range value in their boundary tests. This is a good practice.

18

Test case design Testing state transitions

 

Every interactive program moves from one visible state to another. If you do something that changes the range of available choices or makes the program display something different on the screen, you’ve changed the program state Advice:

  

Test all paths that you think people are particularly likely to follow If you suspect that choices at one menu level or data entry screen can affect the presentation of choices elsewhere, tests the effects of those choices or entries Try a few random paths through the program. 19

Implementation Phase   

FC Implementation Phase

Coding Bug fixing Activities

– getting it done

    

Coding, creating unit tests, Test automation Testing and bug fixing Quality work: API Reviews, Threat models, etc.

Fundamentals (perf, stress, etc.)

Deliverables     

Daily SCRUM meetings (minimally quick hallway/email sync-ups among FC) Feature code and unit tests complete Automation tests in place Documentation, including tech reviews of content, samples, app building All FC bugs addressed

How it works    

Dev and Test work in tandem - iterative, parallel progress made throughout PM facilitates, tracks/communicates status, drives bug triage All FC members help out as appropriate for load balancing – no discipline boundaries Checkpoint #2 scheduled by FC during Implementation phase when main coding is complete

Testing

20

Testing – Implementation Phase

Coding  Implementation Phase Bug fixing Activities

– getting it done



Find bugs

    

File problem report , Test automation Run automated tests and perform analysis on the failures Quality work: API Reviews, Threat models, etc.

Fundamentals (perf, stress, etc.)

Testing  Deliverables   

70% automation tests in place 100% passing rate on the automated suite All problem report (bugs) are verified/closed 21

Fundamentals Details

          

Perf and Scalability Stress and Capacity

Globalization/Localization

Security

Compatibility Side-by-Side Accessibility User Experience Hosting 64-bit Servicing

22

Problem report

A good report is You need to track the problem so you must describe it in writing. Otherwise, some details (or the whole problem) will be forgotten. You also need a report for testing the fix later 23

Problem report

A good report is



Track Problem Reports numerically.



Assign a unique number to each report.



If you use a computerized database, the report number will serve as a key field.



This is the one piece of information that always distinguishes one report from all the rest.

24

Problem report

A good report is simple = not compound Describe only one problem on one report Why multiple bugs on a single report are a problem?



The programer can fix only some of them and pass the report back as fixed, even though some bugs have not been fixed. This wastes time and leads to bad feelings. Remaining problems often stay unfixed long time… 25

Problem report

A good report is You must describe the program’s problematic behavior clearly. Keep all unnecessary steps out of the steps required to reproduce the problem.

26

Problem report

A good report is Think of the person reading it. Unless you are reporting a disaster, the programmer will toss an illegible report onto his/her pile of things to look at next year 27

Problem report

A good report is Nobody likes being told that what they did was wrong. As a tester, that’s what you tell people every day. Do not express personal judgments in your reports The programmer and the tester are a team that makes the product better 28

Problem report

A good report is How can I make the bug reproducible?



You can describe how to get the program into a known state, and from there describe an exact series of steps



Find the simplest, shortest and most general conditions that will trigger the bug



Find related problems



Find alternate paths to the same problem



Find the most serious consequences



Look for the critical step 29

Test Automation

  Automation framework (in house)     Building Maintainable Testsuites Automation Results Reporting Automation auto-analysis tools Code coverage Keeping test suites reliable  Gauntlet and Code Reviews

30

FC Integration Phase

   Activities

– making sure it works

    

integrate from PU branch Run all tests, verify feature works in integrated world



Several functional runs

 

Stress Performance



Code Coverage Run Fix and verify bugs (includes Product bugs) Quality Gates final verification integrate into PU branch, final verification in PU branch

Deliverables   

Feature List data updated Quality Gates complete, verified Final integration testing complete

How it works 

FC tests feature with latest integrated code from PU Branch

 

All FC members verify feature meets requirements and works as expected Team members move onto next Feature Crew 31

Resources:



Books

  

How to Break Software: A Practical Guide to Testing,

by Whittaker, James A. (Author)

Lessons Learned in Software Testing,

by Cem Kaner (Author)

Effective Software Testing: 50 Specific Ways to Improve Your Testing

by Elfriede Dustin (Author)

32

Comments, Q & A

33