Welcome

IPMA and SolutionsIQ

Professional Event Testing, Testing, 1…2…3… Improving software quality -- one bug at a time October 15, 2004 – 1

Agenda

• • • Building the Test Framework – Jan McCollum, SolutionsIQ • Break Practical Panel Discussion – Cheryl Hainje – AFRS Product Manager, OFM – Dotti Lane – QA Project Manager, OFM – Tim Vessey – POS Project Manager, LCB – Stein Wang – Quality Assurance Lead, SolutionsIQ • Break Testing Templates & Checklists October 15, 2004 – 2

SolutionsIQ Overview

• • • • • • • SolutionsIQ is a full-spectrum IT services company 25 years of technology services and solutions 400+ consultants Corporate headquarters (Bellevue, WA) Professional Services (Bellevue, WA) Oregon Branch Office (Lake Oswego, OR) 8+ years of serving the State of WA – DOC, AOC, LCB, DNR, DOL, LNI, & DSHS October 15, 2004 – 3

SolutionsIQ Expertise

• Professional Services Division – Consulting and Analytical Solutions • Project management • Assessments and feasibility studies • Design and architecture roadmaps – Development and Test Solutions • Full life cycle development projects • Custom application development • EAI, portals, and business intelligence • Quality assurance and testing solutions October 15, 2004 – 4

Building the Testing Framework

Jan McCollum Manager, Quality Assurance and Testing Solutions October 15, 2004 – 5

Setting Goals

• Knowing WHAT you want is as important as knowing how to get it – Defining the vision – Defining the timeline – Gaining acceptance and buy in October 15, 2004 – 6

Defining the Vision

• To define the vision look at what came before – What went well – What went badly – What now – Where do you want to go October 15, 2004 – 7

Testing vs. Quality Assurance

• Testing is about finding bugs • Quality Assurance is about preventing them!

October 15, 2004 – 8

Quality Assurance

• Takes time • Is about the overall effort – including development • Methodologies can be very formal October 15, 2004 – 9

QA Applied to Testing

• Quality assurance principals applied to the testing effort will produce higher quality work October 15, 2004 – 10

Establishing a Timeline

• The 6 month / 1 year / 3 year plan – Implement processes and strategies that give the best return on investment October 15, 2004 – 11

Quality Testing Roadmap

• After the goals and objectives are complete, make them real by publishing the quality testing roadmap October 15, 2004 – 12

Quality Testing Roadmap

• Roadmap should include… – Test team structure – Communications plans – Test processes – Test procedures October 15, 2004 – 13

Quality Testing Roadmap

• Test scope • Test dependencies and impacts • Automation transition plan • Test deliverables October 15, 2004 – 14

Gaining Acceptance and Buy In • Development • Business management • Project management • IT management • Customer/product support October 15, 2004 – 15

Making it Happen!

• Organizational structure • Qualified candidates • Roles and responsibilities October 15, 2004 – 16

Test Planning

• The master test plan: a one-stop shopping guide for your project – Contents – Contributing documents – Sign-off procedures October 15, 2004 – 17

Test Planning

• Test matrix and test suites – Detailed test steps – Pass/Fail results – Tester who performed tests October 15, 2004 – 18

Test Planning

• Test case design – what is a good test case?

– Accurate – tests what it’s designed to test – Repeatable, reusable – has a life after this release – Economical – no unnecessary steps October 15, 2004 – 19

Test Planning

• Test case design – Traceable to a requirement – Appropriate for test environment, testers – Self-standing has enough information for anyone to run October 15, 2004 – 20

Test Planning

• Test case design: How to make good test cases better – Setup, environment, data – Steps, actions and expected results – Use active voice in expected results – System displays this, does that – Simple, conversational language October 15, 2004 – 21

Test Planning

• Test case design: Why work to improve test cases?

– Productivity – less time to write and maintain cases – Testability – less time to execute them – Scheduling – better reliability in estimates October 15, 2004 – 22

Defect (Bug) Management

• Deciding upon a tool – Easy of configuration – Ability to add/change fields – Reporting capabilities • Integrated solution October 15, 2004 – 23

Defect (Bug) Management

• The bug lifecycle – Who can create bugs – Who can assign bugs – Who can close bugs October 15, 2004 – 24

Defect (Bug) Management

• The bug triage meeting – Purpose and who should go • • Reporting – Determining a trend Bug metrics – Number of bugs found – Bugs found in production vs. test cycle October 15, 2004 – 25

Moving On

• Improving the process: Requirements traceability – Test cases for each requirement – Requirements matrix – Tracing requirements to defects October 15, 2004 – 26

Moving On

• Improving the process: Risk-based testing – You can’t test everything so test what is important – The risk list and how to use it to drive test strategy October 15, 2004 – 27

Broadening Your Scope

• Build verification testing – Also called smoke or acceptance tests – Is a subset of the major functional areas • Integration testing – Testing the entire system October 15, 2004 – 28

Broadening Your Scope

• Compatibility testing – How application works with other apps • • Configuration testing – Testing on different configurations Setup testing – Testing the installation • Regression testing – Verify if bug fixes are successful October 15, 2004 – 29

Broadening Your Scope

• Black box testing • White box testing • Grey box testing October 15, 2004 – 30

Improving Quality

• Testing metrics – measure your success • Bug tracking metrics – Number found – Number found per component – Daily bug find rate October 15, 2004 – 31

Improving Quality

• Test case effectiveness – Metric: Test case effectiveness; test case effectiveness = bugs found in test/total found * 100 • Test coverage – Metric: Test coverage (absolute) = tests conducted/total tests * 100 October 15, 2004 – 32

Improving Quality

• Test team performance – Metric: Test process effectiveness: test process effectiveness = bugs fixed/bugs found * 100 – Metric: Planned days vs. actual days in test October 15, 2004 – 33

Improving Quality

• QA and test involvement early!

• Design reviews – Why testers should attend • Develop and use checklists • Project closeout meetings – You should have them October 15, 2004 – 34

Questions?

• For additonal information, email – [email protected]

October 15, 2004 – 35