Transcript Test Process Improvement

Test Process Improvement
The Model – Audit Methods –
Lessons Learned
Q
TPI Assessment Training
Ver. 1.2
1
My Experience in Process Audit
• ISO 9000 training – TickIT Lead Auditor
• Process audits and improvements
• Building test teams & configuration management
CIS Technologies
HTE
London Bridge
Intersil
Duval County School Board
Parts.com
TransTegrity
Q
TPI Assessment Training
Ver. 1.2
3
Assessing & Auditing
•
•
•
•
•
The Hunt for Facts
Face Time Techniques
The TPI Framework
TPI Scoring
On Stage Reporting
Q
TPI Assessment Training
Ver. 1.2
5
The Steve Irwin Model
In search of the
phenomenon.
Q
TPI Assessment Training
Ver. 1.2
6
Assessing & Auditing
• Self Assessment – assess your own group
• Internal Assessment – assess another COF
group
• Formal Assessment – assessment by
outside entity
Q
TPI Assessment Training
Ver. 1.2
7
Assessing & Auditing
Internal versus External Audit
– Fluid definitions, relative terms
– External – outside the organization
Today’s discussion addresses both.
Q
TPI Assessment Training
Ver. 1.2
8
Assessing & Auditing
1. Personal Observation
2. Documentation
3. Interviews
Q
TPI Assessment Training
Ver. 1.2
10
The Tom Brokaw Model
Separate Fact From
Perception & Feelings.
Q
TPI Assessment Training
Ver. 1.2
11
Audit Interviews?
Why should self assessment involve
interviews?
– One viewpoint is not as robust.
– “Reading between the lines”
Compliance is through consistent,
uniform practice!
Q
TPI Assessment Training
Ver. 1.2
12
What does a successful
audit interview look like?
Q
TPI Assessment Training
Ver. 1.2
14
Audit Interview
What things are done
– Exchange information – auditor about the audit,
audited about the content
– Set boundaries – topics, interruptions, timeconsuming fact hunts
– Respect the audited, be cordial
– Stay on the topic, with minor casual exceptions
Q
TPI Assessment Training
Ver. 1.2
15
Audit Interview
•
•
•
•
Develop working hypotheses.
Prove right or wrong.
Validate.
Follow-up with questions & calls.
Q
TPI Assessment Training
Ver. 1.2
16
The Sherlock Holmes Model
Solve the mystery.
Q
TPI Assessment Training
Ver. 1.2
17
Audit Interview
What shouldn’t happen
– Lots of casual unrelated conversations
– Educating the audited extensively on
procedures, TPI
– Stating conclusions as auditor
Q
TPI Assessment Training
Ver. 1.2
18
Audit Interview
Feelings
– Auditing is all about listening! People like to
be heard.
– Defensiveness can occur.
Q
TPI Assessment Training
Ver. 1.2
19
The Sigmund Freud Model
What personal
experiences has brought
this person(/team) to this
point?
Q
TPI Assessment Training
Ver. 1.2
20
Audit Interview
Outputs:
–
–
–
–
Set of notes – absolutely necessary!
Other pertinent contacts – full names & roles
New documents & their locations
Action items – both sides
Q
TPI Assessment Training
Ver. 1.2
21
TPI as an Audit Model
• Audit for Compliance Findings –
benchmarking, not strict adherence to
regulations & policies
• Make practical recommendations to
improve work life
• CapitalOne culture readily accepts
Q
TPI Assessment Training
Ver. 1.2
23
Interview Techniques
Q
TPI Assessment Training
Ver. 1.2
30
Interviews – Etiquette &
Personalities
• Pre-interview – Provide an agenda &
background material.
• Dress for success.
• Be punctual.
• Address with respect.
• Frame open-ended questions with charity.
• Avoid confrontational questioning.
Q
TPI Assessment Training
Ver. 1.2
32
Interviews – Putting the Interviewee
at Ease
• Agenda spelled out.
• Use appropriate humor.
• Listen Actively.
Q
TPI Assessment Training
Ver. 1.2
33
Interview Difficulties
Misunderstandings sometimes occur.
– Listen carefully & actively.
– Address concerns fully & repeat for feedback.
– Stay grounded in facts.
Q
TPI Assessment Training
Ver. 1.2
36
Validation of Responses
• Review and complete your notes soon after
the interview.
• Refer to documentation.
• Chat informally with others in the group.
• Set up additional interviews.
• Phone or stop by for follow-up questions.
Q
TPI Assessment Training
Ver. 1.2
37
Pacing the Assessment
• Two weeks includes interviews, research,
and reporting.
• Total effort should be less than 40 hours.
Don’t forget the delay for scheduling!
Schedule all interviews and the end report
immediately!
Q
TPI Assessment Training
Ver. 1.2
38
The Stephen Covey Model
Begin with the End in
Mind.
First Things First.
Think Win-Win.
Q
TPI Assessment Training
Ver. 1.2
39
Scoring the Assessment
Q
TPI Assessment Training
Ver. 1.2
40
Develop the Score
Key Area
descriptions
Interview questions
Answers to
questions
Decisions
about
Key Areas
Assessment
Results
Q
TPI Assessment Training
Ver. 1.2
42
Develop the Score
KA 1.A.1 “A motivated consideration of
the product risks takes place, for which
knowledge of the system, its use and its
operational management is required.”
Question: “What is your test strategy,
and how did you formulate it?”
Q
TPI Assessment Training
Ver. 1.2
43
Develop the Score
Develop this on your own, as needed!
Q
TPI Assessment Training
Ver. 1.2
44
Develop the Score
Key Area
descriptions
Interview questions
Answers to
questions
Decisions
about
Key Areas
Assessment
Results
Q
TPI Assessment Training
Ver. 1.2
45
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
46
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
47
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
48
Develop the Score
• Y,N
• X or blank
= don’t know
• Any X, blank or N
answers
= “no pass for level”
• Only all Y answers
= pass
Q
TPI Assessment Training
Ver. 1.2
49
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
50
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
51
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
52
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
53
Reporting Findings
Q
TPI Assessment Training
Ver. 1.2
54
The Peter Senge Model
Systems Thinking –
People respond to the
systems in which they
are immersed.
Q
TPI Assessment Training
Ver. 1.2
55
Finalizing the Presentation
•
•
•
•
Remember your audience.
Should be a mere formality.
Tell them more than they know.
Provide actionable plan
Q
TPI Assessment Training
Ver. 1.2
58
Reporting Findings
• Management team receives report
• Written as well as verbal presentation
• No surprises
Q
TPI Assessment Training
Ver. 1.2
61
Presentation – Focused Clarity
One page summary
– Key Area map
– Brief explanations only
where necessary
Q
TPI Assessment Training
Ver. 1.2
66
The Follow-Up
• Provide further information sharing as
required.
• Avoid betraying trusts.
• Provide engagement planning.
Q
TPI Assessment Training
Ver. 1.2
69
The Take-Home
What’s the take-home?
Q
TPI Assessment Training
Ver. 1.2
75
Contents
Agenda
TPI Assessor’s Tools
Chapter Selections
Exercises
References
The 4Q03 Plan
Q
TPI Assessment Training
TPI Assessments
Ver. 1.2
77
Chapter Selections
The Successful Audit
CapitalOne Environment
LB Audit Experiences
Interview Techniques
Scoring the Assessment
Reporting Findings
Report Presentation
Postscriptum
About TPI
Q
TPI Assessment Training
TPI Assessments
Ver. 1.2
78
Exercises
The TPI Interview
Scoring TPI Results
The TPI Results Presentation
Q
TPI Assessment Training
TPI Assessments
Ver. 1.2
79
TPI Audits 4Q03
Q
TPI Assessment Training
TPI Assessments
Ver. 1.2
81
TPI Audits 4Q03
• PQM coordinates logistics.
• Auditors will be responsible for setting up
meetings, performing assessments,
reporting results.
• Forward the established timetable for an
assessment to PQM. (~2-3 weeks)
Q
TPI Assessment Training
Ver. 1.2
82
TPI Audits 4Q03
• PQM will call status meetings periodically.
• Forward completed audit results to PQM.
Q
TPI Assessment Training
Ver. 1.2
83
About TPI
Q
TPI Assessment Training
TPI Assessments
Ver. 1.2
85
The TPI Model
• Software Testing Methodology
• Developed by Tim Koomen & Martin Pol
• Worksheet and scoring tools provided &
maintained by Sogeti, links on QS website.
• CapitalOne adaptation of the scoring tool
available on the intranet.
Q
TPI Assessment Training
Ver. 1.2
86
The TPI Key Areas
Q
TPI Assessment Training
Ver. 1.2
87
Interview Structure
Q
TPI Assessment Training
Ver. 1.2
88
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
89
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
90
Develop the Score
Q
TPI Assessment Training
Ver. 1.2
91
TPI Scoring
Q
TPI Assessment Training
Ver. 1.2
92
References
• Tim Koomen and Martin Pol, Test Process
Improvement, 1999, ISBN 0201596245
• Stephen Covey, The 7 Habits of Highly
Effective People, 1989
• Sir Arthur Conan Doyle, The Hound of the
Baskervilles, 1901
• Sigmund Freud, The Ego and the Id, 1923
Q
TPI Assessment Training
TPI Assessments
Ver. 1.2
94
References
• Peter Senge, The Fifth Discipline: The Art
and Practice of the Learning Organization,
1990
• Dale Carnegie, How to Win Friends and
Influence People, 1936
• ANSI/ISO 9000: 9001, 9000-3
Q
TPI Assessment Training
Ver. 1.2
95
Exercise:
The TPI Interview
Q
TPI Assessment Training
TPI Assessments
Ver. 1.2
97
The TPI Interview
Conduct a TPI Interview
1. Review the handout for questions to ask.
2. Greet the interviewee and ask questions.
3. Make notes on the handout.
4. 5 minutes total
Interviewees – content of question responses are not
important, but make it real.
Q
TPI Assessment Training
Ver. 1.2
98
Exercise:
Scoring TPI Results
Q
TPI Assessment Training
Ver. 1.2
100
Scoring TPI Results
•
•
•
•
Scoring in reality will be done on computer.
Conduct scoring based on notes taken.
How did the project score?
Is follow-up required?
Q
TPI Assessment Training
Ver. 1.2
101
Scoring the Assessment
Scoring the Assessment
• The interview worksheets build the picture.
• Formulate responses as you go.
• Finalize Key Areas as soon as you are
able.
• Let the spreadsheet do the work.
Adherence to achieve compliance is through
consistent, uniform practice!
Develop the Score
Key Area
descriptions
Interview questions
Answers to
questions
Decisions
about
Key Areas
Assessment
Results
Develop the Score
KA 1.A.1 “A motivated consideration of
the product risks takes place…”
Question: “What is your test strategy,
and how did you formulate it?”
Develop the Score
Develop this on your own, as needed!
Develop the Score
Key Area
descriptions
Interview questions
Answers to
questions
Decisions
about
Key Areas
Assessment
Results
Develop the Score
Develop the Score
Develop the Score
Develop the Score
• Y,N
• X or blank
= don’t know
• Any X, blank or N
answers
= “no pass for level”
• Only all Y answers
= pass
Develop the Score
Develop the Score
Develop the Score
Develop the Score
Exercise:
The TPI Results Presentation
Q
TPI Assessment Training
Ver. 1.2
118
TPI Results Presentation
• Distribute the results to your recipient
manager.
• Discuss the high points.
• Take notes on responses & need for followup.
• 5 minutes total
Q
TPI Assessment Training
Ver. 1.2
119