Transcript Malcolm Baldrige National Quality Award

Welcome back, Examiners!
Washington State Quality Award
Examiner Training
2010
Session 2
First things first!
• THANK YOU again!
• Our mission: Improving the way we live, learn, and
work in Washington by helping organizations
improve through the use of the Baldrige Criteria
for Performance Excellence
WSQA Examiner 2009
2
Objectives For This Class
• Learn to evaluate a WSQA application
through all stages of the assessment
process
• Do this with confidence
• In yourself as an Examiner and
• In the assessment process
WSQA Examiner 2009
3
Once-around question
• Why do you think someone suggested you
would make a good WSQA Examiner?
WSQA Examiner 2009
4
Agenda: Session 1
1.
2.
3.
4.
5.
WSQA organization overview
Baldrige criteria overview
WSQA Assessment process
Online scorebook preview
Independent review,
step-by-step
•
•
•
•
Preparing
Key Factors
Process review
Results review
WSQA Examiner 2009
Mostly presentation
Mostly exercises
5
Agenda: Session 2
6. Ethics, code of conduct
7. Consensus, step-by-step
8. Key Themes
9. Site visit preview
10. Final scorebook preparation
11. “Lite” vs. “Full”
12. Summary
WSQA Examiner 2009
Mostly exercises
Mostly presentation
6
5. Ethics, Code of Conduct
Objectives:
1. Understand the importance of
ethics in this process
2. Interpret and apply the Code
Conflict of interest issues
• WSQA seeks to avoid conflict of interest to
protect process integrity
• Conflicts are frequent!
• First step in assignment process: determine
any conflicts
WSQA Examiner 2009
8
Code of Ethical Conduct
• Document is in your workbook
• Why are ethics of utmost importance to
WSQA and this process?
WSQA Examiner 2009
9
Exercise 4:
Code of Ethical Conduct Homework
Lessons Learned from Home work
Questions on Conflict of Interest
WSQA Examiner 2009
10
6. Consensus, step-by-step
Objectives:
1. Understand consensus stage
activities, roles, responsibilities
2. Be able to develop valuable
comments
Topics
• Consensus stage activities:
• Developing process and results comments
• Recommending scores
• Comment guidelines – do’s and don’t’s
WSQA Examiner 2009
12
Stages of the Assessment Process
Receive applications
Individual
Scorebooks
Stage 1:
Independent Review
Final product
Stage 2:
Consensus Review
Consensus
Scorebook
Lite or Full
application?
Editor
updates
scorebook
Consensus
Scorebook
Site visit
scorebook
Lead & Mentor
meet with
Judges Panel
Lite
WSQA
updates
scorebook
Feedback
report
Executive
Briefing
Full
Lead & Mentor meet
with Judges Panel
Selected for
site visit?
No
Yes
Stage 3:
Site Visit
Judges recommend
award level to
WSQA Board
WSQA Symposium
& Awards
Ceremony
WSQA Consensus: Step-by-step
Application
Criteria
Consensus
planning call
Independent
review
scorebooks
Observations
(from Ind.
Review)
Key Factors
Editor or Lead
Category Lead
Category Lead
1. Consolidate
scorebooks
2. Review and group
team’s observations
3. Select relevant Key
Factors
Single
consensus
scorebook
Grouped
observations
4-6 (3-5)
Relevant Key
Factors
For your category/
categories only
Numbers in
parentheses apply for
Lite assessment only
1
WSQA Consensus: Step-by-step
1
Grouped
observations
Below-the-line
observations
Back-up’s
comments &
feedback
Category Lead
Category Lead
Category Lead
4. Select and develop
comments
5. Write rationales for
below-the-line
observations
6. Exchange feedback
with Back-up, update
comments
About 6
comments
Rationales
Updated
comments
Below-the-line
observations
Scoring
recommendation
Lite: 4-6
2
WSQA Consensus: Step-by-step
2
Consensusready
comments
Draft
consensus
scorebook
Updated
Comments,
score
Team
Team
Editor
7. Review team
comments, prepare
notes, enter key
theme ideas
8. Hold consensus
call(s) or meetings
9. Prepare consensus
scorebook
Updated
consensus
scorebook
Key themes,
site visit
recommendation
Notes
Draft Key
Theme topics
FULL
only
FULL
only
Consensus
scorebook
End
Consensus preparation and call(s)
• Each Examiner responds with agreement,
suggestions for changes, or disagreement,
stating rationales
• Category Lead facilitates consensus
• Team Lead and/or PEPD mentor leads final
scoring discussion
• Can take anywhere from 4-8 hours
(depending on what?)
WSQA Examiner 2009
17
Writing Well Written
Comments
How do good comments add value?
Organization
Profile
Application
content
Criteria
The most valuable Comments/Key Themes
WSQA Examiner 2009
19
The Structure of Well Written Comments
Well written comments incorporate criteria
language, ADLI/LeTCI, scoring language, and
application language.
WSQA Examiner 2009
20
Well Written Comment Guidelines
Category 1-6
 Does the comment link to the Criteria?
 Does it consist of 2-3 grammatically correct sentences
(about 1000 characters)?
 Does it use examples from the application that illustrate
the OFI/Strength?
 Does it include a so what if it is an OFI?
 Is the statement non-judgmental and non-prescriptive?
 Does the comment link to the scoring guidelines?
WSQA Examiner 2009
21
Well Written Comment Guidelines
• Tie your comments to the basic, overall, or multiple level Criteria
requirements that are most important to the applicant.
• Write process comments so they contain: a subject identified from
the criteria, the application, or the scoring guidelines; verb(s) and
requirements from the Criteria; examples from the application, and
figure numbers, as appropriate.
• Use the evaluation factors (ADLI or LeTCI), and scoring language, to
clearly articulate the areas of the applicant’s response to clarify the
strength or provide insights that will help the applicant improve
overall organizational effectiveness and capabilities.
• Draw linkages between an Item and the applicant's Organizational
Profile.
• Ensure that the comment does not contradict other comments in the
same or other Items or in the Key Themes Worksheet.
WSQA Examiner 2009
22
The Style of Well Written Comments
• Use active voice and present tense (e.g., 'completes' rather than 'is
completed').
• Use a polite, professional, and positive tone.
• Use vocabulary/phrasing from the Criteria and the Scoring
Guidelines.
• Describe what is missing if something "is not clear". “Not clear”
comments must be resolved if the applicant has a site visit. If they
don’t do it based on site experience then say they don’t do it.
• In lieu of the applicant’s name, use such phrases like "the applicant"
or "the organization" to refer to the applicant in the independent
and consensus scorebooks. Use the applicant's name in final
feedback reports.
• Use the applicant's language/terminology when appropriate.
WSQA Examiner 2009
23
“Don’ts” to avoid in comments
1. Avoid parroting
2. Avoid being prescriptive
3. Avoid going beyond the criteria
WSQA Examiner 2009
24
“Do’s” for well-written comments
(refer to comment guideline in workbook)
WSQA Examiner 2009
25
Example Strength Comment
• The applicant identifies their core competencies as the
participating dentist network; expert claims administration;
extraordinary customer service and performance culture. A
SWOT analysis is used in the annual strategic planning process
to identify core competencies. Correlation analysis is
performed on satisfaction survey data and is used to
determine key drivers of customer satisfaction. Initiative
action plans accentuate core competency strengths.
Deployment is evidenced by the use of bi-annual CAG
meetings, ensuring alignment with the mission and the
integration of data for easy access across all departments and
customer segments
• Red-Criteria Green-ADLI Blue-Application Pink-Key Factor
WSQA Examiner 2009
26
Well Written Comments Don’t….
•
•
•
•
•
•
•
Go beyond the requirements of the Criteria or assert your
personal opinions.
Be prescriptive by using "could", "should", or "would".
Be judgmental by using terms such as "good", "bad", or
"inadequate".
Tell the applicant that they aren’t doing enough of something
Comment on the applicant's style of writing or data
presentation.
Use jargon or acronyms unless they are used by the applicant
Only “Parrot” the application or the Criteria. Provide only
enough language to add clarity—seek to add value rather than
restate information.
WSQA Examiner 2009
27
How Could This Comment Be Improved?
Application language:
• Share Food uses a biennial 12 step Strategic Planning Process that
involves community leaders, volunteers, member agencies, and
donors. Strategic challenges are identified in the “Current State”
step which includes inputs from the SWOT analysis, environmental
scan, and funding mandate review. The Strategic Planning Process
was initiated in 1997 and restructured in 2004 as a result of
benchmarking analysis of food banks and other nonprofit
organizations, and feedback from the state award process.
Comment:
• Share Food uses a biennial 12 step Strategic Planning Process (SPP)
that involves community leaders, volunteers, member agencies, and
donors. Strategic challenges are identified in the Define step which
includes inputs from the SWOT analysis, environmental scan, and
funding mandate review. This is proof that the organization has a
great systematic approach to the SPP and extensive cycles of
learning in 1997, and 2004.
WSQA Examiner 2009
28
How Could This Comment Be Improved?
Application language:
• The manager’s vision helps Our Store to maintain a strong
focus on customer satisfaction. The manager has worked
hard to instill the belief that leadership and responsibility
should be present at all levels. This rewards everyone at Our
Store with the responsibility of planning, carrying out, and
evaluating their own work.
Comment:
• Our Store is in the beginning stages of building an approach to
setting a vision for customer satisfaction, this includes the
manager working very hard to instill the beliefs in leadership
and responsibility. Everyone in the store receives a reward
and this will ensure that everyone will have ownership in the
organization and the ability to evaluate their own work, which
should improve performance overall.
WSQA Examiner 2009
29
How Could This Comment Be Improved?
Application language:
• Our Hospital builds relationships with customers through a variety of
methods including patient enrollment, which serves as an
orientation to enabling services; Patient-Family Advisory Boards,
which have recommended and implemented improvements to
patient care; and high-visibility community health activities. Using
patient surveys, conducted randomly after discharge, Our Hospital
determines the issues that lead to poor relationship results and uses
this information to implement new relationship-building processes.
The Patient-Family Advisory Board is an example of the results of this
process.
Comment:
• Customer relationships are built through a variety of methods
including Patient enrollment, Patient-Family Advisory Boards, and
community health activities. Surveys are conducted randomly after
patients are discharged. The Patient-Family Advisory Board is used to
determine issues that lead to poor relationships and implements
new relationship-building processes.
WSQA Examiner 2009
30
How Could This Comment Be Improved?
Application language:
• Customer groups are identified annually as part of the Strategic
Planning Process using Our Hospital’s vision/mission/values as a
focus. By analyzing demographic data from multiple sources, the
cross-location Service with Spirit Team (SWST) is able to identify the
gaps, look at disparities, and identify potential customers.
Deployment of this process through multiple cycles has resulted in
the development of several services designed to meet the unmet
needs of customers in the Our Hospital’s service area.
Comment:
• Through the SPP, customer groups are annually identified using the
MVV as a focus. The SWST uses demographic data to identify gaps,
disparities and identify potential customers. This process has been
deployed throughout the organization, and through multiple cycles
of refinement the customers needs in the service area have been
met.
WSQA Examiner 2009
31
Well Written OFI Comment Guidelines
• May be used to give credit to the applicant for some areas that
HAVE been addressed
• “While the applicant does x, y, z, it is not evident that a, b,
and c are performed.”
• May begin with a statement about what is missing:
• “There is no evidence that…”
• “There is no indication of how…”
• “It is unclear…”
• “It is not apparent that there is a …”
• Must tie to criteria language, (ADLI), and/or to Key Factors that
the applicant has identified in the Organizational Profile
• Must include a “so what” (Why does the applicant care about
this? Why would they take action?)
• Usually these come from Key Factors / Org Profile. The closer
the ‘so what’ can be to identifying the applicant’s specific
needs, the better.
WSQA Examiner 2009
32
Example OFI Comment
• Although cross functional teams are chartered and aligned with the
SPP, there is no indication of processes used by the teams or otherwise
by the applicant to design and innovate its work processes to meet key
requirements. There is no indication of how design and innovation
processes address opportunities for agility and improvements in cycle
time and productivity. It is also unclear how the team work process
improvement approach is evaluated for annual process changes or
improvements. An annual evaluation of strategic team alignment and
improvements may provide efficiency and effectiveness in the
accomplishment of strategic objectives, initiatives, and action plans.
• Red-Criteria Green-ADLI Blue-Application Pink-Key Factor
WSQA Examiner 2009
33
How Could This Comment Be Improved?
Application language:
• Share Food uses a biennial 12 step Strategic Planning Process that
involves community leaders, volunteers, member agencies, and
donors. Strategic challenges are identified in the “Current State”
step which includes inputs from the SWOT analysis, environmental
scan, and funding mandate review. The Strategic Planning Process
was initiated in 1997 and restructured in 2004 as a result of
benchmarking analysis of food banks and other nonprofit
organizations, and feedback from the state award process.
Comment:
• Although Share Food uses a biennial 12 step Strategic Planning
Process (SPP) that involves community leaders, volunteers, member
agencies, and donors, there is no evidence of how blind spots are
identified, nor how time horizons are set and how the SPP addresses
those horizons. If the applicant implemented a more extensive
benchmarking process that identified blind spots and included time
horizons, there may be more of an opportunity for this organization
to have a more successful Strategic Planning Process.
WSQA Examiner 2009
34
How Could This Comment Be Improved?
Application language:
• The manager’s vision helps Our Store to maintain a strong
focus on customer satisfaction. The manager has worked
hard to instill the belief that leadership and responsibility
should be present at all levels. This rewards everyone at Our
Store with the responsibility of planning, carrying out, and
evaluating their own work.
Comment:
• Although the manager has worked hard to instill the belief
that leadership and responsibility should be present at all
levels, there is no indication of a systematic approach, nor is
there deployment at any level, no integration, nor are there
any cycles of learning.
WSQA Examiner 2009
35
How Could This Comment Be Improved?
Application language:
• Our Hospital builds relationships with customers through a variety of
methods including patient enrollment, which serves as an
orientation to enabling services; Patient-Family Advisory Boards,
which have recommended and implemented improvements to
patient care; and high-visibility community health activities. Using
patient surveys, conducted randomly after discharge, Our Hospital
determines the issues that lead to poor relationship results and uses
this information to implement new relationship-building processes.
The Patient-Family Advisory Board is an example of the results of this
process.
Comment:
• There is no systematic approach to building customer relationships,
or in conducting patient surveys. And although there is a PatientFamily Advisory Board there is no approach described for how issues
are determined and how new relationship-building processes are
developed.
WSQA Examiner 2009
36
How Could This Comment Be Improved?
Application language:
• Customer groups are identified annually as part of the Strategic
Planning Process using Our Hospital’s vision/mission/values as a
focus. By analyzing demographic data from multiple sources, the
cross-location Service with Spirit Team (SWST) is able to identify the
gaps, look at disparities, and identify potential customers.
Deployment of this process through multiple cycles has resulted in
the development of several services designed to meet the unmet
needs of customers in the Our Hospital’s service area.
Comment:
• Although customer groups are identified annually in the SPP, and the
Spirit Team (SWST) identifies gaps, disparities, and identifies
potential customers, there is no indication how this information is
deployed throughout the organization and how the information is
used to meet the unmet needs of customers in the area.
WSQA Examiner 2009
37
Comment Writing Exercise
• Each person at the table select an independent
review item (Cat 1-6 strength or OFI)
• Write a well written comment
• Swap scorebooks
• What feedback would you give to this examiner
• Use the comment checklist for examples
• Practice giving feedback to each other
• Report out the most significant key learnings
• (If time swap with to another person at the
table)
WSQA Examiner 2009
38
Category 7 Comment Guidelines
Category 7
 Are specific results (numbers) used rather than
generalities?
 Have trends, comparisons and/or appropriate
benchmarks been considered?
 Does the comment include figure numbers?
 Have we summarized multiple charts with a consistent
theme (trends, missing, comparisons, etc.)
WSQA Examiner 2009
39
(Well Written Result Comment Example
There is a lack of comparative data in a number of
the Results sub-categories; as an example, no
comparative data is available for measures such as
Discharge Weight of ELBW Infants (7.1-4),
Improvement % of ELBW Infants (Figure Figure 7.15), % of Breast Milk Available (Figure Figure 7.1-6),
Outpatient Lactation (7.1-7) and Catheter Related
Bloodstream Infections (7.1-8), Survival to Discharge
by Rhythm (Figure Figure 7.1-17), and Well Child
Visits Figure (7.1-19). The lack of comparative
results may impact the value of Excellence - setting
and achieving the highest standards.
WSQA Examiner 2009
40
How Could This Comment Be Improved?
The applicant’s market share results for key service lines is
strong, exceeding all comparative data across Discharges;
Inpatient Surgeries; the applicant's Heart Services; and
Neurosurgery Market. The applicant enjoys market
dominance in these service lines and has shifted its focus to
adolescent care and services to new geographic markets. The
applicant has received numerous recognitions and awards for
their leadership in these markets and service lines. These
successes are important to their goal of achieving national
recognition as a premier hospital, a tertiary and quaternary
hub, and preferred provider of regional health care services.
WSQA Examiner 2009
41
How Could
These Comments Be Improved?
• The applicant's Net Income from Operations (7.3.8) is above
the comparison median and trending up over the past three
years and overall applicant’s turnover rate has been
decreasing.
• The applicant’s Net Sales between 2006 and 2008, have
increased by 10% in 2006 to over 12.2% in 2008. The applicant
is consistently above the goal of 9% for all three years, but is
below the Baldrige comparison of 15% for all three years.
• Although the turnover rate for RNs has been increasing
between 2006 and 2008. The turnover rate for RNs was 7.7%
in 2006 and 10.3% in 2008. There have been substantial
improvements in physician satisfaction from 13% to over 25%
during the same time period.
WSQA Examiner 2009
42
How Could
These Comments Be Improved?
• The applicant's Net Income from Operations (7.3.8)
is below the comparison median and trending down
over the past three years and overall applicant’s
turnover rate has been increasing between 2006
and 2008, from 10% in 2006 to 12.2% in 2008.
• The turnover rate for RNs has been increasing
between 2006 and 2008. The turnover rate for RNs
was 7.7% in 2006 and 10.3% in 2008.
WSQA Examiner 2009
43
Comment Writing Exercise
• Each person at the table select an independent
review item (Cat 7 strength or OFI)
• Write a well written comment
• Swap scorebooks
• What feedback would you give to this examiner
• Use the comment checklist for examples
• Practice giving feedback to each other
• Report out the most significant key learnings
• (If time swap with to another person at the
table)
WSQA Examiner 2009
44
Consensus and conclusion
• If using online tool during consensus,
changes may be made in real time
• Category Lead notifies Team Lead and
Scorebook Editor when done
• Scorebook Editor compiles consensus
scorebook with final edits, notifies WSQA
• Consensus scorebook should be finalized
ASAP
WSQA Examiner 2009
45
Comment, scoring considerations
• “Benefit of the doubt”
• Give credit for what’s in the application
• Don’t penalize for incidental exceptions
• Not every process must show complete “DLI”
• Not every results example must include “TCI”
WSQA Examiner 2009
46
Comment, scoring considerations
• For LITE, only one score for entire results
Category 7
• Propose a score for results related to your
categories
• Be prepared to reach team consensus for all of
Category 7
• Using scoring language can help the
applicant understand comments and score
WSQA Examiner 2009
47
Category feedback
• Check and provide feedback on whether Cat. Lead’s
comments:
•
•
•
•
•
•
•
Include criteria requirements
Provide meaningful feedback
Address ADLI or LeTCI
OFI’s provide relevant “so what” (at least some of them)
Results cite figure numbers/titles and specifics
References to application are correct
Recommended score reflects comments in content and
balance (strengths:OFI’s ratio)
• Back-up does this for full, team does this for lite
WSQA Examiner 2009
48
Information resources
• WSQA comment guideline handout
• Your back-up, team lead, PEPD mentor
• Reference materials in
• Your workbook: comment and scorebook examples
• http://www.wsqa.net/extraining.php
• Baldrige web site
http://baldrige.nist.gov/Examiner_Resources.htm
WSQA Examiner 2009
49
7. Key themes
Objectives:
1. Be able to identify Key Themes
2. Be able to develop a Key Theme
comment
How do good comments add value?
Organization
Profile
Application
content
Criteria
The most valuable Comments/Key Themes
WSQA Examiner 2009
51
What is a Key Theme?
A “key theme” is a strength or OFI that
• Addresses a central requirement of the
Criteria,
• Is common to more than one Item or
Category (crosscutting),
• Is especially significant in terms of the
applicant’s key factors, and/or
• Addresses a Core Value of the Criteria
WSQA Examiner 2009
52
Developing Key Theme comments
• Address all double strengths and OFI’s individually
and/or in groups
• Address related comments across categories
• Develop 6-10 (Guideline!) key themes each for
significant
•
•
•
•
Process strengths
Process OFI’s
Results strengths
Results OFI’s
• See example Key Themes in the workbook for this
class
WSQA Examiner 2009
53
Key Theme Development
using the Baldrige Core Values
 Purpose:
Create Key Themes at the end of the consensus meeting or at
the end of a site visit
WHY?
to identify Key Themes that cut across categories and items
What is a Key Theme?
 A high level perception or observation, relevant to the
applicant’s key factors, that recurs in the scorebook and is
expanded upon with evidence during a site visit. Key
themes are often common to more than one category or
item. They may address one or more Baldrige core value.
 A role model result, practice, process, or innovation, that
might be used by other organizations, or an observation so
important, strength or OFI, process or result, that it should
be brought to leadership’s attention.
 The executive summary of the feedback report. It may be
the only part of the feedback report that some people in
the organization read.
WSQA Examiner 2009
Cross Cutting Themes
• In the early days of the Baldrige, Key Themes
were called Cross Cutting Themes, or Cross
Cutting Issues, because they described topics
that cut across the categories and items.
WSQA Examiner 2009
Core Values-The Foundation of the
Baldrige Criteria
•
•
•
•
•
•
•
•
•
•
•
Visionary Leadership
Customer driven excellence
Organizational and personal learning
Valuing workforce members and
partners
Agility
Focus on the future
Managing for innovation
Management by fact
Societal responsibility
Focus on results and creating value
Systems perspective
• If you think of the 7
Baldrige categories as
the vertical beams in a
house frame, the 11
Baldrige Core Values are
the cross members that
provide strength to the
frame.
WSQA Examiner 2009
Process for using Baldrige Core Values
to Develop Key Themes
• Each Team Member:
• Review the applicant
• Read the Baldrige Core Values
• Write 11 post-its rating the applicant against each Baldrige Core
Value with a +, 0, or – and write a few short thoughts for each
core value explaining why you rated the applicant as you did.
• Key Themes Lead
• Sum the plusses and the minuses. Core Values that have many
plusses, or minuses, may be candidates for Key Themes.
Typically only a few will stand out.
• Report results to team.
• Draft Key themes for these high scoring values using feedback
report comments and post-it thoughts as inputs.
WSQA Examiner 2009
Using Core Values to ID Key Themes
Baldrige Core Value
Score (+,0,-)
Visionary Leadership
+,+,+,+,+,+ (6+) Probable KT
Customer driven excellence
+,+,0,0,0,-
Organizational and personal learning
+,+,+, 0,0,0
Valuing workforce members and partners
+,+,+,+,+,- (5+) Possible KT
Agility
+,+,0,0,-,-
Focus on the future
+,+,+,0,-,-
Managing for innovation
0,0,-,-,-,-
Management by fact
-,-,-,-,-,-, (6-) Probable KT
Societal responsibility
+,+,+,+,+,0 (5+) Possible KT
Focus on results and creating value
0,-,-,-,-,- (5-) Possible KT
Systems perspective
+,+,0,0,-,WSQA Examiner 2009
Caveat to this approach
• While the Baldrige Core Values are the
foundation upon which the Criteria are
based, they are not examination items. Key
themes are written based on comments
developed relating to the criteria. The
feedback report comments MUST support
the Key Themes.
WSQA Examiner 2009
9. Site visit preview
Objectives:
• Understand value of site visit
• Set expectations
Site visit preview
• Site visit is awarded based on
• Examiners’ recommendations
• Scoring band (Leadership & higher is automatic)
• Judges’ final determination
• Purpose of site visit
• Verify strengths
• Clarify opportunities for improvement
• Improve quality of feedback report
WSQA Examiner 2009
62
Site visit preview
Preparation
•
•
•
•
•
Site Visit JIT training
Planning calls, team assignments
Develop Site Visit Issue worksheets
Request documents
Lead drafts interview schedule as possible
WSQA Examiner 2009
63
Site visit preview
Day 1: Sunday (planning day)
• Applicant provides requested documents,
equipment (printer, shredder, etc.)
• Draft and/or update interview schedules
• Review and update SVI worksheets
• Develop walk-around questions
WSQA Examiner 2009
64
Site visit preview
• Monday – Wednesday ON-SITE
•
•
•
•
•
Introductory meeting (meet & greet only)
Interviews – individual, teams, groups
Team caucuses, lunches
Walk-around questions
Visit all possible sites and shifts
• Back at the hotel…
• Work on SVI worksheets, close out as possible
• Team reviews: “Walk-the-wall”
• All issues closed by end of day Wednesday
WSQA Examiner 2009
65
Site visit preview
• Thursday – Friday
• Draft comments based on updated SVI
worksheets
• Review and finalize comments
• Review and finalize Key Themes
• Determine final scoring recommendation
• Assemble final scorebook
• Celebrate!
• Saturday: return home!
WSQA Examiner 2009
66
10. Final scorebook preparation
Objective: understand preparation
of WSQA final product
10. Final scorebook preparation
Topics:
1. What does the team do?
2. What do the Team Lead, Editor,
and Judges do?
3. What does WSQA do?
4. Final scorebook preview
What does the team do?
• Everything possible to save your Editor work!
• Follow the comment guidelines (refer to
examples)
• Update comments during or immediately after
consensus
• Let your Team Lead and Editor know when your
comments have been updated
• Continue to use “applicant” throughout, not
the applicant’s actual name
WSQA Examiner 2009
69
What does the Editor do?
• Attends Senior Examiner training
• Corrects/edits the scorebook for consistency
with comment guidelines
• See workbook for details
WSQA Examiner 2009
70
What do the Team Lead,
PEPD Mentor, and Judges do?
• Lead and Mentor prepare for presentation of
final scorebook to Judges
• Lead and Mentor present team’s scorebook
at Judges’ review
• Summary/highlights of comments
• Summary of Key Themes (Full)
• Scoring recommendation
• Judges determine award level based on
team’s recommendation
WSQA Examiner 2009
71
What does WSQA do?
• Copies team scorebook into Feedback Report
template
• Replaces “applicant” with applicant’s actual
name throughout
• Sends Feedback Report to applicant
• Coordinates Executive review meeting date
and participants
WSQA Examiner 2009
72
Final scorebook preview
• (Open scorebook file)
• Case Study final scorebook is online
WSQA Examiner 2009
73
11. “Lite” vs. “Full”
Objective: Understand distinctions in
• Application Process
• Examination Process
Lite vs. Full application process
Topic
Lite (Assessment)
Full Examination
Application Deadlines
Quarterly
Once annually
Intent to Apply/
Application Form
Required. Accepted
throughout the
year. Intent to Apply forms
must be submitted at least
4 months prior to the
submission of your Lite
Application.
Required. See Web site for
due date.
Criteria
Lite (Assessment) Criteria,
A subset of questions from
the Full criteria
Full version Baldrige
criteria
Version
Current or previous year
Current year only
WSQA Examiner 2009
75
Lite vs. full application process
Topic
Lite (Assessment)
Full Examination
Organizational Profile
Required, up to 5 pages
Required, up to 5 pages
Page limit, application
body (not including
2007: 15 2008: 20
50 pages
Examiner participation
At least one required
At least one required
Recognition
None
As determined by
Judges
Executive briefing
Yes
Yes
profile, org chart,
glossary)
WSQA Examiner 2009
76
Lite vs. full: examination process
Topic
Lite (Assessment)
Full Examination
Team size
2-3 plus PEPD Mentor
5-10 plus PEPD Mentor
Stage 1:
Independent
review
(Same)
(Same)
Stage 2:
Consensus
Omits Key Themes
Includes Key Themes
Judges Panel
review
None
Recommends site visit OR
determines award level
Stage 3: Site visit
None
As determined by Judges
Post-site Judges
panel review
None
Determines final scoring and
award level
Feedback report
Final report prepared by WSQA
Final report prepared by WSQA
WSQA Examiner 2009
77
TIME TABLE
• Always refer to WSQA website
• Examiner Schedule for Both Full and Lite
Schedules
http://www.wsqa.net/extimetable.php
• Applicant Schedule
http://www.wsqa.net/timeTable.php
Schedule & Time Commitment
Activity
Full
Lite
Training
20 hours
20 hours
Individual review
20-40
16
Consensus preparation: comment writing
8-16
4
8
4
16-24
4-8
60+
None
Many!
(not as
many)
Participate in consensus meeting(s)
Add if Team Lead, Back-up, Editor
Site visit including preparation & training
Total:
WSQA Examiner 2009
79
Discussion question
Considering “Full” vs. “Lite” criteria, what are
the trade-offs for each
• From the Applicant’s point of view?
• From the Examiner’s point of view?
WSQA Examiner 2009
80
12. Summary
Objectives:
Summarize what we’ve learned
Answer any remaining questions
High-level objectives
• Learn to evaluate a WSQA application
through all stages of the evaluation process
• Do this with confidence in yourself and in the
assessment process
WSQA Examiner 2009
82
What we’ve covered: Session 1
1.
2.
3.
WSQA organization overview
Baldrige criteria overview
WSQA Assessment process
4.
Independent review,
step-by-step
•
•
•
•
Preparing
Key Factors
Process review
Results review
WSQA Examiner 2009
83
What we’ve covered: Session 2
5. Ethics, code of conduct
6. Consensus, step-by-step
7. Key Themes
8. Online scorebook preview
9. Site visit preview
10. Final scorebook preparation
11. Executive briefing
12. “Lite” vs. “Full”
WSQA Examiner 2009
84
Managing your time
• Put aside time in your
calendar (refer to time
commitments)
• Work with your lead &
team
• Set meetings early
• Coordinate
• Communicate
• Stories to tell?
WSQA Examiner 2009
85
Managing your time
• Schedule work hours
• Break tasks into little
pieces
• Reward yourself for wins
• Move quickly past
setbacks
• Call your lead
• Call your back-up
WSQA Examiner 2009
86
Your questions?
WSQA Examiner 2009
87
Your feedback, please!
• Please complete surveys before leaving
WSQA Examiner 2009
88
THANK YOU!!
Your support and participation as
Examiners helps us all by helping WSQA
fulfill its mission!
WSQA Examiner 2009
89