Transcript Malcolm Baldrige National Quality Award

Welcome, Examiners!
Washington State Quality Award
Return Examiner Training
2010
First things first!
• THANK YOU!
• WSQA’s mission: Improving the way we live, learn,
and work in Washington by helping organizations
improve through the use of the Baldrige Criteria
for Performance Excellence
2
Objectives For This Class
• Learn changes to process
• Be aware of upcoming changes from
National
• Using Baldrige values to build key themes
• Reinforce comment writing
• Using the expected results
• Reinforce scoring
3
Introductions
• Name, organization, what you do
• Your experience with WSQA
4
Training Materials
•Slide set
•Sample review worksheets
•Independent review blank worksheets
•Comment writing guideline
5
Signed Ethical Statements
• Collect Signed Statement
6
Learning Opportunities
• WSQA offers workshops
•
•
•
•
•
•
•
•
Introduction
Self Assessment
Case Study
Preparation for Application
Annual conferences
Consulting
Performance Excellence Collaborative
Become an Examiner
Washington State
5/10/06
7
Discuss the linkage between “Key Factors”
and “Key Themes.”
Over the years, some of us have learned that when
there are multiple questions in an “Area to Address”
then generally the first question is more important
than the second one, which is more important than
the third question, etc. Is this true?
Can you describe “work systems”
more fully and perhaps
compare/contrast with “work
processes” and “core
competencies.”
Are “projections” the same or different than
“targets or goals?” (See page 58 of the
criteria, which seems to say that projections
can be targets.) Do performance projections
impact the scoring in Cat. 7?
How does an Examiner assess
innovation in an application?
How to determine if a results item
should be given credit in 7.5 rather
than 7.1?
Describe the fact-based process the
Board of Overseers uses in shaping
and improving the Baldrige criteria.
What are the likely changes to the
2011-2012 criteria?
Harry’s Session on Upcoming Changes
Process Changes
• Category 3: Rewritten to improve logic flow,
also to discuss more of the organization—
less of the content
• Category 5: Consolidation/reorder to
improve the logic flow
• Category 6: Reorganized around
“Operations Focus” with better separation
between 6.1 and 6.2. Examiners were
struggling between Work Systems and Work
Processes.
12
Upcoming Changes to Results
• Category 7: Will have only 5 items – 7.1 and
7.5 are to be combined and 7.4 will now
have to address Governance and Societal
Responsibilities
• Performance Projections will only be a factor
up in the 90-100% scoring ranges. The
“hows” will be expected to be addressed in
Categories 2 and 4.
13
General Upcoming Changes
• The criteria will now “gently introduce” both
topics of Social Media and Intelligent Risk.
• The sub-areas listed within the items will
now have Titles to help improve the outlines
and as a guide for better organization.
14
What is a Key Theme?
 A high level perception or observation, relevant to the
applicant’s key factors, that recurs in the scorebook and is
expanded upon with evidence during a site visit. Key
themes are often common to more than one category or
item. They may address one or more Baldrige core value.
 A role model result, practice, process, or innovation, that
might be used by other organizations, or an observation so
important, strength or OFI, process or result, that it should
be brought to leadership’s attention.
 The executive summary of the feedback report. It may be
the only part of the feedback report that some people in
the organization read.
Cross Cutting Themes
• In the early days of the Baldrige, Key Themes
were called Cross Cutting Themes, or Cross
Cutting Issues, because they described topics
that cut across the categories and items.
Core Values-The Foundation of the
Baldrige Criteria
•
•
•
•
•
•
•
•
•
•
•
Visionary Leadership
Customer driven excellence
Organizational and personal learning
Valuing workforce members and
partners
Agility
Focus on the future
Managing for innovation
Management by fact
Societal responsibility
Focus on results and creating value
Systems perspective
• If you think of the 7
Baldrige categories as
the vertical beams in a
house frame, the 11
Baldrige Core Values are
the cross members that
provide strength to the
frame.
Process for using Baldrige Core Values
to Develop Key Themes
• Each Team Member:
• Review the applicant
• Read the Baldrige Core Values
• Write 11 post-its rating the applicant against each Baldrige Core
Value with a +, 0, or – and write a few short thoughts for each
core value explaining why you rated the applicant as you did.
• Key Themes Lead
• Sum the plusses and the minuses. Core Values that have many
plusses, or minuses, may be candidates for Key Themes.
Typically only a few will stand out.
• Report results to team.
• Draft Key themes for these high scoring values using feedback
report comments and post-it thoughts as inputs.
Using Core Values to ID Key Themes
Baldrige Core Value
Score (+,0,-)
Visionary Leadership
+,+,+,+,+,+ (6+) Probable KT
Customer driven excellence
+,+,0,0,0,-
Organizational and personal learning
+,+,+, 0,0,0
Valuing workforce members and partners
+,+,+,+,+,- (5+) Possible KT
Agility
+,+,0,0,-,-
Focus on the future
+,+,+,0,-,-
Managing for innovation
0,0,-,-,-,-
Management by fact
-,-,-,-,-,-, (6-) Probable KT
Societal responsibility
+,+,+,+,+,0 (5+) Possible KT
Focus on results and creating value
0,-,-,-,-,- (5-) Possible KT
Systems perspective
+,+,0,0,-,-
Caveat to this approach
• While the Baldrige Core Values are the
foundation upon which the Criteria are
based, they are not examination items. Key
themes are written based on comments
developed relating to the criteria. The
feedback report comments MUST support
the Key Themes.
Six-Step Evaluation Process
Step 1: Ground yourself in the Criteria
Item
Item Question Organization
Glossary
Understanding the Questions
•
The quality of the feedback provided depends on understanding of
the meaning and significance of the questions
•
•
A Glossary of Key Terms is provided; Terms with definitions are
identified with CAPs
•
•
Open your criteria, find a few Key Terms and look up the definition
Notes are provided to clarify key requirements, give instructions,
and indicate important linkages
•
•
The Criteria book provides aids to understanding
Read N1 on Item 1.1 (page 7) to find an expectation concerning the
applicant’s Vision
Category and Item Descriptions discuss the purpose for the
questions and comment on expectations for answers
•
Read the first Comment in Section 1.1 and consider how it will affect
the applicant’s discussions in 1.1
3. WSQA
Assessment Process
Objective:
Introduce the Assessment process
What Examiners Do…
Read Criteria
Meet with co-examiners
to consolidate analysis
Read Applicant
Response
Analyze
Response
Visit the Applicant
to clarify/verify
Record Analysis
Co-write an analysis
report
Preparing for independent review
• Read the criteria
• Read the application
• Familiarize yourself with the online tool
http://www.wsqa.net/extraining.php
• Check in with Team Lead or Mentor after preparing
a few Key Factors
27
Independent review:
Process Items
Objective:
Be able to diagnose a process item
for independent review
Independent review steps - process
1.
2.
3.
4.
5.
Review related criteria
Read the category under review, take notes
Select relevant key factors from list
Identify & record processes
Record observations on worksheet under strengths
and Opportunities
6. Indicate
•
•
•
•
Approach
Deployment
Learning
Integration
29
Identify and record processes
• “Process” includes
•
•
•
•
•
•
•
Inputs
Steps (related activities)
Time frames
Outputs
End users (internal/external customers)
Standards
Key measures to evaluate and improve the process
• Not always formal or named – you might have to
read between the lines
• Look to criteria for guidance
30
Identify and record processes
“Systematic” refers to processes that
•
•
•
•
Are well-ordered
Are repeatable
Use data and information so learning is possible
Build in the opportunity for evaluation,
improvement, and sharing
31
Identify and record processes
• “Effective” refers to how well a process or measure
addresses its intended purpose
• Determining effectiveness requires evaluating
• How well the approach is aligned with the organization’s
needs
• How well it is deployed, and/or
• How the evaluation of the outcome of its measures are
used, i.e., for learning and improvement
32
Indicating ADLI- Approach
• “Approach” refers to
• Methods used to accomplish the process
• How appropriate the methods are to the item
requirements
• The effectiveness of their use
• The degree to which the process is repeatable
and is based on reliable data and information
(i.e., systematic)
33
Indicate “ADLI”- Deployment
“Deployment” refers to the extent to which
• The approach is applied in addressing relevant
and important item requirements
• It is applied consistently
• It is used by all appropriate work units
34
Indicate “ADLI”- Learning
“Learning” refers to
• Refining the approach through cycles of
evaluation and improvement
• Encouraging breakthrough change to your
approach (innovation)
• Sharing refinements and innovations with other
relevant work units and processes
35
Indicate“ADLI”- Integration
“Integration” refers to the harmonization of
•
•
•
•
•
•
•
Plans
Processes
Information
Resource decisions
Actions
Results
Analyses
to support key, organization-wide goals
36
Indicate ADLI- Integration
Questions to ask in analyzing “Integration”
• Do the individual components of a performance
management system operate as a fully
interconnected unit?
• Is the approach aligned with the organizational
needs identified in the Organizational Profile?
37
ADLI-key concepts
Integration examples:
• Alignment of objectives and action plans with
strategic challenges AND mission, vision, values
• Alignment of product/service delivery methods
with KEY customer requirements
• Alignment of key process measures with KEY
customer requirements
38
Scoring Categories 1 – 6 (Process
Categories)
• Examiners select a score (0-100) to
summarize their observed strengths and
opportunities for improvement (OFI’s)
• Scoring Guidelines are provided for
Approach, Deployment, Learning, and
integration
• Scores are assigned for each ITEM
Scoring Guidelines for Approach
SCORE
0% -5%
PROCESS
No SYSTEMATIC APPROACH is evident; information is
ANECDOTAL. (A)
10% - 25%
The beginning of a SYSTEMATIC APPROACH to the
BASIC REQUIREMENTS of the Item is evident. (A)
30% - 45%
An EFFECTIVE, SYSTEMATIC APPROACH, responsive to the
BASIC REQUIREMENTS of the Item, is evident. (A)
50% - 65%
An EFFECTIVE, SYSTEMATIC APPROACH, responsive to the
OVERALL REQUIREMENTS of the Item, is evident. (A)
70% - 85%
An EFFECTIVE, SYSTEMATIC APPROACH, responsive to the
MULTIPLE REQUIREMENTS of the Item, is evident. (A)
90% - 100%
An EFFECTIVE, SYSTEMATIC APPROACH, fully responsive to the
MULTIPLE REQUIREMENTS of the Item, is evident. (A)
Category/Item Question Organization
BASIC
OVERALL
MULTIPLE
Scoring Guidelines for DEPLOYMENT
SCORE
0% -5%
PROCESS
Little or no DEPLOYMENT of an APPROACH is evident.
10% - 25%
The APPROACH is in the early stages of DEPLOYMENT in most areas
or work units, inhibiting progress in achieving the BASIC
REQUIREMENTS of the Item.
30% - 45%
The APPROACH is DEPLOYED, although some areas or work units are
in early stages of DEPLOYMENT.
50% - 65%
The APPROACH is well DEPLOYED, although DEPLOYMENT may vary
in some areas or work units.
70% - 85%
The APPROACH is well DEPLOYED, with no significant gaps.
90% - 100%
The APPROACH is fully DEPLOYED without significant weaknesses or
gaps in any areas or work units.
Scoring Guidelines for LEARNING
SCORE
0% -5%
PROCESS
An improvement orientation is not evident; improvement is achieved through
reacting to problems.
10% - 25%
Early stages of a transition from reacting to problems to a general
improvement orientation are evident.
30% - 45%
The beginning of a SYSTEMATIC APPROACH to evaluation and improvement
of KEY PROCESSES is evident.
50% - 65%
A fact-based, SYSTEMATIC evaluation and improvement PROCESS and some
organizational LEARNING are in place for improving the efficiency and
EFFECTIVENESS of KEY PROCESSES.
70% - 85%
Fact-based, SYSTEMATIC evaluation and improvement and organizational
LEARNING are KEY management tools; there is clear evidence of refinement
and INNOVATION as a result of organizational-level ANALYSIS and sharing.
90% - 100%
Fact-based, SYSTEMATIC evaluation and improvement and organizational
LEARNING are KEY organization-wide tools; refinement and INNOVATION,
backed by ANALYSIS and sharing, are evident throughout the organization.
Scoring Guidelines for INTEGRATION
SCORE
PROCESS
0% -5%
No organizational ALIGNMENT is evident; individual areas or work
units operate independently.
10% - 25%
The APPROACH is ALIGNED with other areas or work units largely
through joint problem solving.
30% - 45%
The APPROACH is in early stages of ALIGNMENT with your basic
organizational needs identified in response to the Organizational
Profile and other Process Items.
50% - 65%
The APPROACH is ALIGNED with your organizational needs identified
in response to the Organizational Profile and other Process Items.
70% - 85%
The APPROACH is INTEGRATED with your organizational needs
identified in response to the Organizational Profile and other Process
Items
90% - 100%
The APPROACH is well INTEGRATED with your organizational needs
identified in response to the Organizational Profile and other Process
Items.
Overall Score for Categories 1 – 6
Items
• An applicant’s score for approach [A], deployment
[D], learning [L], and integration [I] depends on
their ability to demonstrate the characteristics
associated with that score
• Although there are 4 factors to score for each
Item, only one overall score is assigned
• The examination team selects the “best fit” score,
which is likely to be between the highest and
lowest score for the ADLI factors
“Best Fit” Example
Consider two applicants, both scoring 60% for A, D, and I,
and 20% for L
• Applicant
Market Growth
Technology
A
10%/year
Stable
B
2X/year
Major change
• Item Score:
• Applicant A:
50%
(Learning could lead to incremental results
improvements)
• Applicant B:
30%
(Learning needed to sustain the organization)
Competitors
Stable
Many new
Step 5: Draft feedback-ready
comments
Step 6: Determine the scoring range
and score for the Item
Independent Process Review
• Refer to samples
49
6. Independent review:
LeTCI – Results Items
Objective:
Be able to evaluate a results item
for independent review
Using a Table of Expected Results to
Identify Category 7 OFI’s
Purpose:
Identify results that the applicant hasn’t provided
WHY?
Because applicants like to show results that are favorable, but may not
always show results that are important.
Table of Expected Results
Title of Expected
Result
Source Reference Category 7
Reference
Result Reference
(Describe expected
result and
segments)
(List where in the
application you
learned this was
important to report
in Cat 7)
(List where in Cat 7
the result belongs)
(Identify the page
or Figure that
contains the
results)
Examples:
Examples:
Examples:
Status of Action Plans
(% of Action Plans on
target)
OP pg.7
1, 2.1a2
Not found
Workforce segments
OP p6
7.4 Expect to see
segmentation by FT,
PT and on-call staff
Found in 7.4-4, 6, 10
Not found in 7.4-7, 8,
9
Process for using a
Table of Expected Results
As you read the application, pay attention for processes that the
applicant cites as important, measures that are discussed,
segments that are used, etc.
As you find examples of these, include them in the table of
expected results in columns 1-3.
When you review Category 7, review it for the expected results
and complete column 4.
Based on the comments in Column 4 of the table, an OFI, or OFI’s
can be drafted citing missing expected results.
Note: These are results that the applicant has introduced an
expectation for in the Organizational Profile and Categories 1-6.
They are NOT results that the examiner would like to see.
Evaluation Factors for Category 7
(results)
•
Le = Performance Levels
Numerical information that places an organization’s results on a meaningful
measurement scale. Performance levels permit evaluation relative to past
performance, projection goals, and appropriate comparisons.
•
T = Trends
Numerical information indicating the direction, rate and breadth of
performance improvements. A minimum of 3 data points is needed to begin
to ascertain a trend. More data points are needed to define a statistically
valid trend.
•
C = Comparisons
Establishing the value of results by their relationship to similar or equivalent
measures. Comparisons can be made to results of competitors, industry
averages, or best-in-class organizations.
•
I = Integration
Connection to important customer, product and service, market, process
and action plan performance measurements identified in the Organizational
Profile and in Process Items.
•
G = Gaps
Absence of results addressing specific areas of Category 7 Items, including
the absence of results on key measures discussed in Categories 1–6
Results Evaluation Factors
95
Le
Denotes
“Good”
Trend
direction
Percentile
90
85
80
75
Trend
70
65
60
55
Comparison
50
45
93-94
95-96
97-98
MCB Percentile
National Mean
99-00
01-02
03-04
Top 10%
MCB Trend
Results strengths and OFI;s
• Strengths are identified if the following is observed:
• Performance levels [Le] are equivalent or better than
comparatives, and/or benchmarks
• Trends [T] show consistent improvement, and
• Results are linked [Li] to key requirements
• Opportunities for improvement are identified if:
• Performance levels [Le] are not as good as
comparatives
• Trends [T] show degrading performance
• Comparisons are not shown
• Results are not linked [Li] to key requirements
• Results are not provided [G] for key processes and/or
action items
Scoring a Results Item
• Scoring a Results item is similar to a
Process Item
• Scoring Guidelines for Le, T, C, and I
• A single “best fit” score is selected for
each results item
Results Scoring for Level [Le]
SCORE
0% -5%
RESULTS
There are no organizational PERFORMANCE RESULTS or poor
RESULTS in areas reported.
10% - 25%
A few organizational PERFORMANCE RESULTS are reported; there
are some improvements and/or early good PERFORMANCE LEVELS
in a few areas.
30% - 45%
Improvements and/or good PERFORMANCE LEVELS are reported in
many areas addressed in the Item requirements.
50% - 65%
Improvement TRENDS and/or good PERFORMANCE LEVELS are
reported for most areas addressed in the Item requirements.
70% - 85%
Current PERFORMANCE LEVELS are good to excellent in most areas
of importance to the Item requirements.
90% - 100%
Current PERFORMANCE LEVELS are excellent in most areas of
importance to the Item requirements.
Results Scoring for Trend [T]
SCORE
0% -5%
RESULTS
TREND data either are not reported or show mainly adverse
TRENDS.
10% - 25%
Little or no TREND data are reported, or many of the trends shown
are adverse.
30% - 45%
Early stages of developing TRENDS are evident.
50% - 65%
No pattern of adverse TRENDS and no poor PERFORMANCE LEVELS
are evident in areas of importance to your organization’s KEY
MISSION or business requirements.
70% - 85%
Most improvement TRENDS and/or current PERFORMANCE LEVELS
have been sustained over time.
90% - 100%
Excellent improvement TRENDS and/or consistently excellent
PERFORMANCE LEVELS are reported in most areas.
Results Scoring for Comparison [C]
SCORE
0% -5%
RESULTS
Comparative information is not reported.
10% - 25%
Little or no comparative information is reported.
30% - 45%
Early stages of obtaining comparative information are evident.
50% - 65%
Some TRENDS and/or current PERFORMANCE LEVELS—evaluated
against relevant comparisons and/or BENCHMARKS—show areas of
good to very good relative PERFORMANCE.
70% - 85%
Many to most reported TRENDS and/or current PERFORMANCE
LEVELS—evaluated against relevant comparisons and/or
BENCHMARKS—show areas of leadership and very good relative
PERFORMANCE.
90% - 100%
Evidence of industry and BENCHMARK leadership is demonstrated
in many areas.
Results evaluation: key concepts
•
Integration
•
•
•
Results align with key factors, e.g.,
• Strategic challenges,
• Workforce requirements
• Vision, mission, values
Results presented for
• Key processes
• Key products, services
• Strategic accomplishments
What examples can you think of?
•
•
Strong integration?
Not-so-strong integration?
61
Results evaluation: key concepts
•
Strong integration: results presented for
•
•
•
•
Key areas addressing strategic challenges,
Key competitive advantages
Key customer requirements
Not-so-strong integration:
•
•
Results missing for the above
Results presented that the Examiner can’t match to
process items or Organizational Profile
62
Results Scoring for Integration [I]
SCORE
0% -5%
RESULTS
RESULTS are not reported for any areas of importance to your
organization’s KEY MISSION or business requirements.
10% - 25%
RESULTS are reported for a few areas of importance to your
organization’s KEY MISSION or business requirements.
30% - 45%
RESULTS are reported for many areas of importance to your
organization’s KEY MISSION or business requirements.
50% - 65%
Organizational PERFORMANCE RESULTS address most KEY
CUSTOMER, market, and PROCESS requirements
70% - 85%
Organizational PERFORMANCE RESULTS address most KEY
CUSTOMER, market, PROCESS, and ACTION PLAN requirements.
90% - 100%
Organizational PERFORMANCE RESULTS fully address KEY
CUSTOMER, market, PROCESS, and ACTION PLAN requirements.
“Key” results
• Refers to elements or factors most critical to
achieving the intended outcome
• In terms of results, look for
• Those responsive to the Criteria requirements
• Those most important to the organization’s success
• Those results that are essential elements for the
organization to pursue or monitor in order to achieve its
desired outcome
64
Step 4:Draft around six strengths and
opportunities for improvement
Step 5: Draft feedback-ready
comments
Results worksheet example
• REFER TO EXAMPLE IN MANUAL
67
Writing Well Written
Comments
How do good comments add value?
Organization
Profile
Application
content
Criteria
The most valuable Comments/Key Themes
69
The Structure of Well Written Comments
Well written comments incorporate criteria
language, ADLI/LeTCI, scoring language, and
application language.
70
Well Written Comment Guidelines
Category 1-6
 Does the comment link to the Criteria?
 Does it consist of 2-3 grammatically correct sentences
(about 1000 characters)?
 Does it use examples from the application that illustrate
the OFI/Strength?
 Does it include a so what if it is an OFI?
 Is the statement non-judgmental and non-prescriptive?
 Does the comment link to the scoring guidelines?
71
Well Written Comment Guidelines
• Tie your comments to the basic, overall, or multiple level Criteria
requirements that are most important to the applicant.
• Write process comments so they contain: a subject identified from
the criteria, the application, or the scoring guidelines; verb(s) and
requirements from the Criteria; examples from the application, and
figure numbers, as appropriate.
• Use the evaluation factors (ADLI or LeTCI), and scoring language, to
clearly articulate the areas of the applicant’s response to clarify the
strength or provide insights that will help the applicant improve
overall organizational effectiveness and capabilities.
• Draw linkages between an Item and the applicant's Organizational
Profile.
• Ensure that the comment does not contradict other comments in the
same or other Items or in the Key Themes Worksheet.
72
The Style of Well Written Comments
• Use active voice and present tense (e.g., 'completes' rather than 'is
completed').
• Use a polite, professional, and positive tone.
• Use vocabulary/phrasing from the Criteria and the Scoring
Guidelines.
• Describe what is missing if something "is not clear". “Not clear”
comments must be resolved if the applicant has a site visit. If they
don’t do it based on site experience then say they don’t do it.
• In lieu of the applicant’s name, use such phrases like "the applicant"
or "the organization" to refer to the applicant in the independent
and consensus scorebooks. Use the applicant's name in final
feedback reports.
• Use the applicant's language/terminology when appropriate.
73
“Don’ts” to avoid in comments
1. Avoid parroting
2. Avoid being prescriptive
3. Avoid going beyond the criteria
74
“Do’s” for well-written comments
(refer to comment guideline in workbook)
75
Example Strength Comment
• The applicant identifies their core competencies as the
participating dentist network; expert claims administration;
extraordinary customer service and performance culture. A
SWOT analysis is used in the annual strategic planning process
to identify core competencies. Correlation analysis is
performed on satisfaction survey data and is used to
determine key drivers of customer satisfaction. Initiative
action plans accentuate core competency strengths.
Deployment is evidenced by the use of bi-annual CAG
meetings, ensuring alignment with the mission and the
integration of data for easy access across all departments and
customer segments
• Red-Criteria Green-ADLI Blue-Application Pink-Key Factor
76
Well Written Comments Don’t….
•
•
•
•
•
•
•
Go beyond the requirements of the Criteria or assert your
personal opinions.
Be prescriptive by using "could", "should", or "would".
Be judgmental by using terms such as "good", "bad", or
"inadequate".
Tell the applicant that they aren’t doing enough of something
Comment on the applicant's style of writing or data
presentation.
Use jargon or acronyms unless they are used by the applicant
Only “Parrot” the application or the Criteria. Provide only
enough language to add clarity—seek to add value rather than
restate information.
77
How Could This Comment Be Improved?
Application language:
• Share Food uses a biennial 12 step Strategic Planning Process that
involves community leaders, volunteers, member agencies, and
donors. Strategic challenges are identified in the “Current State”
step which includes inputs from the SWOT analysis, environmental
scan, and funding mandate review. The Strategic Planning Process
was initiated in 1997 and restructured in 2004 as a result of
benchmarking analysis of food banks and other nonprofit
organizations, and feedback from the state award process.
Comment:
• Share Food uses a biennial 12 step Strategic Planning Process (SPP)
that involves community leaders, volunteers, member agencies, and
donors. Strategic challenges are identified in the Define step which
includes inputs from the SWOT analysis, environmental scan, and
funding mandate review. This is proof that the organization has a
great systematic approach to the SPP and extensive cycles of
learning in 1997, and 2004.
78
How Could This Comment Be Improved?
Application language:
• The manager’s vision helps Our Store to maintain a strong
focus on customer satisfaction. The manager has worked
hard to instill the belief that leadership and responsibility
should be present at all levels. This rewards everyone at Our
Store with the responsibility of planning, carrying out, and
evaluating their own work.
Comment:
• Our Store is in the beginning stages of building an approach to
setting a vision for customer satisfaction, this includes the
manager working very hard to instill the beliefs in leadership
and responsibility. Everyone in the store receives a reward
and this will ensure that everyone will have ownership in the
organization and the ability to evaluate their own work, which
should improve performance overall.
79
How Could This Comment Be Improved?
Application language:
• Our Hospital builds relationships with customers through a variety of
methods including patient enrollment, which serves as an
orientation to enabling services; Patient-Family Advisory Boards,
which have recommended and implemented improvements to
patient care; and high-visibility community health activities. Using
patient surveys, conducted randomly after discharge, Our Hospital
determines the issues that lead to poor relationship results and uses
this information to implement new relationship-building processes.
The Patient-Family Advisory Board is an example of the results of this
process.
Comment:
• Customer relationships are built through a variety of methods
including Patient enrollment, Patient-Family Advisory Boards, and
community health activities. Surveys are conducted randomly after
patients are discharged. The Patient-Family Advisory Board is used to
determine issues that lead to poor relationships and implements
new relationship-building processes.
80
How Could This Comment Be Improved?
Application language:
• Customer groups are identified annually as part of the Strategic
Planning Process using Our Hospital’s vision/mission/values as a
focus. By analyzing demographic data from multiple sources, the
cross-location Service with Spirit Team (SWST) is able to identify the
gaps, look at disparities, and identify potential customers.
Deployment of this process through multiple cycles has resulted in
the development of several services designed to meet the unmet
needs of customers in the Our Hospital’s service area.
Comment:
• Through the SPP, customer groups are annually identified using the
MVV as a focus. The SWST uses demographic data to identify gaps,
disparities and identify potential customers. This process has been
deployed throughout the organization, and through multiple cycles
of refinement the customers needs in the service area have been
met.
81
Well Written OFI Comment Guidelines
• May be used to give credit to the applicant for some areas that
HAVE been addressed
• “While the applicant does x, y, z, it is not evident that a, b,
and c are performed.”
• May begin with a statement about what is missing:
• “There is no evidence that…”
• “There is no indication of how…”
• “It is unclear…”
• “It is not apparent that there is a …”
• Must tie to criteria language, (ADLI), and/or to Key Factors that
the applicant has identified in the Organizational Profile
• Must include a “so what” (Why does the applicant care about
this? Why would they take action?)
• Usually these come from Key Factors / Org Profile. The closer
the ‘so what’ can be to identifying the applicant’s specific
needs, the better.
82
Example OFI Comment
• Although cross functional teams are chartered and aligned with the
SPP, there is no indication of processes used by the teams or otherwise
by the applicant to design and innovate its work processes to meet key
requirements. There is no indication of how design and innovation
processes address opportunities for agility and improvements in cycle
time and productivity. It is also unclear how the team work process
improvement approach is evaluated for annual process changes or
improvements. An annual evaluation of strategic team alignment and
improvements may provide efficiency and effectiveness in the
accomplishment of strategic objectives, initiatives, and action plans.
• Red-Criteria Green-ADLI Blue-Application Pink-Key Factor
83
How Could This Comment Be Improved?
Application language:
• Share Food uses a biennial 12 step Strategic Planning Process that
involves community leaders, volunteers, member agencies, and
donors. Strategic challenges are identified in the “Current State”
step which includes inputs from the SWOT analysis, environmental
scan, and funding mandate review. The Strategic Planning Process
was initiated in 1997 and restructured in 2004 as a result of
benchmarking analysis of food banks and other nonprofit
organizations, and feedback from the state award process.
Comment:
• Although Share Food uses a biennial 12 step Strategic Planning
Process (SPP) that involves community leaders, volunteers, member
agencies, and donors, there is no evidence of how blind spots are
identified, nor how time horizons are set and how the SPP addresses
those horizons. If the applicant implemented a more extensive
benchmarking process that identified blind spots and included time
horizons, there may be more of an opportunity for this organization
to have a more successful Strategic Planning Process.
84
How Could This Comment Be Improved?
Application language:
• The manager’s vision helps Our Store to maintain a strong
focus on customer satisfaction. The manager has worked
hard to instill the belief that leadership and responsibility
should be present at all levels. This rewards everyone at Our
Store with the responsibility of planning, carrying out, and
evaluating their own work.
Comment:
• Although the manager has worked hard to instill the belief
that leadership and responsibility should be present at all
levels, there is no indication of a systematic approach, nor is
there deployment at any level, no integration, nor are there
any cycles of learning.
85
How Could This Comment Be Improved?
Application language:
• Our Hospital builds relationships with customers through a variety of
methods including patient enrollment, which serves as an
orientation to enabling services; Patient-Family Advisory Boards,
which have recommended and implemented improvements to
patient care; and high-visibility community health activities. Using
patient surveys, conducted randomly after discharge, Our Hospital
determines the issues that lead to poor relationship results and uses
this information to implement new relationship-building processes.
The Patient-Family Advisory Board is an example of the results of this
process.
Comment:
• There is no systematic approach to building customer relationships,
or in conducting patient surveys. And although there is a PatientFamily Advisory Board there is no approach described for how issues
are determined and how new relationship-building processes are
developed.
86
How Could This Comment Be Improved?
Application language:
• Customer groups are identified annually as part of the Strategic
Planning Process using Our Hospital’s vision/mission/values as a
focus. By analyzing demographic data from multiple sources, the
cross-location Service with Spirit Team (SWST) is able to identify the
gaps, look at disparities, and identify potential customers.
Deployment of this process through multiple cycles has resulted in
the development of several services designed to meet the unmet
needs of customers in the Our Hospital’s service area.
Comment:
• Although customer groups are identified annually in the SPP, and the
Spirit Team (SWST) identifies gaps, disparities, and identifies
potential customers, there is no indication how this information is
deployed throughout the organization and how the information is
used to meet the unmet needs of customers in the area.
87
Comment Writing Exercise
• Each person at the table select an independent
review item (Cat 1-6 strength or OFI)
• Write a well written comment
• Swap scorebooks
• What feedback would you give to this examiner
• Use the comment checklist for examples
• Practice giving feedback to each other
• Report out the most significant key learnings
• (If time swap with to another person at the
table)
88
Category 7 Comment Guidelines
Category 7
 Are specific results (numbers) used rather than
generalities?
 Have trends, comparisons and/or appropriate
benchmarks been considered?
 Does the comment include figure numbers?
 Have we summarized multiple charts with a consistent
theme (trends, missing, comparisons, etc.)
89
(Well Written Result Comment Example
There is a lack of comparative data in a number of
the Results sub-categories; as an example, no
comparative data is available for measures such as
Discharge Weight of ELBW Infants (7.1-4),
Improvement % of ELBW Infants (Figure Figure 7.15), % of Breast Milk Available (Figure Figure 7.1-6),
Outpatient Lactation (7.1-7) and Catheter Related
Bloodstream Infections (7.1-8), Survival to Discharge
by Rhythm (Figure Figure 7.1-17), and Well Child
Visits Figure (7.1-19). The lack of comparative
results may impact the value of Excellence - setting
and achieving the highest standards.
90
How Could This Comment Be Improved?
The applicant’s market share results for key service lines is
strong, exceeding all comparative data across Discharges;
Inpatient Surgeries; the applicant's Heart Services; and
Neurosurgery Market. The applicant enjoys market
dominance in these service lines and has shifted its focus to
adolescent care and services to new geographic markets. The
applicant has received numerous recognitions and awards for
their leadership in these markets and service lines. These
successes are important to their goal of achieving national
recognition as a premier hospital, a tertiary and quaternary
hub, and preferred provider of regional health care services.
91
How Could
These Comments Be Improved?
• The applicant's Net Income from Operations (7.3.8) is above
the comparison median and trending up over the past three
years and overall applicant’s turnover rate has been
decreasing.
• The applicant’s Net Sales between 2006 and 2008, have
increased by 10% in 2006 to over 12.2% in 2008. The applicant
is consistently above the goal of 9% for all three years, but is
below the Baldrige comparison of 15% for all three years.
• Although the turnover rate for RNs has been increasing
between 2006 and 2008. The turnover rate for RNs was 7.7%
in 2006 and 10.3% in 2008. There have been substantial
improvements in physician satisfaction from 13% to over 25%
during the same time period.
92
How Could
These Comments Be Improved?
• The applicant's Net Income from Operations (7.3.8)
is below the comparison median and trending down
over the past three years and overall applicant’s
turnover rate has been increasing between 2006
and 2008, from 10% in 2006 to 12.2% in 2008.
• The turnover rate for RNs has been increasing
between 2006 and 2008. The turnover rate for RNs
was 7.7% in 2006 and 10.3% in 2008.
93
Comment, scoring considerations
• For LITE, only one score for entire results
Category 7
• Propose a score for results related to your
categories
• Be prepared to reach team consensus for all of
Category 7
• Using scoring language can help the
applicant understand comments and score
94
Category feedback
• Check and provide feedback on whether Cat. Lead’s
comments:
•
•
•
•
•
•
•
Include criteria requirements
Provide meaningful feedback
Address ADLI or LeTCI
OFI’s provide relevant “so what” (at least some of them)
Results cite figure numbers/titles and specifics
References to application are correct
Recommended score reflects comments in content and
balance (strengths:OFI’s ratio)
• Back-up does this for full, team does this for lite
95
Information resources
• WSQA comment guideline handout
• Your back-up, team lead, PEPD mentor
• Reference materials in
• Your workbook: comment and scorebook examples
• http://www.wsqa.net/extraining.php
• Baldrige web site
http://baldrige.nist.gov/Examiner_Resources.htm
96
Six-Step Evaluation Process
Key themes
Objectives:
1. Be able to identify Key Themes
2. Be able to develop a Key Theme
comment
How do good comments add value?
Organization
Profile
Application
content
Criteria
The most valuable Comments/Key Themes
99
What is a Key Theme?
A “key theme” is a strength or OFI that
• Addresses a central requirement of the
Criteria,
• Is common to more than one Item or
Category (crosscutting),
• Is especially significant in terms of the
applicant’s key factors, and/or
• Addresses a Core Value of the Criteria
100
Developing Key Theme comments
• Address all double strengths and OFI’s individually
and/or in groups
• Address related comments across categories
• Develop 6-10 (Guideline!) key themes each for
significant
•
•
•
•
Process strengths
Process OFI’s
Results strengths
Results OFI’s
• See example Key Themes in the workbook for this
class
101
Key Theme Development
using the Baldrige Core Values
 Purpose:
Create Key Themes at the end of the consensus meeting or at
the end of a site visit
WHY?
to identify Key Themes that cut across categories and items
Managing your time
• Put aside time in your
calendar (refer to time
commitments)
• Work with your lead &
team
• Set meetings early
• Coordinate
• Communicate
• Stories to tell?
103
Managing your time
• Schedule work hours
• Break tasks into little
pieces
• Reward yourself for wins
• Move quickly past
setbacks
• Call your lead
• Call your back-up
104
Software
• Link to software
http://www.wsqa.net/extraining.php
• Username and Password: firstlast name
• Link for software manual and taped webinar
http://www.wsqa.net/Extimetable.php
Click on scorebook, select Key factors, then
select Item evaluation
Save Frequently, Save OFTEN, Save A Lot!save in software- not on your browser
105
Your questions?
106
Your feedback, please!
• Please complete surveys before leaving
107
THANK YOU!!
Your support and participation as
Examiners helps us all by helping WSQA
fulfill its mission!
108