Transcript Malcolm Baldrige National Quality Award

Welcome, Examiners!
Washington State Quality Award
Return Examiner Training Webinar
2009
First things first!
• THANK YOU!
• WSQA’s mission: Improving the way we live, learn,
and work in Washington by helping organizations
improve through the use of the Baldrige Criteria
for Performance Excellence
WSQA Return Examiner 2009
2
Objectives For This Class
• Review Changes to the Baldrige Criteria
• Strengthen Skills in Areas of Opportunity
• Review Resources for Examiner Software
WSQA Return Examiner 2009
3
Agenda:
1.
2.
3.
4.
5.
WSQA organization overview
Baldrige criteria changes
WSQA ethics review
WSQA process Full Vs Lite
Review process
1.
2.
3.
6.
7.
Individual review worksheets
Consensus process
Improvements
Examiner Software
Summary
WSQA Return Examiner 2009
4
For a Successful Course…
•
•
•
•
•
•
Be an active participant
Give only constructive feedback
Be courteous to your colleagues
Honor diverse opinions
Seek to understand
Ask questions- Can use chat
capability, also just ask
• Have fun!!
Introductions
• Name, organization, what you do
• Questions and expectations?
• WSQA or National examiner or applicant
experience?
• What do you like to do when
you’re not working?
WSQA Return Examiner 2009
6
What is WSQA?
• Created by State law to:
• Promote excellence
• Recognize achievement
• Educate and train
• Patterned after the Baldrige National Quality Award
• Awards presented annually by the Governor
• Awarded to private, public, and not-for-profit
organizations in manufacturing, service, education,
and healthcare
WSQA Return Examiner 2009
7
What is WSQA?
WSQA is a 501(c)(3) nonprofit organization
supported by
•
•
•
•
Corporate donations
Individual memberships
Workshop, collaborative and conference fees
Many (many, many!) volunteer hours
WSQA Return Examiner 2009
8
Who is WSQA?
• Examiners
• Lite Assessment
• Full Assessment
• Certification follows assessment
• Senior Examiners
• Level 3 Team Leads
• Scorebook Editors
•
•
•
•
Panel of Examiner & Process Development (PEPD) Mentors
Panel of Judges
Board of Directors
Executive Director and support team
WSQA Return Examiner 2009
9
2. Baldrige criteria changes
Baldrige criteria booklet
Beginning in 2009 revised every 2 years
WSQA Return Examiner 2009
11
Baldrige 2007 Criteria Changes
• Strategic advantages
• Workforce engagement
• Leadership- performance measures
• Strategic Planning-innovation, strategic
advantages
WSQA Return Examiner 2009
12
Baldrige 2008 Criteria Changes
• Order of information in the booklet
• Scoring system description
• Diagrams added for core values and maturity
levels
• Added strategic advantages as a term to the
Glossary
WSQA Return Examiner 2009
13
2009 Criteria Changes
• Customer Focus/ Engagement
• Organizational Core Competencies
• Sustainability and Social Responsibilities
WSQA Return Examiner 2009
14
Customer Focus/ Engagement
• Customer commitment to your brand and
offerings
• Characterized by loyalty and willingness to
make an effort to do business with you,
willingness to do business with you
WSQA Return Examiner 2009
15
Core Competencies
• Areas of greatest expertise
• Strategically important
• Provide market place advantage
• May involve technologies, unique offerings,
market place niche, business acumen
WSQA Return Examiner 2009
16
Sustainability
• Organization’s ability to address business
needs and agility and strategy to address the
future
• Considers external and internal factors
• Considers impact on society and well being
of environment, social and economic
systems
WSQA Return Examiner 2009
17
Conflict of interest issues
• WSQA seeks to avoid conflict of interest to
protect process integrity
• Conflicts are frequent!
• First step in assignment process: determine
any conflicts
WSQA Return Examiner 2009
18
Code of Ethical Conduct
• Why are ethics of utmost importance to
WSQA and this process?
WSQA Return Examiner 2009
19
4. WSQA Full and Lite Process
Comparison table
WSQA Return Examiner 2009
20
5. WSQA Process
Objective:
Review key steps in the process. Review key
process opportunities.
Receive applications
Stages of the Assessment Process
Individual
Scorebooks
Stage 1:
Independent Review
Final product
Consensus
Scorebook
Stage 2:
Consensus Review
Lite or Full
application?
Lite
Editor
updates
scorebook
Consensus
Scorebook
Site visit
scorebook
Lead & Mentor
meet with
Judges Panel
WSQA
updates
scorebook
Feedback
report
Executive
Briefing
Full
Lead & Mentor meet
with Judges Panel
Selected for
site visit?
No
Yes
Stage 3:
Site Visit
Judges recommend
award level to
WSQA Board
WSQA Symposium
& Awards
Ceremony
Preparing for independent review
•
•
•
Read the criteria
Read the application
Familiarize yourself with the online tool
•
•
All information on the WSQA website: webinars and
manuals http://www.wsqa.net/extimetable.php
Check in with Team Lead on Key Factors and after
entering category 1 and associated results
WSQA Return Examiner 2009
23
Independent review
1.
2.
3.
4.
5.
6.
Select the most relevant Key Factors (from list)
Read the criteria for the category
Read (and annotate) the relevant section of the
application
Identify and record the applicant’s processes
Record observations. Note missing responses and
significant observations
Recommend a scoring range for the category
WSQA Return Examiner 2009
24
Independent review steps - process
1.
2.
3.
4.
5.
Review related criteria
Read the category under review, take notes
Select relevant key factors from list
Identify & record processes
Record observations on ADLI:
•
•
•
•
Approach
Deployment
Learning
Integration
6. Identify any significant or missing responses
WSQA Return Examiner 2009
25
Record observations on “ADLI”
• “Approach” refers to
• Methods used to accomplish the process
• How appropriate the methods are to the item
requirements
• The effectiveness of their use
• The degree to which the process is repeatable
and is based on reliable data and information
(i.e., systematic)
WSQA Return Examiner 2009
26
Record observations on “ADLI”
“Deployment” refers to the extent to which
• The approach is applied in addressing relevant
and important item requirements
• It is applied consistently
• It is used by all appropriate work units
WSQA Return Examiner 2009
27
Record observations on “ADLI”
“Learning” refers to
• Refining the approach through cycles of
evaluation and improvement
• Encouraging breakthrough change to your
approach (innovation)
• Sharing refinements and innovations with other
relevant work units and processes
WSQA Return Examiner 2009
28
Record observations on “ADLI”
“Integration” refers to the harmonization of
•
•
•
•
•
•
•
Plans
Processes
Information
Resource decisions
Actions
Results
Analyses
to support key, organization-wide goals
WSQA Return Examiner 2009
29
Record observations on “ADLI”
Questions to ask in analyzing “Integration”
• Do the individual components of a performance
management system operate as a fully
interconnected unit?
• Is the approach aligned with the organizational
needs identified in the Organizational Profile?
WSQA Return Examiner 2009
30
Process diagnosis: key concepts
Integration examples:
• Alignment of objectives and action plans with
strategic challenges AND mission, vision, values
• Alignment of product/service delivery methods
with KEY customer requirements
• Alignment of key process measures with KEY
customer requirements
WSQA Return Examiner 2009
31
Process diagnosis: key concepts
1. Approach
4. Integration
PROCESS
2. Deployment
3. Learning
WSQA Return Examiner 2009
32
Building the ADLI worksheet
Item
1.1a(2)
Process
Foster/
require
legal &
ethical
behavior
KF’s
#, #,
#...
A
D
++ Annual
overview of
obligations,
online training,
post-course
survey
+ Covers
staff, board,
partners,
volunteers
+ Daily
Huddle
addresses
concerns
quickly
L
- None
seen
I
+
Addresses
HIPAA,
patient
confidentiality
How would you score this single response?
WSQA Return Examiner 2009
33
Independent review steps - results
1. Review the Criteria to identify required results
2. Review the relevant process in the application to
identify KEY results important to the applicant (use
KF’s, application, your diagnosis)
3. Review the Results in the application
•
•
What IS there?
What ISN’T there that should be?
4. Evaluate & score on the basis of LeTCI
WSQA Return Examiner 2009
34
Results evaluation: key concepts
Levels
• Numbers that position results on a meaningful
measurement scale
• Permit evaluation relative to
•
•
•
•
Past performance
Projections
Goals
Appropriate comparisons
WSQA Return Examiner 2009
35
Results evaluation: key concepts
• Trends
• Numbers that indicate direction and rate of change
• Provide a time sequence of performance
• Require at least three data points
• Criteria call for trend data on
•
•
•
•
•
Product, service performance
Customer, workforce satisfaction
Financial performance
Market performance
Operational performance (cycle time, productivity)
WSQA Return Examiner 2009
36
Results evaluation: key concepts
Comparisons
• External requirements
• External benchmarks
• Similar OR different organizations with similar
characteristics
WSQA Return Examiner 2009
37
Results evaluation: key concepts
•
Integration
•
•
•
Results align with key factors, e.g.,
• Strategic challenges,
• Workforce requirements
• Vision, mission, values
Results presented for
• Key processes
• Key products, services
• Strategic accomplishments
What examples can you think of?
•
•
Strong integration?
Not-so-strong integration?
WSQA Return Examiner 2009
38
Results evaluation: key concepts
•
Strong integration: results presented for
•
•
•
•
Key areas addressing strategic challenges,
Key competitive advantages
Key customer requirements
Not-so-strong integration:
•
•
Results missing for the above
Results presented that the Examiner can’t match to
process items or Organizational Profile
WSQA Return Examiner 2009
39
“Key” results
• Refers to elements or factors most critical to
achieving the intended outcome
• In terms of results, look for
• Those responsive to the Criteria requirements
• Those most important to the organization’s success
• Those results that are essential elements for the
organization to pursue or monitor in order to achieve its
desired outcome
WSQA Return Examiner 2009
40
Break time!
Stages of the Assessment Process
Receive applications
Individual
Scorebooks
Stage 1:
Independent Review
Final product
Stage 2:
Consensus Review
Consensus
Scorebook
Lite or Full
application?
Editor
updates
scorebook
Lite
Consensus
Scorebook
WSQA
updates
scorebook
Feedback
report
Executive
Briefing
Full
Lead & Mentor meet
with Judges Panel
Selected for
site visit?
No
Yes
Stage 3:
Site Visit
Site visit
scorebook
Lead & Mentor
Judges recommend
meetReturn
with Examineraward
WSQA
2009 level to
Judges Panel
WSQA Board
WSQA Symposium
& Awards
Ceremony
WSQA Consensus: Step-by-step
Application
Criteria
Consensus
planning call
Independent
review
scorebooks
Observations
(from Ind.
Review)
Key Factors
Editor or Lead
Category Lead
Category Lead
1. Consolidate
scorebooks
2. Review and group
team’s observations
3. Select relevant Key
Factors
Single
consensus
scorebook
Grouped
observations
4-6 (3-5)
Relevant Key
Factors
For your category/
categories only
Numbers in
parentheses apply for
Lite assessment only
WSQA Return Examiner 2009
1
WSQA Consensus: Step-by-step
1
Grouped
observations
Below-the-line
observations
Back-up’s
comments &
feedback
Category Lead
Category Lead
Category Lead
4. Select and develop
comments
5. Write rationales for
below-the-line
observations
6. Exchange feedback
with Back-up, update
comments
About 6
comments
Rationales
Updated
comments
Below-the-line
observations
Scoring
recommendation
Lite: 4-6
WSQA Return Examiner 2009
2
WSQA Consensus: Step-by-step
2
Consensusready
comments
Draft
consensus
scorebook
Updated
Comments,
score
Team
Team
Editor
7. Review team
comments, prepare
notes, enter key
theme ideas
8. Hold consensus
call(s) or meetings
9. Prepare consensus
scorebook
Updated
consensus
scorebook
Key themes,
site visit
recommendation
Notes
Draft Key
Theme topics
FULL
only
FULL
only
WSQA Return Examiner 2009
Consensus
scorebook
End
Stage 2 Consensus
• Webinar
http://www.wsqa.net/extimetable.php
• Comments from all Independent reviews
• Ensure comments reflect scoring. - what
is preventing applicant from moving to
next level? Is this explained in the
comments
WSQA Return Examiner 2009
46
Consensus preparation and call(s)
• Each Examiner responds with agreement,
suggestions for changes, or disagreement,
stating rationales
• Category Lead facilitates consensus
• Team Lead and/or PEPD mentor leads final
scoring discussion
• Can take anywhere from 4-8 hours
(depending on what?)
WSQA Return Examiner 2009
47
Consensus and conclusion
• If using online tool during consensus,
changes may be made in real time
• If not, Category Leads update comments &
scores after call according to consensus
decisions
• Category Lead notifies Team Lead and
Scorebook Editor when done
• Scorebook Editor compiles consensus
scorebook with final edits, notifies WSQA
• Consensus scorebook should be finalized
ASAP
WSQA Return Examiner 2009
48
Comment, scoring considerations
• “Benefit of the doubt”
• Give credit for what’s in the application
• Don’t penalize for incidental exceptions
• Not every process must show complete “DLI”
• Not every results example must include “TCI”
WSQA Return Examiner 2009
49
Comment, scoring considerations
• For LITE, only one score for entire results
Category 7
• Propose a score for results related to your
categories
• Be prepared to reach team consensus for all of
Category 7
• Using scoring language can help the
applicant understand comments and score
WSQA Return Examiner 2009
50
Opportunities
•
•
•
•
•
•
Examiner training
Completing the stage 1 grid
Using the gaps and notes tab
Comment writing
Consensus Meetings
Scoring consistency
• Integration
• Learning
• Comparative Data
• Considerations for small organizations
WSQA Return Examiner 2009
51
Baldrige National Quality Program
National Examiner Questions
2009
Harry Hertz Responses
WSQA Return Examiner 2009
Question 1
• Further clarification of “performance
projections” vs “goals”
• Inclusion of goals in results category – why
are they not included in the evaluation?
• Why are projections expected for all
measures in all scoring ranges?
• Why are projections not in the Results
Criteria?
WSQA Return Examiner 2009
Question 2
• Why don’t we align results items with
process items? Currently 1.1 is aligned
with 7.6 instead of being aligned with 7.1.
Question 3
l
Without comparisons, how do you
interpret levels? Without a point of
reference, how can an Examiner
understand a “level”?
WSQA Return Examiner 2009
Question 4
l
How are key themes assessed (from
applicant’s perspective)?
Question 5
l
How do we evaluate applicants that
are part of a larger system? For
example an applicant might say that
they use a process established by the
system for which they are a part.
WSQA Return Examiner 2009
Question 6
• Why don’t we ask applicants to
define/describe their work system(s),
e.g., in 6.1.a.1, “What is/are your work
system(s)?”
Question7
l
Clarify the difference between work
systems and work processes. There
appears to be overlap and the
category is labeled process
management.
WSQA Return Examiner 2009
Core Competencies
WSQA Return Examiner 2009
Information resources
• WSQA comment guideline handout
• Your back-up, team lead, PEPD mentor
• Reference materials in
• Baldrige web site
http://baldrige.nist.gov/Examiner_Resources.htm
• WSQA Website
http://www.wsqa.net/extraining.php
WSQA Return Examiner 2009
58
What we’ve covered today
•
•
•
•
•
•
•
WSQA overview
Baldrige criteria changes
WSQA ethics review
WSQA Lite versus Full process
WSQA process review
WSQA process improvements
Questions from National Training
WSQA Return Examiner 2009
59
THANK YOU!!
Your support and participation as
Examiners helps us all by helping WSQA
fulfill its mission!
WSQA Return Examiner 2009
60