Transcript Tips for Writing Feedback Comments for Evaluators

EVALUATOR TIPS FOR REVIEW
AND COMMENT WRITING
The following slides were excerpted from
an evaluator training session presented as
part of the June 2011 CBFA Conference.
Special thanks to Dr. Annette Craven for
providing resources she received after
participating in a recent national Baldrige
examiner training.
Use of Standards & Criteria
The Standards and Criteria serve as the basis for:
Providing feedback to applicants in the form of
Process & Results
Strength comments
OFI comments
Process Maturity
Individual Review
Individual Review Process
Read Overview
Read Standards
One - Six
Evaluation Dimensions
Approach
Deployment
Results
Improvement (Learning)
Approach
Appropriateness of methods
Effectiveness of the methods
The degree to which the approach is
repeatable and based on reliable data
and information (i.e., systematic)
Systematic
Documented
Repeated
Evaluated
Improved
Deployment (Implementation)
the approach applied in addressing
the requirements is relevant and
important to the organization
the approach is applied consistently
the approach is used by all
appropriate work units
Results
•
Current performance
•
Rate and breadth of your performance
improvements
•
Performance relative to appropriate comparisons
and/or benchmarks
•
Linking results measures to important customer,
market, process, and action plan performance
requirements identified in the Overview and in
Approach-Deployment Items.
Where to Look for Results
• Results are found throughout the ACBSP Standards, for
example:
• Student/Stakeholder Satisfaction (Standard 3)
• Student Learning Outcomes (Standard 4)
• Faculty Deployment & Qualifications (Standard 5)
• Business Process Results (Standard 6)
• Performance is generally presented in one of three ways
Charts/Graphs
Tables
Narrative
What Should You Look For?
Expected Results
Appropriate Comparisons
Important Segmentation
Appropriate Comparisons
“Performance is examined relative to
competitors and/or other organizations
providing similar products and services”
“Include appropriate comparative
data”
Improvement (Learning)
Refining the approach through cycles of
evaluation and improvement
Encouraging breakthrough change to the
approach through innovation
Sharing refinements and innovations with
other academic departments or disciplines
and processes in your school of business
or institution
Writing Evaluator Observations
Self-study
Standards and Criteria Booklet
ACBSP handouts including evaluator
workbook
Observation Guidelines
Observations should be:
Standards and Criterion based
Relevant
Clear
Concise
Actionable
Observation Guidelines (Cont.)
Observations should not:
“Parrot” the self-study
Be prescriptive
Be judgmental
Conflict with one another
Strength Observations: 3 Components
Nugget
what the business unit does to merit the
score you assign to them
Example(s)
are the things the organization does with
some level of proficiency
Relevance
Why the observation is important for the
business unit (e.g. mission related, etc.)
Strength Observation Example
• “The development of the budget is an
integral part of the strategic planning
process. This ensures that the budget
reflects the business unit’s strategic
priorities. As a result, both long-term plans
and shorter range operational activities are
focused on the organization’s mission,
vision, and values.”
OFI’s
Is an opportunity for improvement
Is not necessarily a weakness
Is what prevents an applicant from
scoring at a higher level
Is tied to the Standards and Criteria
Example of an OFI Observation
• the business unit, through its strategic
planning process has identified four key
initiatives with associated goals. It is not
clear, however, how these goals are being
measured and the lack of an established
timeframe make it difficult to gauge progress
toward achieving them.
So What’s
Explains the significance of the
observation so that the business unit
doesn’t have to ask, “So what?”
Can be used with both Strengths and
OFI’s though more commonly used
with OFI’s
Where’s the So What?
• the business unit, through its strategic
planning process has identified four key
initiatives with associated goals. It is not
clear, however, how these goals are being
measured and the lack of an established
timeframe make it difficult to gauge progress
toward achieving them.
Where’s the So What?
• the business unit, through its strategic
planning process has identified four key
initiatives with associated goals. It is not
clear, however, how these goals are
being measured and the lack of an
established timeframe make it difficult
to gauge progress toward achieving
them.
Guidelines for Results
Start with a subject from the self-study or
the Criteria
Include the time frame you are writing
about -- such as “in 2009” or “from 2005 to
2010”
Include the actual numbers observed in the
levels or trends
Include figure references
Results Strength
• the business unit operates in a fiscally
sound manner as evidenced by its Aaa and
AAA bond ratings (Figure 7.3-1) and debt
capacity (Figures 7.3-8 and 7.3-11). These
results suggest that the business unit makes
financial decisions with its long-term
financial viability and sustainability in mind.
Results OFI
• Only 3 of the 13 figures in Standard 4,
Criterion 4.3 contain comparative data.
Without comparative data it is difficult
to evaluate the business unit’s current
level of performance or the trends they
report in these figures.
What’s Wrong with this Observation?
• “The standard being addressed is in
need of improvement.”
The OFI does not provide enough
information – how does the business unit
respond to the comment and how do the
commissioners identify the weak area.
What’s Wrong with this Observation?
• An opportunity exists to broaden the advanced educational
experience of the faculty by hiring and/or sending current
faculty to a broader range of institutions than is currently the
case. This may result in a broader base of knowledge for
students in the Department of Business.
• This is a prescriptive comment – the Standards and Criteria do
not prescribe which institutions faculty members should be
hired from.
What’s Wrong with this Observation?
• “Although the team was delighted with the depth
of the understanding and the commitment to
assessment, there were numerous closing the loop
issues based on the self study report review. Upon
examination and audit during the site visit,
numerous closing the loop evidences were
discovered.”
• “Team was delighted” is judgmental and the issues
discussed in the OFI were resolved while on site.
This is not an OFI – it can actually be a strength.
Common Mistakes








Critique vs. information
“It is not clear”
Not addressing both feedback report
customers: Commissioner and school/program
Too much brevity
Lack of clarity; “So what”?
Program Evaluator Self-Study Review not
converted to Feedback Report format
Comments limited to “they have a plan…”
Preliminary comments not prepared prior to
site visit
• QUESTIONS TO ASK YOURSELF
• WHEN REVIEWING THE SELF-STUDY:
1. What approach is described? Is it systematic?
2. How is the approach deployed and to what extent for
applicable stakeholder groups?
3. Are the results presented?
•4. Is there evidence of learning or improvements?
• QUESTIONS TO ASK YOURSELF
• WHEN REVIEWING THE SELF-STUDY:
1. Are the results presented?
2. Are there three to five data points?
3. Are the results segmented?
3. Are comparisons provided (e.g.,peer, national norm,
etc.) ?
•4. Are targets/benchmarks identified?