Transcript Webinar for First-Time or Previous Evaluators

ACCREDITATION COUNCIL FOR
BUSINESS SCHOOLS
AND PROGRAMS
WEBINAR FOR
ACBSP
FIRST-TIME EVALUATORS
Presented Spring 2013
Purpose of this webinar:
 Provide additional information for our first
time evaluators.
 Provide additional information on how to
write strengths and opportunities for
improvement comments.
 Provide an opportunity for you ask any
questions you may have.
Agenda:
•
•
•
•
•
•
•
•
•
•
•
Welcome and Introductions
Role of the evaluator
Resources available to site visit team
Scheduling of Consensus Calls and your role
during the call
Travel arrangements and travel policies
Communication with institutional contact
Site Visit Schedule
Format for final comments to be included in
the feedback report
Review and submission of feedback report
Submission of expense reports and other
forms
Any other topics
ACBSP STAFF CONTACTS:
Steve Parscale
Director of Accreditation
913-339-9356,
Mobile: 913-526-5285
[email protected]
Diana Hallerud
Assistant Director of Accreditation
913-339-9356,
Mobile: 816-699-9964
[email protected]
ACBSP STAFF CONTACTS:
Ron DeYoung, Executive Liaison
for the Baccalaureate/Graduate
Degree Board of Commissioners,
[email protected]
Mobile: 816-695-9585
Larry Zachrich, Executive Liaison
for the Associate Degree Board of
Commisisoners,
[email protected]
Mobile: 419-966-7091
ACBSP STAFF CONTACTS:
Maria Hallerud, Accreditation
Manager,
[email protected]
Office: 913-339-9356
Andrea (Andy) Ernst,
Accreditation
Administrative Assistant
[email protected]
Office: 913-339-9356
Primary Role of the Evaluator:
• Evaluate and validate the
information provided in the selfstudy to the standards and criteria.
• Provide feedback ready comments to
the business unit regarding
strengths and opportunities for
improvement based on information
from the self-study and observations
during the site visit.
Travel arrangements:
 Review the Travel Guidelines
 Evaluators should plan on using their personal credit cards
to pay for airfare and be reimbursed through ACBSP.
 There are exceptions for international travel, when the
institution may choose to make the flight arrangements for
you or when you can use the ACBSP travel agent. You
should always check with Steve Parscale or Diana Hallerud
before contacting the ACBSP travel agent.
 The institution will typically make the hotel reservation for
the site visit team, but it varies by institution for the billing
arrangements. The institution sometimes sets up a direct
billing arrangement. Most institutions ask the site visit
team to pay for the hotel room directly and seek
reimbursement through ACBSP.
 Travel expenses are processed on the 15th and the 30th of
the month.
 Submit your expense forms and receipts to Mary Riley via
email at [email protected], fax to 913-339-6226, or
send snail mail.
Resources available to you, cont.
Files on the CD and notebook you received
Under Tab One:
Evaluator Training PowerPoint Slides – June 2011
Scoring Guidelines & Worksheet
Practical Guide for Evaluators (includes)
ACBSP Observation Card
ACBSP Site Visit Notes
Do’s & Dont’s for Evaluators
Site visit Issue Worksheet
Site Visit Questions
Site Visit Issue Strategy
Resources available to you, cont.
Files on the CD and
notebook you received
Under Tab Two:
Process Manuals
Under Tab Three:
Standards and Criteria
Under Tab Four:
Evaluator Workbook
Feedback Report
Executive Summary Template
Resources available to you, cont.
Files on the CD and notebook you
received
Under Tab Five:
Travel Reimbursement Guidelines
Travel Reimbursement Form
Travel Expense Form
Evaluator Agreement Form
Evaluator Critique Form
Review Process:
• Review contents of package and verify
that you have received items checked
on the checklist.
• Read the entire self-study, including
the overview, prior to the consensus
call. The overview will provide
important information regarding
mission, vision, student demographics,
strategic challenges, etc.
Reviewing the self-study, cont.
READING THE ENTIRE SELF-STUDY DOCUMENT IS IMPORTANT
BECAUSE
 a standard or specific criteria may not be addressed where you
expect to see it. Don’t be surprised if you see student satisfaction
results in Standard 4 rather than Standard 3 or Standard 4 items
such as learning outcomes in Standard 3 or Standard 6.
 the standards have many linkages and information can be found
throughout the document. Some self-studies even have most of the
results in the appendices rather than the self-study document.
 the information may not be in the correct place, but the business
unit can still meet the standard. It just requires additional effort on
your part to locate the information.
 Using post-its or different colored highlighters when you find
information referencing the standards you are responsible for can be
helpful when you begin preparing your comments.
Review Process:
• Use the evaluator workbook and
begin writing observations prior to
the consensus call.
• The team chair will notify each team
member regarding the standards you
will have primary responsibility for
reviewing and providing strengths
and OFI’s. The chair may ask you if
you have a preference based on your
expertise.
Contact with the Institutional
Coordinator/Contact
 All communication with the institutional
coordinator or contact should be routed
through your team chair.
 Any requests for clarification or additional
information should be forwarded to the
team chair.
 This way the institutional coordinator is
not responding to duplicate requests for
information. Your cooperation is
appreciated.
The Consensus Call:
• Using the scoring worksheet,
evaluate each of the standards
using the best in class to major
improvements needed criteria.
• The team chair will review site
visit logistics and lead the
discussion regarding
observations after reviewing the
self-study.
The Consensus Call continued.
• For baccalaureate/graduate
degree site visits, the
commissioner and the mentor
are invited to participate in the
call and answer questions.
• For associate degree site visits,
the mentor is invited to
participate in the consensus call.
The Consensus Call continued.
• The team will identify site visit
issues and additional information
needed to be supplied prior to or
during the site visit.
• Discussion regarding meetings
and the site visit agenda will be
finalized.
Overall Scoring Summary Worksheet
 Use these scoring dimensions to complete the
table on the following page. Your individual
review will be discussed during the consensus
call.
 Individual Assessment Dimensions
 Best in Class
 Good to Excellent
 Good
 Major Improvements Needed
 Poor
Scoring Band
Qualitative
Dimensions
Approach/ Deployment
Quantitative
Dimensions
Results
Poor
* no systematic approach evident; anecdotal information
Poor
* no results or poor results in areas reported
Major
Improvements
Needed
* beginning of a systematic approach to the primary
purposes of the item
* early stages of a transition from reacting to problems to
a general improvement orientation
* major gaps exist in deployment that would inhibit progress in achieving the primary purposes of the item
* a sound, systematic approach, responsive to the primary purposes of the item
* a fact-based improvement process in place in key
areas; more emphasis is placed on improvement than
on reaction to problems
* no major gaps in deployment, though some areas or
work units may be in very early stages of deployment
Major
Improvements
Needed
* early stages of developing trends; some improvements
and/ or early good performance levels in a few areas
* results not reported for many to most areas of importance to the applicant’s key business requirements
Good
* a sound, systematic approach, responsive to the overall purposes of the item
* a fact-based improvement process is a key management tool; clear evidence of refinement and improved
integration as a result of improvement cycles and analysis
Good to
Excellent
* improvement trends and/ or good performance levels
reported for many to most areas of importance to the
applicant’s key business requirements
* no pattern of adverse trends and/ or poor performance
levels in areas of importance to the applicant’s key
business requirements
* some trends and/ or current performance levels -evaluated against relevant comparisons and/ or good to
very good relative performance levels
* current performance is good to excellent in most areas of
importance to the applicant’s key business requirements
* most improvement trends and/ or performance levels are
sustained
* many to most trends and/ or current performance levels -evaluated against relevant comparisons and/ or bench-
Good
Good to
Excellent
Best in Class
* approach is well-deployed, with no major gaps; deployment may vary in some areas or work units
* a sound, systematic approach, fully responsive to all
the requirements of the item
* a very strong, fact-based improvement process is a
key management tool; strong refinement and integration -- backed by excellent analysis
* approach is fully deployed without any significant
weaknesses or gaps in any areas or work units
Best in Class
marks -- show areas of leadership and very good relative
performance levels
* current performance is excellent in most areas of
importance to the applicant’s key business requirements
* excellent improvement trends and/ or sustained excellent
performance levels in most areas
* strong evidence of industry and benchmark leadership
demonstrated in many areas
ASSOCIATE DEGREE SCORING SUMMARY WORKSHEET
Evaluator Name:
School Name:
SUMMARY OF EXAMINATION ITEMS
1 LEADERSHIP
2 STRATEGIC PLANNING
3 STUDENT, STAKEHOLDER, AND MARKET FOCUS
4 MEASUREMENT, ANALYSIS, AND KNOWLEDGE MANAGEMENT
4.1 Student Learning Outcomes Assessment
4.2 Program Evaluation
4.3 Student Assessment
5 FACULTY AND STAFF FOCUS
5.1 Human Resource Planning
5.2 Faculty Qualifications
5.3 Faculty Composition
5.4 Faculty Deployment
5.5 Faculty Load
5.6 Faculty Evaluation
5.7 Faculty and Staff Professional Development & Scholarly Activities
5.8 Faculty Instructional Development
5.9 Faculty Operational Policies, Procedures, & Practices
6 PROCESS MANAGEMENT
6.1 Curriculum
6.2 Professional Component
6.3 General Education Component
6.4 Business Major Component
6.5 Off-Campus Operations and Unique Items
6.6 Minimum Grade Requirement
6.7 Learning and Academic Resources
6.8 Support Services
6.9 Educational Innovation
6.10 Articulation and Transfer Relationships
Individual Assessment Dimension
Scoring Summary worksheet
Baccalaureate/Graduate Degree:
Examiner Name:
School's Name:
SUMMARY OF EXAMINATION ITEMS
1 LEADERSHIP
2 STRATEGIC PLANNING
3 STUDENT AND STAKEHOLDER FOCUS
4 MEASUREMENT AND ANALYSIS OF STUDENT LEARNING AND PERFORMANCE
4.1 Selection and Use of Information and Data
4.2 Selection and Use of Information Results
4.3 Selection and Use of Comparative Information and Data
4.4 Continuous Process Improvement
5 FACULTY AND STAFF FOCUS
5.1 Human Resource Planning
5.2 Employment
5.3 Faculty Deployment
5.4 Faculty Size and Load
5.5 Faculty Evaluation
5.6 Faculty and Staff Development
5.7 Faculty Operational Procedures, Policies, and Practices
5.8 Scholarly and Professional Activities
6 EDUCATIONAL AND BUSINESS PROCESS MANAGEMENT
6.1 Education Design and Delivery
6.2 Management of Educational Support Service Processes and Business Operation Processes
6.3 Enrollment Management
Individual Assessment
Dimension
Typical Site Visit Schedule:
 Review the typical schedule outlined in the Practical Guide for
Evaluators.
 Typically, team members arrive Sunday afternoon/early evening
and have a team meeting to discuss site visit issues and finalize
the schedule.
 Monday is typically filled with meetings with administrators,
faculty, students, advisory boards, assessment officer, etc.
 Most teams complete all of the meetings on Tuesday morning
and hold any follow-up meetings early afternoon. Tuesday
afternoon and evening will be used to finish writing your portion
of the report.
 Your completed portion of the report should be given to the team
chair to be used to prepare the executive summary and as a part
of the exit interview.
CONTINUOUS IMPROVEMENT PROCESS
APPROACH
IMPROVEMENT
DEPLOYMENT OR
IMPLEMENTATION
RESULTS
and
LEARNING
Keep in mind:
When describing processes for the
business school or programs, that
the overall value of the results and
improvements taking place is
dependent on the approach (what
they do) and the level and extent of
deployment (how they do it).
Review Process after
Consensus Call:
• Continue to refine comments for
the feedback report prior to the
site visit – share with the team
chair.
• Ask for guidance when needed –
there are no dumb questions and
every chair has experienced a
first visit – he or she will
welcome any questions you
have.
MOST IMPORTANT THING TO
REMEMBER AS AN EVALUATOR:
A site visit that is conducted in an organized
and professional manner followed by a
feedback report that is professionally and
clearly written will create a positive
impression of what ACBSP represents as an
accrediting body.
While you may travel to a location new to
you, participating in a site visit is not a
vacation and should not be considered as
such.
AS AN EVALUATOR, DO…
 Remember that you serve as an ambassador for ACBSP. Most
of the faculty and staff you will be interacting with may not have an
opportunity to attend regional or national conferences. The site visit
experience will be what they base their perception of ACBSP on.
Make it a positive one!
 Come prepared for the site visit with materials read in
advance. You are part of a team with your team members
depending on you. You will work hard during the site visit and will
find it immensely rewarding.
 Say thank you and complement the institution for their
hospitality and availability of materials. Many institutions provide
resources and snacks over and above what is expected.
 Make every attempt to stay on the established schedule and let
the next person know in advance if the team is falling behind
schedule.
 Let the team chair contact the institutional contact if
additional information or resource is needed. A little diligence
on your part will result in a stronger report and assist the Board of
Commissioners in making the accreditation decision.
AS AN EVALUATOR, DO…
 Let the team chair serve as the primary point of contact
in making the arrangements both in advance and during the
site visit.
 Contact the ACBSP office if you encounter any problems
and are not sure how to proceed.
 Remember that each institution has its own mission and
operating processes and procedures. You’re only
concerned with how the processes and procedures used serve
the institution, not how they compare to your institution.
 Remember that the size and location of the institution you
are visiting will determine the type of lodging available
during your stay. Institutions are asked to provide
reasonable, clean, comfortable accommodations as close to
the campus as possible.
AS AN EVALUATOR, DO NOT…
 Market your services as a consultant or as a potential
recruit at the institution. The evaluator’s role is to evaluate
and validate the information in the self-study, not to offer
personal assistance when a gap in a particular area is observed or
to look for employment opportunities while on campus.
 Under any circumstances, attempt to recruit faculty as either
a student or employee from the institution you are visiting.
Even if a faculty member voices an interest in leaving the
institution and there is a position available at your institution, do
not discuss this opening with the faculty member. Faculty
members who express an interest in seeking additional degrees
should be directed to ACBSP for a list of institutions available
rather than to your specific institution.
 Cancel appointments with administrators unless you can
provide a valid reason for the cancellation. Usually the
President or Vice-Presidents work their schedules around the site
visit schedule and do not appreciate a sudden cancellation.
AS AN EVALUATOR, DO NOT…
 Under any circumstances, make negative comments
regarding the business faculty or administration to
anyone during the site visit.
 Criticize the content or quality of the self-study during
the site visit. If a standard is not adequately addressed, let
the institutional contact know what is missing and ask if the
information can be derived from other sources.
 Wait more than 10 days to submit the feedback report
to ACBSP. Do not submit the report unless you
proofread the report thoroughly before submitting to
ACBSP. All team members should have the opportunity to
read the final report prior to submitting the final report.
Please make sure the comments are complete and accurately
reflect your observations as an evaluator.
Writing Comments:
 THINGS TO REMEMBER:
Comments are based on the requirements of the standards
and criteria, not personal opinions or how processes are
followed at other institutions.
Comments should be clear and concise and should address the
situation, how the business unit does (for strengths) or does
not meet (for OFI’s) the standard, and explain the significance
of the situation.
Comments should not be prescriptive unless you are using
language directly from the standards and criteria.
Comments should be written in complete sentences and
should use proper grammar and punctuation.
Avoid the use of slang or local expressions others may not be
familiar with, especially when on international visits.
Example: “pig in a poke”.
Writing a strength:
If the institution has a strong strategic plan that includes both short and
long-term goals and
has a good approach,
has been deployed or implemented,
demonstrates learning,
and identifies improvements, your comment could state:
John Doe School of Business has a strategic plan that
has identified both short and long-term goals. The plan is
systematic with an approach that has been implemented
for all disciplines, provides examples of learning and
evidence of improvements the school has made. The
strategic plan helps the school to plan for increasing
enrollment and the replacement of faculty members as
they retire.”
 “The
Writing a strength:
If the institution has an assessment instrument and
provides evidence of a systematic process and
examples of how the results are used to make
improvements, you could provide a comment such
as:
 “The John Doe School of Business has administered
the MFT test for all disciplines for five years. Each
department documents their review of the pre and
post results each semester during department
meetings and makes improvements based on the
results. For example, the accounting department
added additional projects for preparing financial
statements in the ACCT 3300 course.”
Writing an opportunity for
improvement:
Suppose the institution has identified:
 increasing enrollment and a large number of faculty
scheduled for retirement as strategic challenges.
 All the strategic objectives in the strategic plan are for
one to three years, and faculty are not consulted in the
development of the objectives.
The opportunity for improvement could read,
“While the John Doe School of business has strategic
objectives for up to three years, there was no evidence of
planning beyond three years or involvement by faculty in
the development of the objectives. Developing long term
goals and involving more stakeholders such as faculty could
provide resource planning to meet the increasing
enrollment and replacement of faculty members scheduled
to retire in the next five years.
Writing an Opportunity for Improvement:
Suppose the institution administers the Major Field Test
(MFT) to all students as a pre and post test.
 There is no evidence, either in the self-study or on-site,
that the results are reviewed to indicate that there is
learning or subsequent improvements are made. In
addition, the school does not use the data for
comparative purposes
Your opportunity for improvement comment could read:
“The John Doe School of Business provided three years of
data for the Major Field Test (MFT). However, there is no
evidence that the school is using this data to measure how
their students are performing in each discipline and how
the results compare to other institutions with a similar
mission. Analyzing the results to identify program
strengths and weaknesses could allow the school of
business to make improvements where needed and provide
an external source of comparative data.”
Submitting your final comments:
 Using your initial review, you will conduct interviews and
review documentation to validate the information
provided in the self-study.
You may find that some initial strengths become OFI’s and
some OFI’s will either be reduced and/or become a
strength.
Wording for OFI’s should change from “it is not clear” to
“there is no evidence” following the site visit.
Before submitting your comments to the chair, read them
from the perspective of someone who did not write the selfstudy. Based on the comment, would the person
understand the jist of the comment? Can they relate it to
the standard or criteria you are referencing?
Ask your team members for help if you get stuck. They
won’t mind and will be asking you to read a comment for
clarity as well.
Final Steps:
 On Tuesday evening, you will submit your final
comments to the chair and confer with the team if you
have questions.
 Wednesday morning, you will participate in the exit
interview and return home.
 The chair should send the report for a final review to
team members before submitting it to ACBSP. Respond
with your comments promptly and let the chair know if
you saw evidence of meeting a standard that a team
member may have missed or if a comment is not clear
to you.
 Submit your expense report, receipts and evaluation
form to ACBSP as soon as you return from the trip.
Questions?
 While you will work hard, most evaluators find the
site visit experience to be very rewarding and no
matter how many visits they have participated in,
they always learn something new that they can use
to improve the processes at their own institution.
 Thank you for participating in this webinar. If you
have any suggestions for improvement, please feel
free to contact me by phone or via email at
[email protected].
 We hope you enjoy your first site-visit experience.