Transcript How Can I Miss You If You Won't Go Away? And Other

WASC Evaluator Workshop
Fall Visits 2009
1
Workshop Outcomes
•
Understand WASC’s three-stage process and how your visit
fits into the process
•
Be familiar with the WASC Standards and CFRs and how to
use them
•
Know how to prepare for and conduct an effective visit and
produce a useful, high-quality team report
•
Be prepared to make sound judgments about institutions
under the Standards
•
Be familiar with resources that support your work on a team
2
Agenda
I.
Context for the Visit/Accreditation
II.
WASC Three-Stage Review Process
III.
Standards and CFRs
IV.
Preparing for the Visit
V.
Conducting the Visit
VI.
Developing Team Recommendations
VII.
Writing the Team Report
VIII.
After the Visit
3
Context for Accreditation and Visits
• The Continuing Evolution of the WASC
Process and Standards
• The Accountability Movement
– Retaining Peer Review
• The Impact of the Economy
• Value Added, Collaboration, and Ongoing
Efforts to Refine and Improve
4
The WASC
Review Process
5
Three-Stage Process
1. Institutional Proposal: Identifies priorities,
themes/areas of emphasis, and outcomes.
Aligns work with institutional plans and
needs.
2. Capacity/Preparatory Review: Focuses on
capacity (systems, policies, resources) and
readiness for educational effectiveness.
3. Educational Effectiveness Review:
Focuses on results, findings.
6
The Two Reviews
Capacity and Preparatory
Educational Effectiveness
• Preparatory = readiness
for the Educational
Effectiveness Review
• Capacity = purposes,
integrity, stability,
resources, structures,
policies, processes
Demonstrating:
• Student learning
• Institutional learning
• Evidence-based decisionmaking
7
The CPR and EER as a Whole
• The CPR evaluates what an institution has
for infrastructure (staff/faculty, resources,
processes, facilities, systems, structures).
• The EER evaluates how well that
infrastructure works and the results that the
institution achieves.
8
Navigating Multiple Purposes (1)
Focusing on the institution
 Applying the Standards and
 CFRs
Focusing on Proposal
themes/topics
 Evaluating capacity and
 effectiveness under Standards;
addressing team-identified
issues
Reviewing the whole
institution
 Focusing on specifics, e.g.,
 distance education, samples of
program reviews
Advancing institutional
development
 Addressing “compliance”
 matters
9
Navigating Multiple Purposes (2)
Allowing flexibility and
experimentation on visit
 Ensuring consistency and
 fairness among visits and
quality control of visits and
reports
Using the CPR to evaluate
EE readiness
 Leaving evaluation of
 educational effectiveness until
EER
Supporting institutional
creativity and excitement
 Reporting to the Commission
 and serving the public
10
The Special Visit
• Intended to monitor institutional progress on
issues identified by the Commission
• May or may not be connected to a sanction
• Limited to a few specific areas of concern
• Intended to assess how institution will move
into compliance (if on sanction)
11
Understanding the Team’s Impact
Why were you chosen for a team?
• Peer review is the foundation of accreditation.
• The team report forms the basis for the Commission
action and its letter.
• The team report and action letter inform the work of
the institution for years to come.
12
Working with the Standards
and CFRs
13
Understanding the Standards and CFRs
• Two Core Commitments: Capacity and Educational
Effectiveness
• Standards: Broad, holistic, encompassing
• Criteria for Review: More specific and detailed
• Guidelines: Ways to demonstrate compliance with
the relevant CFR
14
Using the Standards and CFRs
• Team judgments must be linked to specific Standards
and CFRs
• CFRs must be cited in reports
• Standards and CFRs form the basis for Commission
decisions
• Standards and CFRs provide a context for continuous
quality improvement
15
STANDARD 1:
Defining Institutional Purposes and
Ensuring Educational Objectives
Institutional Purposes
Integrity
16
STANDARD 2:
Achieving Educational Objectives
Through Core Functions
Teaching and Learning
Scholarship and Creativity
Support for Student Learning
17
STANDARD 3:
Developing and Applying Resources
and Organizational Structures to
Ensure Sustainability
Faculty and Staff
Fiscal, Physical & Information Resources
Organizational Structures &
Decision-Making Processes
18
STANDARD 4:
Creating an Organization Committed
to Learning and Improvement
Strategic Thinking and Planning
Commitment to Learning and Improvement
19
Tool: Standards at a Glance
20
Comparing the Two Visits:
Different Views of a CFR (2.6)
The institution demonstrates that its graduates
consistently achieve its stated levels of
attainment and ensures that its expectations for
student learning are embedded in the standards
faculty use to evaluate student work.
21
CFR 2.6: Two Views
Capacity and Preparatory
 Has the institution
defined expected levels
of attainment for SL?
 Are they embedded in
the standards and
measures for student
work?
 How does the institution
know if students are
meeting expectations?
 What data are collected
and how analyzed?
 How is student learning
measured?
Educational Effectiveness
 What do data show
about student learning?
 Are data disaggregated
and analyzed?
 Did the students learn
what the faculty
intended them to learn?
At what levels of
performance?
 Has the institution used
data to make changes
and/or improvements?
22
Changes in 2009:
• Implement changes to Institutional Review Process
re: Student Success, Program Review and EE
Sustainability
• Implement changes to CFRs
• Clarify the scope of the CPR visit to review the
“infrastructure” for assessment of student learning
• Examine Program Review and Program-Level
Student Learning in a systematic way
• Allow teams more time together on visits
Tool: Table A & B (RB pg. 47)
23
Preparing for the Visit
(Visit Guide, Part II, pp. 29-52)
24
Timeline For CPR/EER Reviews
12 weeks
Institution
mails report
to team and
WASC
Team holds
conference
call
2 months
Site visit held
and team report
written
Institution
responds to
errors of fact in
team report
Institution
responds to
final team
report
Commission
acts at
February or
June meeting
25
Roles and Responsibilities of
Team Members and Staff
• Role of team chair (RB pg. 189)
• Role of team assistant chair (RB pg. 191)
• Role of assigned WASC staff liaison (VG
pg. 7)
• Team assignments
26
Pre-visit Preparation
• Read all the documents from WASC
– Standards, CFRs, policies, visit guide, rubrics
– Background documents re: institution and purpose of
the visit, including Proposal and/or last action
letter/team report
• Read the institutional report
• Review the data portfolio and exhibits
– What to look for and highlight?
Tools: Timeline (VG pg. 8, VG pg. 29)
27
Reviewing the Exhibits
• Enrollment data
– Headcounts and FTE
• Graduation data
• Faculty data
• Key financial indicators
• Inventory of Educational Effectiveness Indicators
• Inventory of Concurrent Accreditation and Key
Performance Indicators
Tool: How to Review WASC Data Exhibits (RB pg. 61)
28
Reading the Report
• Has the institution done what it said it would
do in its Proposal?
• Has it collected and analyzed data effectively?
• Are its conclusions supported by evidence?
• Are there serious problems or potential areas
of noncompliance?
• Does the report contain recommendations for
further institutional action?
29
Developing Visit Strategies and
Lines of Inquiry
• What are areas needing clarification and/or more
information?
• What are the major issues challenging the institution?
• What is raised by the themes that needs to be verified
or explored?
• What are the strategies that will be most effective?
30
Worksheet for Team Conference Call
• Organizes team’s responses to institutional materials
• Helps team make preliminary evaluation under the
Standards
• Provides basis for team to work toward consensus
• Should be submitted in advance of call
Tool: Team Worksheet (VG pg. 43)
31
Team Conference Call
• Evaluate quality of institutional report and alignment
with Proposal and previous action letter(s)
• Identify areas of good practice, improvement, and
further inquiry
• Identify issues, strategies, evidence needed
• Identify persons and entities to be interviewed
• Make or refine team assignments
• Plan visit logistics
32
Off-Campus Sites and
Distance Education Programs
Prior to Visit: Sites will be identified and assignments made
• Review substantive change action letters to determine if issues
have been identified
• Develop plan for the review of the programs and/or sites
During Visit
• Interview faculty, administrators and students
• Evaluate facilities OR online infrastructure
• Observe classes
• Document visit and findings in appendix
• Discuss important findings with team for inclusion in report, as
appropriate
Tools: Protocols (RB pg. 158, RB pg. 160)
Forms (RB pg. 55, RB pg. 58)
33
Compliance Audit
• Required for:
– Institutions seeking Candidacy and Initial
Accreditation
– Some institutions under sanction
• Additional report submitted by institution in
advance of the visit—with links to documents
Tool: Compliance Audit Checklist (RB, pg. 51)
34
Determining Strategy for CPR Visit
• What evidence is provided to show capacity and
readiness for EE?
• Why was it chosen?
• What are the strengths and weaknesses of the
evidence?
• What other evidence do you want to review to evaluate
capacity and preparation for EE?
• Do any issues arise with regard to the Standards?
• Meetings: format/methodologies
35
Determining Strategy for EER Visit
• What evidence is provided to show EE?
• Why was it chosen?
• What are the strengths and weaknesses of the
evidence?
• What other evidence do you want to see to evaluate
effectiveness?
• Do any issues arise with regard to the Standards?
• Meetings: format/methodologies
36
Drafting in Advance of the Visit
• Assistant Chair draft outline of team report
and context sections
• Team members draft outline or text for which
they are responsible, using data from
institution, with space for additional data,
analysis and conclusions
Tool: Team Reports (VG pg. 53)
37
Conducting the Visit
38
Process of Visit
• Team meets at start of visit to confirm roles,
assignments, logistics, and agenda
• Team meets frequently re: observations, emerging
recommendations, and issues
• Team members draft sections of report and turn in to
assistant chair on the last day
• Team agrees on report recommendations and
confidential recommendation to Commission
39
Visit Schedule
• Executive sessions and debriefings with team only
• Meetings and interviews with key individuals and
groups
• Open meetings with students, faculty and staff
• Document review
• Time for drafting report sections
• Final exit meeting
Tool: Sample Visit Schedule
40
Confidential Email Account
• Set up by WASC as extension of open
meetings
• Checked by assistant chair during visit
• Important emails shared with team and
investigated
• Comments included in team report only if the
institution has a chance to address them
41
Approaches Used on Visits
• Document review
• Interviews and meetings
– Mini-questionnaires
– Techniques for small and large meetings
– Fishbowl exercises
• Audits
Plan visit methodologies in advance
as part of schedule.
42
Document Review
Use to:
Check compliance
Evaluate the level of institutional engagement
Examine the evolution of a policy or process
Identify direct and indirect evidence of student and
organizational learning
Confirm report claims
DO as much as possible in advance
43
Interviews
Use to:
Gather information
Explore issues
Build relationships with members of the
institution
Validate impressions and observations
44
Tips for Good Interviews
 Decide on a protocol for interview
 Prepare questions and lines of inquiry in advance
 Ask questions that elicit information, stimulate
discussion, or require judgment
 Avoid interrogation, leading questions, or loaded
language
 Avoid consultation, giving solutions, or talking about
your institution
 Let them do the talking
45
Alternative Forms of Interview
 Fishbowl
 Brainstorm/free discussion on a salient topic
 Go-round
 Bundling
 Audit
46
Evaluating Program Review
and Student Learning
(EER Visits)
Tool: EE Toolkit
Tool: Suggested Approaches to Evaluating Program
Review on EER Visits
47
Rubrics:
Assessment of Student Learning
1. Academic Program Learning Outcomes
2. Use of Portfolios in Assessing Program Outcomes
3. Use of Capstones in Assessing Program
Outcomes
4. Integration of Student Learning Assessment into
Program Review
5. Assessing General Education
Tool: Program Learning Outcome Rubric
48
Expectations for Two Reviews
Use:
• ‘Student Learning’ questions (p. 2)
• as a monitor to be sure you are within the
proper scope of the visit
Tool: Expectations for Two Reviews
49
Educational Effectiveness
Framework
• Use with team to evaluate institution’s “place”
• Use language of rubric to describe the institution in
the report
• Ask the institution to evaluate itself and discuss
• Confer with team toward end of visit to mark a copy
of the EEF
• Submit the marked EEF confidentially to WASC
Tool: EE Framework
50
The Exit Meeting
• Team chair communicates commendations
and key recommendations that will be
included in report
• Chair may ask team members to participate
• The meeting is not a dialog, discussion or
debate
51
Developing Team
Recommendations
52
Two Kinds of
Recommendations
• Confidential Team Recommendation to the
Commission for action
• Team recommendations at the end of team
report, delivered at the exit meeting
Tool: Commission Decisions on Institutions (VG pg. 84; SVG
Append. F)
Tool: Commission and Team Decision Indicators (RB pg. 173)
53
Team Report Recommendations
Should be:
–
–
–
–
Overarching and important
Supported by evidence
Linked clearly to Standards and CFRs
Supported by text in the report
- Distinguish recommendations from suggestions and
observations embedded in the report
Tool: Educational Effectiveness Framework (RB pg. 176)
54
Confidential Recommendation to Commission
(CPR)
Proceed to EER or reschedule EER visit
Conduct a Special Visit (not preferred)
Add time to EER visit
Issue a notice of concern or impose
a sanction
Tool: Confidential Team Recommendation Form (VG pg. 70)
55
Confidential Recommendation to Commission
(EER)
Grant Candidacy, Initial Accreditation or
Reaffirmation of Accreditation for specified
term
Sanction or Notice of Concern
Interim Report or Special Visit
Tool: Confidential Team Recommendation Form (VG pg.72)
Tool: Commission and Team Decision Indicators (RB pg. 173)
56
Confidential Recommendation to Commission
(Special Visit)
Varies with status of institution
Next steps
Removal or continuation of sanction (note
two-year limit on sanctions)
57
Producing Effective
Team Reports
58
Report Preparation Logistics
• Follow report template
• Start writing before the visit
• Complete your sections on site and give to
assistant chair for editing together
Tool: CPR Report template
59
What is an effective team report?
 Reflects a thorough assessment of the
institution’s capacity, preparation, and/or
effectiveness
 Is evidence based
 Cites the Standards and CFRs
 Provides the basis for a sound and supportable
Commission decision
 Identifies important areas for institution to
address
60
Using Evidence in Team Reports
Use qualitative and quantitative evidence
Select evidence carefully and purposefully
Connect evidence to an assertion or question
Analyze information; do not just set forth data
Let evidence suggest improvements
Use evidence that speaks to the institution’s
themes and the team's questions
61
Tips for Writing Team Reports
 Consider multiple audiences: institution, Commission, and
next team
 Know your areas of responsibility, including length and
depth of your section
 Start writing before you arrive on campus
 Address priorities and goals set by the institution
 Address Commission’s concerns (last action letter)
 Make commendations, but don’t overdo it
 Use praise that doesn’t send wrong or mixed signal
62
More Tips on Team Reports….
 Be sure to check facts
 Support findings and recommendations with
evidence --and tie them to CFRs
 Ensure evidence is sound and valid
 Distinguish recommendations from suggestions or
observations
 Use formal language and tone (e.g., not “we/they”)
 Don’t mention personnel by name
 Don’t prescribe solutions
63
After the Visit
64
What happens next?
↓ AC prepares draft for chair, team and staff
review; changes as needed
↓ Chair sends to institution for corrections
of fact
↓ Chair finalizes draft and submits to WASC
↓ Chair sends Confidential Team
Recommendation and completed EEF
to WASC
↓ WASC sends report to institution
65
Then…
↓ Staff prepares draft action letter, which is reviewed
by team chair
↓ Commission Panel reads report and
documentation including institution’s written
response, meets with institutional representatives
at Commission meeting
↓ Panel makes recommendation to Commission,
and Commission acts
↓ Staff finalizes draft action letter on behalf of
Commission
66
Also after the visit…
Send reimbursement forms to WASC within 30 days
– Hotel arranged and paid directly by institution
– Travel / food reimbursed
– Rental car must be approved in advance by
WASC staff
– Spouse or assistant costs not covered
– See policy for more details
Team members should not have any contact with the
institution
– About the visit OR
– Consult with the institution for one year
67
Common concerns about Visits
• Some team members not well prepared
• The team “did not understand us”
• The CPR team moved into EER “territory”
• Team did not review all the evidence
• Team changed the schedule at the last minute, or did
not stay on schedule
68
Common concerns about Team Reports
• The recommendations were too specific or were unfair
• The report did not show that the team reviewed the
evidence
• The recommendations were not based on good
evidence or supported in report text
• The report did not address all the important issues or
themes
69
Resources for Teams
• Appendices of Visit Guide
• Team Materials and Institutional Report mailed
10-12 weeks in advance of visit
• WASC Website: www.wascsenior.org
• WASC Email Advisory (sent prior to visit)
• WASC Staff
70
Thank you for your service
to the region
71