Transcript Assessing the Work of Higher Education: Institutional

Assessing the Work of Higher Education:
Institutional Effectiveness
and Student Learning
Dr. Jo Allen, Senior Vice President & Provost
Widener University
Middle States Commission on Higher Education, October 2008
Overview of Presentation
Operational Terms
 Drivers of assessment
 Assessment of institutional
effectiveness
 Assessment of student learning
outcomes
 Questions and concerns

Assessment: An Operational Definition
Assessment is the process of asking and
answering questions that seek to align our
stated intentions with documentable realities.
As such, in higher education, it deals with
courses, programs, policies, procedures, and
operations.
Evaluation: An Operational Definition

Evaluation focuses on individual
performance in the sense of task
performance or job completion and quality,
typically resulting in merit raises, plans for
future improvement, or—in less satisfying
cases—probation and possibly firing.
Assessment vs. Evaluation


Assessment focuses on the work to be done,
the outcomes, and the impact on others—
typically, the aggregate situation, not just the
individuals.
Evaluation focuses on the work of the
individuals—their contributions,
effectiveness, creativity, responsibility,
engagement, or whatever factors the
organization deems most desirable.
Assessment vs. Evaluation
Assessment focuses on the work to be done, the
outcomes, and the impact on others—not on the
individuals doing the work.
 Evaluation focuses on the work of the
individuals—their contributions, effectiveness,
creativity, responsibility, engagement, or whatever
factors the organization deems most desirable.
Assessment of Institutional
Effectiveness vs. Student Learning


Institutional effectiveness = the results of
operational processes, policies, duties and
sites—and their success in working
together—to support the management of the
academy
Student learning = the results of curricular
and co-curricular experiences designed to
provide students with knowledge and skills
What or who is driving assessment?

Accreditors…

charged with determining the reputable from nonreputable institutions and programs
charged with ensuring that institutional practices
support the viability and sustainability of the
institution and its offerings
represent disciplinary and institutional interests


Assessment drivers (cont’d.)






The public: “Ivory Tower,” liberal bias, ratings/rankings
Legislators: responsive to citizens’ concerns about quality, costs,
biases….or?
Prospective faculty: Quality and meaningful contributions to students’
lives
Prospective parents: real learning and preparation for careers
Prospective students: How will I measure up? And what kind of job
can I get when I graduate?
Funding agencies/foundations: evidence of an institution’s or
faculty’s commitment to learning and knowledge and evidence of
[prior] success
Matters of Institutional Quality





Can we justify costs/prices of attendance?
Can we verify the quality of our educational offerings
in measurable terms?
Can we verify the effectiveness of operational
contributors to a sustainable educational
experience?
Can we use data and other findings to improve the
quality of our educational and operational offerings?
Can we use those findings to align resources
(financial, staff, curricular, co-curricular) to enhance
desired outcomes?
Sites of Institutional Effectiveness

Processes [existence and transparency]
–
–
–
–
–
–
–
Enrollment: Admissions, financial aid, registration
Curricular: Advising, progress toward degree completion
Budgeting: operations/salaries; capital; bond ratings and ratios;
endowment management; benefits; etc.
Planning: strategic planning, compact planning, curricular
planning, etc.
Judicial: education/training, communication, sanctions, etc.
Residence Life: housing selection, training for RAs, conflict
resolution/mediation
Advancement: fund-raising, alumni relations, public relations,
government/corporate relations, community relations, etc.
Sites of Institutional Effectiveness

Units/Offices of operations (samples)
–
–
–
–
–
–
–
–
Advancement
Admissions, Bursar, Registrar
Athletics
Center for Advising, Academic Support, etc.
Campus Safety
Institutional Research
IT
Maintenance
The Assessment Cycle:
Key Questions for Institutional Effectiveness





What services, programs, or benefits should our
offices provide?
For what purposes or with what intended results?
What evidence do we have that they provide these
outcomes?
How can we use information to improve or celebrate
successes?
Do the improvements we make work?
What are we looking for?
EXAMPLES of EVIDENCE:
 Our admission of students for whom our institution is
the first choice has risen 30%.
 95% of students report satisfaction with the housing
selection process.
 5 faculty committees participated in the last planning
cycle.
 Overall, faculty, staff, and students report feeling
safe on campus, following the new Campus Safety
Improvement initiatives.
Where do we seek improvement [and
what evidence will help us]?



We need to raise the number of students
who choose our institution as their first
choice to 95% by 2010.
All faculty committees will be invited to
participate in the next planning meeting.
Students (39%) still report feeling unsafe in
the mezzanine of the University Gallery. We
will …..
The Iterative
Assessment Cycle for
Institutional
Effectiveness
Gather Evidence
Interpret Evidence
Mission/Purposes
Objectives/Goals
Outcomes
Implement
Methods to
Gather Evidence
Make decisions to improve
programs, services, or benefits;
contribute to institutional
experience; inform institutional
decisionmaking, planning,
budgeting, policy, public
accountability
What qualities point to institutional
effectiveness?




A well-articulated set of processes for critical
functions
A clear line of responsibility and accountability for
critical functions
An alignment of the importance of the function and
sufficient resources (staff, budget, training, etc.) to
support the function
Evidence of institution-wide knowledge of those
critical functions, processes, and lines of
responsibility
What kinds of evidence points to
institutional effectiveness?






Well-managed budgets
Accreditation and governmental compliance
Clearly defined and supported shared governance
(board, president, administration, faculty, staff, and
students)
Communication pathways and strategies
[transparency]
Consensus on mission, strategic plan, goals,
priorities, etc.
Student (and other constituencies’) satisfaction
How do we measure institutional
effectiveness?





Tangible evidence: Audited budget statements,
handbooks, enrollment data, institutional data
Records/reports of activities and/or compliance
Self-studies pointing to documented evidence
Surveys of satisfaction, usage, attitudes, confidence,
etc.
Disciplinary accreditation reports
The Assessment Cycle:
Key Questions for Student Learning




What should our students know or be
able to do by the time they graduate?
What evidence do we have that they
know and can do these things?
How can we use information to improve
or celebrate successes?
Do the improvements we make work?
The Iterative
Assessment Cycle
Gather Evidence
Interpret Evidence
Mission/Purposes
Objectives/Goals
Outcomes
Implement
Methods to
Gather Evidence
Make decisions to improve
programs; enhance student
learning and development;
inform institutional decisionmaking, planning,
budgeting, policy, public
accountability
Student Learning Assessment:
What should students know or be able to
demonstrate by the time they graduate?









Civic engagement
Diversity appreciation
Communication skills
Professional responsibility
Ethics
Critical thinking
Collaborative learning
Leadership
Mathematical or Quantitative
competence









Technological competence
Scientific competence
Research skills
Cultural competence
Interdisciplinary
competence
Civic responsibility
Global competence
Economic/financial
competence
Social justice
What might our sources of evidence
be?










Essays/Theses
Portfolios (faculty or external readers evaluated)
Quizzes
Oral presentations
Homework assignments
Lab experiments
Tests
Journal entries
Projects
Demonstrations
What are we looking for?

Evidence of students’ skill level (basic
competency to mastery)
–
–
–
–
based on faculty-articulated standards of quality
and judgments
applied to all students’ work evenly
indicative of aggregate evaluations of
performance or knowledge
informative for course or program improvements
Can we use the same processes and
strategies to assess both arenas?

Measuring learning versus effectiveness, efficiency, and/or
satisfaction
–

Methods of testing, projects, demonstrations versus surveys,
records, reports
–

BEYOND ANECDOTAL INTO EVIDENCE
QUALIFY OR QUANTIFY THE OUTCOMES
Use of results (revisions versus training versus expansion)
–
MODIFY WHAT YOU DO TO AFFECT OUTCOMES
What is similar?



A commitment to doing the very best job
possible under whatever conditions exist
A commitment to recognizing ways that
altering those conditions can affect the
outcomes
A commitment to recognizing that altering the
outcomes can affect the conditions
Ultimately….
We hold ourselves and our colleagues
accountable for articulating the
intentions of our work and then
measuring the realities, resulting in
designing and implementing
strategies for improvement over time.
How are we doing?
How can we do better?


QUESTIONS?
Comments?