Transcript Helping Faculty and Staff Create Meaningful

Amber Malinovsky
Assistant Director
Institutional Assessment
Texas A&M University

Texas A&M University, College Station, Texas





48,000 students
2,700 faculty
5,500 staff
5,200 acres
WEAVE users and programs
 470+ programs in entity tree
 700 WEAVE users

Office of Institutional Assessment
 Four full-time employees, two graduate assistants
 Variety of duties that support assessment
 Liaison structure
• Ten colleges
• Two branch campuses (Galveston and Qatar)
• Multiple support units (student affairs, undergraduate
programs, etc.)

Choose a topic
 Often suggested by users/Assessment Liaisons
 Sometimes specific to audience (academic vs. support, by
discipline, etc.)

Conduct research and collect examples

Draft and revise slides and materials
 Our “simple” formula: definition + tips + examples + cautions
+ putting it all together = meaningful, measurable, manageable

Choose dates and locations

Advertise and track registrants
 Announcement sent on email lists and posted on website
 Online form for registration

Evaluation and follow-up

Past workshops:
http://assessment.tamu.edu/wkshp_pres/index.htm
Dr. Loraine Phillips
Amber Malinovsky
Part One
 Introduction to Assessment
 Components of an Assessment Plan





Mission
Outcomes
Measures
Achievement Targets
Components of an Assessment Report




Findings
Analysis of Findings
Action Plans
Annual Reporting

May 31, 2009
Data collected

August 1, 2009
Findings entered

September 1, 2009
Analysis Questions answered,
along with any appropriate
action plans

September 15, 2009 Annual Report question
answered

October 1, 2009
2008-2009 Cycle Year CLOSES
no more editing
SACS Comprehensive Standard 3.3.1
3.3
Institutional Effectiveness
3.3.1 The institution identifies expected outcomes, assesses the
extent to which it achieves these outcomes, and provides
evidence of improvement based on analysis of the results in
each of the following areas: (Institutional Effectiveness)
3.3.1.1 educational programs, to include student learning
outcomes
3.3.1.2 administrative support services
3.3.1.3 educational support services
3.3.1.4 research within its educational mission, if appropriate
3.3.1.5 community/public service within its educational
mission, if appropriate
Modify &
Improve
Develop
Program
Mission &
Outcomes
Interpret/
Evaluate
Information
Design an
Assessment
Plan
Implement
the Plan &
Gather
Information
Adapted from: Trudy Banta, IUPUI

The mission statement links the functions of
your unit to the overall mission of TAMU.

A few questions to consider in formulating the
mission of your unit:
 What is the primary function of your unit?
 What core activities are involved?
 What should those you serve experience after
interacting with your unit?

Brief, concise, distinctive

Clearly identifies the program’s purpose

Clearly aligns with the mission of the division and the
University

Explicitly articulates the essential functions/activities of
the program

Clearly identifies the primary stakeholders of the
program: i.e., students, faculty, parents, etc.
“University Career Services, an integral part of the educational
process, assists students and alumni in assessing their career
possibilities, setting their personal goals and achieving their
objectives toward becoming productive citizens in the global
community. While assisting its clients in identifying professional
employment opportunities, University Career Services also
provides the university community with insights into the everchanging world of work to help develop realistic ways to better
educate tomorrow’s leaders.” (Texas Christian University)
When writing Learning Outcomes, the focus must be on the
stakeholders (e.g., students, faculty, staff, and others) and
what they will think, know, do, or value following the
provision of the service.

Process statements
 Relate to what the unit intends to accomplish
• Level or volume of activity
• Efficiency with which you conduct the processes
• Compliance with external standards of “good practice in the field” or
regulations

Satisfaction statements
 Describe how those you serve rate their satisfaction with your unit’s
processes or services





Access
Accountability
Affordability
Climate
Educational
Effectiveness/Quality






Facilities & Equipment
Faculty
Financial Support
Student Preparation
Productivity
Technology
Consider such questions as:
 What are the most important results or impacts that
should occur as a result of your unit’s activities?
 What are your critical work processes and how should
they function?
 What does the end user experience through interaction
with your unit?

Learning Outcomes
 Students receiving advising services will be able to identify and utilize
campus resources.
 Staff and faculty will be able to use the Oracle system for purchasing.

Process statements
 The number of faculty training workshops and participation rates for
WebCT will relate to the needs.
 The travel office will promptly process travel requests.

Satisfaction statements
 Faculty will report satisfaction in the training received for WebCT.
 Faculty and staff will report satisfaction with travel processing under the
Oracle system.
Outcomes should be:
 linked to the unit’s mission
 realistic and attainable
 limited in number (manageable)
 something that is under the control of the unit
 measurable and/or observable
 meaningful
 target key services or change points
 use action verbs
After establishing your outcomes…

Define and identify the sources of evidence you will
use to determine whether you are achieving your
outcomes.

Detail what will be measured and how

Identify or create, if necessary, measures that help your
unit in making critical decisions about its processes and
services.
Some things to think about:
 How would you describe the end result of the outcome?
 How will you know if this outcome is being
accomplished?
 What will provide you with this information?
 Where are you currently delivering the outcome?
 Are there any naturally occurring assessment
opportunities?
 What measures are currently available?
Learning Outcome
Outcome 1
Outcome 2
Outcome 3
Etc.
Method of Delivery
How, When, and Where
Information will be
Gathered

Measurable and/or observable
 You can either observe it, count it, quantify, etc.

Meaningful
 If captures enough of the essential components of the objective to
represent it adequately

Manageable
 It can be measured without excessive cost or effort
Direct measures are those designed to directly measure:

what a stakeholder knows or is able to do (i.e., requires
a stakeholder to actually demonstrate the skill or
knowledge)

The benefit of programming or intervention

Participation data

Observation of behavior

Culminating experiences (e.g., presentation, project,
internships, etc.)

Collection of work samples (portfolios)

Pre- and post-measures

Volume of activity

Level of efficiency (average response time)

Measure of quality (average errors)
Indirect measures focus on:

stakeholders’ perception of their level of learning

stakeholders’ perception of the benefit of programming
or intervention
stakeholders’ satisfaction with some aspect of the
program or service


Surveys

Exit interviews

Retention/graduation data

Demographics

Focus groups

An achievement target is the result, target, benchmark, or
value that will represent success at achieving a given
outcome.

Achievement targets can be specific numbers or trends.

Students will achieve a score of 3 or 4 on the rubric.

Average score on rubric is a 3 or better (on a scale of 0 to
4).

90% of the transcripts will be sent within three days.

Each employee will participate in a minimum of two
training/development programs per year.

Acquisition statistics will indicate growth in the overall
collections that support academic programs.
Outcome:
Improve and strengthen media relations and publications
Measure 1 (Direct):
Track media pitches/releases.
Achievement Target:
Average of 250 media pitches/releases for the year.
Measure 2 (Indirect):
Conduct annual readership survey of university magazine.
Achievement Target:
Feedback via readership survey will indicate average ratings of 7
or better on a 1-10 scale.
Outcome: To provide a library web site that enables users to locate and use
information on their own
Measure 1 (Direct):
Usability testing with small groups
Achievement Target:
Participants in usability testing demonstrate web site’s ease of use
Measure 2 (Indirect):
LibQUAL+ assessment program
Achievement Target:
LibQUAL+ results indicate faculty and student satisfaction

Writing your Achievement Target as part of your Outcome
 Example:
• Outcome: 80% satisfaction rate from instructors
• Measure: Biannual instructor satisfaction survey
• Achievement Target: 80% satisfaction rate
 Possible Steps to Revision:
• Satisfied with what?
• New Outcome: Instructors will report satisfaction with the resources
available for course development.

Having the same Measure and Achievement Target
 Example 1:
• Measure: Annual review of forms
• Achievement Target: Annual review of forms
 Possible Steps to Revision:
• Go back and look at outcome. Outcome states that Forms and paperwork
will be clear, concise & accurate.
• If the Annual review shows that they are not clear, concise & accurate, what
will need to happen?
• New Achievement Target: Decrease in number of revisions needed to make
forms more understandable and accurate.

Having the same Measure and Achievement Target
 Example 2:
• Measure: 50 slots for scholarship recipients secured
• Achievement Target: 50 slots secured for scholarship recipients
 Possible Steps to Revision:
• Remember: Measures are more general. Achievement Targets are
specific.
• Identify what it is that you’re doing to determine whether or not you’ve
met the achievement target. THIS is your measure.
• New Measure: Track number of scholarship recipients.

Writing your Achievement Target as part of your Measure
 Example:
• Measure: Score a 3 or 4 on an identified technology assignment
 Possible Steps to Revision:
• Remember: Measures are more general. Achievement Targets are
specific.
• Outcome states: Demonstrate competency in technology
• Revise Measure to read: Identified technology assignment.
• Add Achievement Target: Students will achieve a 3 or 4 on the rubric.
Assessment Plan
Mission/Purpose
 Outcomes, learning & program
 Assessment Methods with
achievement targets

Assessment Report
Mission/Purpose
 Outcomes, learning & program
 Assessment Methods with
achievement targets
 Findings
 Action Plans
 Analysis Questions
 Annual Report Section

SACS Comprehensive Standard 3.3.1
3.3
Institutional Effectiveness
3.3.1 The institution identifies expected outcomes, assesses the
extent to which it achieves these outcomes, and provides
andevidence
providesofevidence
of improvement
based
on results
analysis
improvement
based on analysis
of the
in
eachresults…
of the following areas: (Institutional Effectiveness)
of the
3.3.1.1 educational programs, to include student learning
outcomes
3.3.1.2 administrative support services
3.3.1.3 educational support services
3.3.1.4 research within its educational mission, if appropriate
3.3.1.5 community/public service within its educational
mission, if appropriate

Findings = Assessment data

On an Assessment Report, “findings” refers to a concise
summary of the results you gathered from a given assessment
measure.

The language of this statement should parallel the
corresponding achievement target

Describe your results in enough detail to prove whether you
have met, partially met, or not met your achievement target.

It is not necessary to provide any interpretation of your data in
your findings summary.

(Optional) Attach documents to support your data. These
can include survey instruments or results, reports,
committee members and minutes from meetings, etc.

Example 1:
 Achievement Target: Overall mean score of students from program
meets or exceeds state average score.
 Findings: The overall mean score of students from the Teaching,
Learning, and Culture program exceeded that of the state average
score of the state certification exam. Results: Program overall mean
scaled score—91.50, State overall mean scaled score—79.13.

Example:
 Achievement Target: Decrease by 40% the number of undeclared
students from the entering major (Fall 2008) to current major (Fall
2009)
 Findings: 34% of undeclared students who had registered for Fall
2009 classes by the beginning of June had declared a major.

Example:
 Achievement Target: 10 campus-wide workshops conducted in
November and December of 2008.
 Findings: 8 campus-wide workshops were conducted in November
and December of 2008.

Reflect on what has been learned during an assessment
cycle

Identify areas that need to be monitored, remediated, or
enhanced

Three key questions are at the heart of your analysis:
 What did you find and learn?
 So What does that mean for your academic program or support
unit?
 Now What will you do as a result of the first two answers?

You will want to reflect on the following areas:
 Student Learning Outcomes, if applicable
 Program Outcomes, if applicable
 The Assessment Process

After reflecting on the findings, you and your colleagues
should determine appropriate action to improve the
program. This will lead to at least one action plan.

Actions outlined in the action plan should be specific and
relate directly to the outcome and the results of
assessment.

We do not recommend having an assessment report
without any action plans.
 SACS 3.3.1

We do recommend that you keep the number of action
plans manageable.

The final step in your assessment report is to consider how
your program contributed to one or more of the following:
 Departmental goals or imperatives
 College goals or imperatives
 University mission, goals or imperatives
Assessment Plans
 Mission/Purpose Statements
 Outcomes
 Measures and Achievement Targets
Findings/Report
 Achievement Targets
 Findings
 Analysis
 Action Plans

You do not have to assess everything every year

Modify something already being done that is
meaningful to the program

Summarize your Findings

Limit your Action Plans and keep them manageable

Consider how your program fits within the university’s
mission

Be flexible—this is an iterative process
The Principles of Accreditation: Foundations for Quality Enhancement. SACS
COC. 2008 Edition.
Banta, Trudy W., & Palomba, C. (1999). Assessment Essentials. San
Francisco: Jossey-Bass.
Banta, Trudy W. (2004). Hallmarks of Effective Outcomes Assessment. San
Francisco: John Wiley and Sons.
Walvoord, Barbara E. (2004). Assessment Clear and Simple: A Practical Guide
for Institutions, Departments, and General Education. San Francisco: JosseyBass.
Assessment manuals from Western Carolina University and Texas
Christian University were very helpful in developing this presentation.
Putting It All Together examples adapted from Wright State University
Libraries Assessment Plan, UHCL President’s Office Assessment Plan





North Carolina State University
http://www2.acs.ncsu.edu/UPA/assmt/admin_examples
_others.htm
University of Central Florida—Handbook for Admin
Units
http://oeas.ucf.edu/doc/adm_assess_handbook.pdf
Texas Christian University
http://www.assessment.tcu.edu/
Each Other
OIA consulting
Amber Malinovsky, Assistant Director
Office of Institutional Assessment
Questions?

What was the most valuable thing you learned?

What is one question that you still have?

What do you think is the next step that your
program needs to take in order to implement
systematic program assessment?

Our Staff
 Dr. Loraine Phillips, [email protected]
 Amber Malinovsky, [email protected]

Our Website
 http://assessment.tamu.edu

TAMU Assessment Conference Website
 http://assessment.tamu.edu/conference/index.html
February 21-23, 2010
College Station, Texas
http://assessment.tamu.edu/conference
Plenary Speakers:
Dr. Belle S. Wheelan
Dr. Clifford Adelman
Registration now open!