Transcript Assessment Workshop - Youngstown State University

Non-Academic Assessment:
Developing an Assessment Plan
OFFICE OF ASSESSMENT
JULY 11, 2012
Goals of Workshop
2
 Current assessment context
 What’s new this year
 Introduce new Higher Learning Commission Criteria
 Review key items on the Assessment Plan template
 Walk through new online reporting form
Assessment Context
3
Assessment Context
4
Assessment Context



Last year participating in the Higher Learning Commission’s
(HLC) Academy for the Assessment of Student Learning
New HLC Criteria effective January 1, 2013
Campus focus on 2020 Strategic Plan alignment
Assessment Reporting Themes



Focus on use of data
Reflect on changes and impact on learning
Streamline reporting process
Future Goals


Longer reporting cycle
Integration with annual reporting
What’s New This Year
5





Online reporting
Fewer questions
Questions “tagged” with New HLC Criteria
Optional question on New HLC Criteria
Report quality threshold to 70%
HLC New Criteria
6
 Guiding Values, includes:
1. Focus on student learning
4. Culture of continuous improvement
5. Evidence-based institutional learning and self-presentation
9. Mission-centered evaluation
Anatomy of Criteria

Criterion—broad statement



Core Component—areas of particular focus

Sub-Component—further elaboration on Core Component, but
not exhaustive (and not always included)
Some criteria now specifically mention certain units
(e.g., advising)
HLC Criteria
7
Criterion Title
and broad
statement
Subcomponent—
specific area
from which
evidence is
expected

HLC Criterion One. Mission
The institution’s mission is
clear and articulated publicly;
it guides the institution’s
operations.
Core Component 1.A. The
institution’s mission is
broadly understood within
the institution and guides
its operations.
 Subcomponent 2.
The institution’s
academic programs,
student support
services, and
enrollment profile
are consistent with
its stated mission.
Core
Component
statement—
particular
aspect of
criterion on
which to
focus
HLC New Criteria
8
 The Five Criteria
1.
Mission
2.
Integrity: Ethical and Responsible Conduct
3.
Teaching and Learning: Quality, Resources, and Support
4.
Teaching and Learning: Evaluation and Improvement
5.
Resources, Planning, and Institutional Effectiveness
Assumed Practices



Won’t check, but we are responsible to ensure/remediate
Notable is B.2.c.1): Faculty participate substantially in:
analysis of data and appropriate action on assessment of
student learning and program completion.
Upcoming Assessment Workshops
9
Developing an Assessment Plan
Thursday, July 19th, 10-11 am
Completing the Assessment Report
Thursday, July 12th, 11-12 am
Tuesday, July 17th, 1-2 pm
Note: workshops/forms overlap
Creating an Assessment Plan
10
AN OPPORTUNITY TO STEP BACK AND PLAN
EFFECTIVE ASSESSMENT AND ANALYSIS
Section 1: Current Unit Status
11
Explain how your unit supports 2020 Strategic
Plan of YSU.
1.

Removed mission; strategic plan operationalizes mission
HLC Criterion One. Mission
The institution’s mission is clear and
articulated publicly; it guides the institution’s
operations.
Core Component 1.A. The institution’s
mission is broadly understood within the
institution and guides its operations.
Sub-component 2. The
institution’s academic
programs, student support
services, and enrollment profile
are consistent with its stated
mission.
HLC Criterion Three. Teaching and Learning:
Quality, Resources, and Support
The institution provides high quality
education, wherever and however its
offerings are delivered.
Core Component 3.E.The institution
fulfills the claims it makes for an
enriched educational environment.
Sub-component 1. Co-curricular
programs are suited to the
institution’s mission and
contribute to the educational
experience of its students.
Example: Strategic Plan Link
12
YSU’s 2020 Strategic Plan
The new University-wide
Strategic Plan is organized
around four cornerstones
 Student success
 The urban research university
mission/transition
 Regional impact
 Institutional accountability
and sustainability
Office of Assessment
(OOA)
Two goals of the OOA (to increase campus
engagement in assessment and to
successfully demonstrate progress in
the HLC Academy for the Assessment
of Student Learning) are directly tied
to 2 cornerstones of the 2020 Plan
Accountability and Sustainability
which emphasizes shared
responsibility.
Student Success since it documents
student achievement in curricular and
co-curricular areas.
The new Engagement in Assessment
Portfolio will provide documentation.
Example: Strategic Plan Link
13
YSU’s 2020 Strategic Plan
Theme 2.2.
Satisfaction with nonacademic experiences and
campus life-including the
importance of
participation in service
learning and community
activities
Office of Community
Involvement
The main goal of this unit is
to increase participation
in community activities
as well as meaningful
reflection on the impact of
those activities on
students’ development
Section 1: Current Unit Status
14
2. Describe the goals for your unit from the previous
academic/fiscal year
Example (from Office of Campus Planning and Community
Partnerships):




to annually update the Campus Master Plan and undertake such studies as
may be required to implement specific recommendations in accordance
with the University’s annual capital improvement program;
to complete the Campus Roadway Study initiated by the City of
Youngstown and focused upon the major arterial roadways serving the
campus;
to implement improved bus service to the campus in collaboration with the
WRTA and the University’s Alternative Transportation Community; and
to collaborate with Student Government to develop a matrix of student
volunteer requirements and opportunities.
Section 1: Current Unit Status
15
3. How do your goals support the learning
environment at YSU?


Think big picture-not just the students, but also faculty, staff
and the university as a system
Every unit contributes to the learning environment!
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational
programs, learning environments, and support services, and it evaluates their
effectiveness for student learning through processes designed to promote
continuous improvement.
Example
16
Support Learning Environment Example (from the
Reading and Study Skills Center):
The Reading and Study Skills Center primarily serves firstyear students….this Center is the “first stop” in building a
foundation for these students’ success and eventual
retention for YSU. By building said skills, instructors and
faculty should be experiencing students in their classrooms
who are capable, responsible students, who read and think
more critically than students who may not have been
exposed to the concepts and skills taught in one of the RSS
courses.
Section 1: Current Unit Status
17
Describe your current assessment activities.
4. Provide a brief summary of unit and/or program
improvement data collected that document your
unit’s assessment activities for the 2011-12
academic/fiscal year



Explain how you measured unit and/or program activities for
continuous improvement
Does not need to be related to learning outcome, only focused
on continuous reflection and improvement
Not data, but methods of collection (e.g., survey, performance
evaluation)
Section 1: Current Unit Status
18
Data collection summary:
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational
programs, learning environments, and support services, and it evaluates their
effectiveness for student learning through processes designed to promote
continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to
educational achievement and improvement through ongoing assessment
of student learning.
Sub-component 2. The institution has clearly stated goals for
student learning and effective processes for assessment of
student learning and achievement of learning goals. The
institution assesses achievement of the learning.
Example
19
Current Assessment Activities Example (excerpt from
Institutional Research and Policy Analysis):
 Collected participation rate data on the CLA, NSSE,
and the Graduate Plans Survey
 Archived data requests from department and unit
faculty and staff
Section 1: Current Unit Status
20
5. How was that data analyzed and used by your unit?


Does not have to be a formalized approach
Ex: a discussion of survey data during a staff meeting
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational
programs, learning environments, and support services, and it evaluates their
effectiveness for student learning through processes designed to promote
continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to
educational achievement and improvement through ongoing assessment
of student learning.
Sub-component 3. The institution uses the information gained
from assessment to improve student learning.
Example
21
Data Analysis and Use Example (from Academic
Advisors):
Survey responses were analyzed and reviewed. The data led
to:



Majors Exploration Fair was held to address concerns raised in the
survey indicating students were uncertain about their major.
A subcommittee formed to address needs of undetermined students
An advising syllabus was developed to address the students’ needs as a
result of narrative comments from students on the survey
Section 1: Current Unit Status
22
6. How did your unit utilize feedback from the OOA
regarding last year’s assessment report?
o If this is your first time completing
a plan or report, answer
“first reporting year”
o This is an important part of
completing a full assessment
cycle/spiral/loop
Assessment process image credit: http://pdx.edu/studentaffairs/assessment-planning-and-practice.
Example
23
OOA Feedback Example (from Academic Advisors):
Based on feedback from OOA to develop a LO that
could lend itself to direct method of measure, added
a LO that included:


Students will be advised in a timely manner
Student will correctly identify their advisor (is being
considered for the future)
Section 2: Assessment Plan Development
24
7. State at least one learning outcome that is relevant
for your unit



Keep to 1 (maybe 2 LO’s)
Keep to 1 focused project (not a laundry list)
What are you interested in?
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational programs,
learning environments, and support services, and it evaluates their effectiveness for student
learning through processes designed to promote continuous improvement.
4.B. The institution demonstrates a commitment to educational achievement and
improvement through ongoing assessment of student learning.
1. The institution has clearly stated goals for student learning and effective
processes for assessment of student learning and achievement of learning
goals.
Section 2: Assessment Plan Development
25
Sources for LO:
 A particular project that your unit will be focusing
on in 2011-12 OR
 LO that staff from your unit develop OR
 YSU’s General Education learning outcomes OR
 The Council for the Advancement of Standards in
Higher Education (CAS) General Standards OR
 Learning Reconsidered
Section 2: Assessment Plan Development
26
Developing a learning outcome:
 Framing Question: What do you want
students/staff/faculty to know and be able to do as
a result of interacting with your office or program
you offer?
 Should be SMART:

Specific, Measurable, Attainable, Realistic, Timely
 Two common methods to writing a learning
outcome:
ABCD Model
 SVP Model

Learning outcome material excerpted from GLACUHO Webinar on Writing Learning Outcomes by Jeff Long, May 2010
Section 2: Assessment Plan Development
27
ABCD Model of Learning Outcomes
Audience-target of the outcome
Behavior-verb in future tense
Condition-activity causing learning
Degree of Achievement (optional)
________ ________ ________ ________
Condition
Audience
Behavior
Degree
Section 2: Assessment Plan Development
28
ABCD Model Examples
________ ________ ________ ________
Condition
Audience
Behavior
(Degree-optional)
 As a result of student staff training, student staff
will be able to list (4 of the 5) department’s values.
 Through publicity and training, staff will correctly
complete online timesheets (90% of the time).
 As a result of this workshop, attendees will
understand the components of a well-written
assessment plan.
Section 2: Assessment Plan Development
29
SVO Model of Learning Outcomes
Subject = learner
Verb = cognitive process
Object = content/knowledge
Note: removes condition (e.g. specific intervention);
somewhat more like academic learning outcomes.
Examples:
 Students will identify academic resources on campus.
 Faculty will use DARS system in advising.
Section 2: Assessment Plan Development
30
Using Verbs in Learning Outcomes
 Avoid “know,” “understand,” “think,” etc. –how will
you measure?
 Consider the level of learning you expect from your
students/staff/faculty—do you want them to
comprehend? Apply knowledge?
 Verb list that considers the level of learning (aka
Bloom’s Taxonomy)--very helpful!:

http://www.clemson.edu/assessment/assessmentpractices/referencem
aterials/documents/Blooms%20Taxonomy%20Action%20Verbs.pdf
Example
31
Academic Advisors Learning Outcomes Example:
Students can independently seek out resources to
aid in evaluating their progress toward degree
completion
2. Students will be knowledgeable of the curriculum
requirements for their chosen major
1.
Section 2: Assessment Plan Development
32
8. How does this learning outcome fit your unit?

Why did you pick this LO?
Example (for fictional Office of Community
Involvement):
This is directly related to our unit’s purpose to
promote the benefit of community involvement
to our students
Section 2: Assessment Plan Development
33
9. How does this learning outcome fit in with the YSU
2020 Strategic Plan?
 Consider how the learning relates to the larger goals of the university
HLC Criterion One. Mission
The institution’s mission is clear and
articulated publicly; it guides the institution’s
operations.
Core Component1.A. The institution’s
mission is broadly understood within the
institution and guides its operations.
Sub-component 2. The
institution’s academic
programs, student support
services, and enrollment profile
are consistent with its stated
mission.
HLC Criterion Three. Teaching and Learning:
Quality, Resources, and Support
The institution provides high quality
education, wherever and however its
offerings are delivered.
Core Component 3.E.The institution
fulfills the claims it makes for an
enriched educational environment.
Sub-component 1. Co-curricular
programs are suited to the
institution’s mission and
contribute to the educational
experience of its students.
Section 2: Assessment Plan Development
34
Example, Questions 7-9—how do they align?
Learning Outcome
Relation to Unit
Relation to YSU’s
Strategic Plan
As a result of a unit publicity
campaign, students will be more
aware of the importance of
community involvement
This is directly related to our unit’s
purpose, to promote the benefit of
community involvement to our students
Promoting the importance of
community involvement helps to
encourage participation, and the
Regional Engagement Cornerstone
outlines the importance of student
involvement in the community to
support quality learning experiences
and positive community change (in
particular Theme 1, Priority 1).
Staff demonstrate awareness of
professional responsibilities of
students serving in community
organizations
By ensuring our staff are responsive to
the needs of the community
organization with which we
partner…we are well positioned to
expand our community service
opportunities
By maintaining positive and responsive
relationships with our community
partners, we enable students to interact
positively with community partners,
and foster the collaborative
relationships with community
organizations outlined in the Regional
Engagement Cornerstone (in particular
Theme 1, Priority 1).
Section 2: Assessment Plan Development
35
Measuring your learning outcome
10. What is the way (or ways) you will measure the
status and/or progress in this learning outcome over
the 2012-13 academic year?



Note: consider tools that would incorporate both direct and
indirect measures
A minimum of one tool but we recommend no more than two
measures
An “exemplary” answer would include a direct measure
Examples: Direct vs. Indirect Measures
36
Direct Measures
Indirect Measures
Direct evidence of learning
outcome performance:
Self-reported achievement
or proxy evidence:
 Factual knowledge
 Surveys, group




questions embedded in a
survey
Presentations
Performances
Simulations
Internship supervisors’
evaluations
discussions, focus groups,
exit
 Interviews
 Employment placement
rates
Methods: Both Direct and Indirect
37
Tools may be direct or indirect, depending upon how
they are used and their purposes:
Method
Indirect
Direct
Minute paper after a
workshop on diversity
Perceptions of moral
dilemmas regarding
diversity
Factual question on the
definition of diversity
Telephone calls to a Unit
Caller’s satisfaction with
technology
A qualitative analyses of
callers knowledge of
technology based on their
questions
Survey after a training
Teacher and learner
satisfaction
Factual question on
knowledge of workshop
content
Section 2: Assessment Plan Development
38
11. What do you anticipate the outcome of your
measures to be? Or, what do you hope to find out?


This can be a check on whether you have chosen the right LO
Results do not need to be “positive” to be valuable
Example
39
Outcome Example (From Alumni and Events
Management):
With our event changes we do receive positive and
constructive feedback. That is what we are always
looking to get from student and the public. No event
is perfect. We are constantly trying to grow and
develop our ceremonies and that is not possible
without constructive feedback.
Section 2: Assessment Plan Development
40
Close the Feedback Loop
12. What are the major constituency groups
for your unit?

Note: think not just within your unit or your supervisory
structure (internal stakeholders), but about
students/faculty/staff, the larger university, the surrounding
community, and beyond (external stakeholders).
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational programs,
learning environments, and support services, and it evaluates their effectiveness for student
learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational
achievement and improvement through ongoing assessment of student learning.
Sub-component 4. The institution’s processes and methodologies to assess
student learning reflect good practice, including the substantial
participation of faculty and other instructional staff members.
Example
41
Constituency Groups Example :
Internally: we share and seek recommendations
from all our staff as well as the Division VP
Externally: we share and seek recommendations
students, campus units who do work on community
involvement




University Honors and Scholars
YSU’s Task Force on Civic Engagement
WYSU
Alumni Services
Section 2: Assessment Plan Development
42
13. How and when do you plan to involve your major
constituency groups in the assessment process e.g.
data sharing and recommendation gathering)?




Mention both the venue and frequency of meeting
share with groups outside your unit
use active ways rather than passive ways to share findings
seek recommendations from groups inside and outside the unit
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational programs,
learning environments, and support services, and it evaluates their effectiveness for student
learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational
achievement and improvement through ongoing assessment of student learning.
Sub-component 3. The institution uses the information gained from
assessment to improve student learning.
Example
43
Feedback Loop Example (from fictional Office of
Community Engagement):
We will compile the results from our survey and
share the information with the groups listed above
through: monthly meetings with internal staff to
update, present data, and solicit recommendations;
soliciting a story about the results in the Jambar for
students (spring 2013); and inviting other campus
units to come in person for a presentation and
discussion of the data (April 2013).
Section 3: Relation to HLC Criteria
44
14. (OPTIONAL FOR 2012) Please provide an example of
evidence from your unit for the new HLC Criterion Five
(Resources, Planning, and Institutional Effectiveness), Core
Component 5A (The institution engages in systematic and
integrated planning.), Sub-component 2 (The institution
links its processes for assessment of student learning,
evaluation of operations, planning, and budgeting.):



Can help to begin familiarizing yourself with new HLC Criteria
How will assessment activities you are planning impact your
planning and budget decisions?
How does your work help contribute to YSU’s body of evidence for
this criterion?
Example
45
Aligning Assessment with Budget Example (from
fictional Office of Community Engagement):
“While we didn’t think of it this way before reading
this question, the 2011 survey results made it clear
that more staff and financial resources needed to be
devoted to a campus campaign. Through this
assessment plan, we are devoting both fiscal and
staff resources in order to carry out plans. Results
from this year’s assessment will inform future
planning and budgets.”
Section 4: Conclusion
46
Executive Summary
15. Summarize the key ideas from Section Two only
Assessment Plan Development



one paragraph is fine
a maximum length of one half page
allows us to compile and develop themes from 40+ units
Section 4: Conclusion
47
16. (Optional) Is there anything else you would like to
share regarding your assessment plan and/or is there
any particular area on which you would like
assistance or feedback?


This is a chance to share something that didn’t “fit” with our
questions but is important to your assessment “story”
Or, it is something about which you could use our help or
advice
Examples
48
 We need help in having a forum to share our data
from our prior year’s report. Can you help us set up
a workshop or forum?
 We are not certain how to incorporate a direct
measure with this learning outcome-do you have
any ideas
Example
49
 We would love some examples of rubrics that have
already been developed to measure community
engagement. Do you have some instruments to
share with us
 We would like to use the institutional account for
survey monkey. How do we go about learning more
about using this survey?
New Online Reporting Format
50
Online Assessment Plan Submission Form:
www.jotform.us/ysuassessment/2012naplan
Thank you for your
participation!
51