Transcript Assessment Workshop - Youngstown State University

Non-Academic Assessment:
Completing an Assessment Report
OFFICE OF ASSESSMENT
JULY 12, 2012
Goals of Workshop
2
 Current assessment context
 What’s new this year
 Introduce new Higher Learning Commission Criteria
 Review items on the Assessment Report template
 Walk through new online reporting form
Assessment Context
3
Assessment Context
4
Assessment Context



Last year participating in the Higher Learning Commission’s
(HLC) Academy for the Assessment of Student Learning
New HLC Criteria effective January 1, 2013
Campus focus on 2020 Strategic Plan alignment
Assessment Reporting Themes



Focus on use of data
Reflect on changes and impact on learning
Streamline reporting process
Future Goals


Longer reporting cycle
Integration with annual reporting
What’s New This Year
5





Online reporting
Fewer questions
Questions “tagged” with New HLC Criteria
Optional question on New HLC Criteria
Report quality threshold to 70%
HLC New Criteria
6
 Guiding Values, includes:
1. Focus on student learning
4. Culture of continuous improvement
5. Evidence-based institutional learning and self-presentation
9. Mission-centered evaluation
Anatomy of Criteria

Criterion—broad statement



Core Component—areas of particular focus

Sub-Component—further elaboration on Core Component, but
not exhaustive (and not always included)
Some criteria now specifically mention certain units
(e.g., advising)
HLC Criteria
7
Criterion Title
and broad
statement
Subcomponent—
specific area
from which
evidence is
expected

HLC Criterion One. Mission
The institution’s mission is
clear and articulated publicly;
it guides the institution’s
operations.
Core Component 1.A. The
institution’s mission is
broadly understood within
the institution and guides
its operations.
 Subcomponent 2.
The institution’s
academic programs,
student support
services, and
enrollment profile
are consistent with
its stated mission.
Core
Component
statement—
particular
aspect of
criterion on
which to
focus
HLC New Criteria
8
 The Five Criteria
1.
Mission
2.
Integrity: Ethical and Responsible Conduct
3.
Teaching and Learning: Quality, Resources, and Support
4.
Teaching and Learning: Evaluation and Improvement
5.
Resources, Planning, and Institutional Effectiveness
Assumed Practices



Won’t check, but we are responsible to ensure/remediate
Notable is B.2.c.1): Faculty participate substantially in:
analysis of data and appropriate action on assessment of
student learning and program completion.
Upcoming Assessment Workshops
9
Developing an Assessment Plan
Thursday, July 19th, 10-11 am
Completing the Assessment Report
Tuesday, July 17th, 1-2 pm
Note: workshops/forms overlap
Completing an Assessment
Report
10
Unit Information
11
Explain how your unit supports 2020 Strategic
Plan of YSU.
1.

Removed mission; strategic plan operationalizes mission
HLC Criterion One. Mission
The institution’s mission is clear and
articulated publicly; it guides the institution’s
operations.
Core Component 1.A. The institution’s
mission is broadly understood within the
institution and guides its operations.
Sub-component 2. The
institution’s academic
programs, student support
services, and enrollment profile
are consistent with its stated
mission.
HLC Criterion Three. Teaching and Learning:
Quality, Resources, and Support
The institution provides high quality
education, wherever and however its
offerings are delivered.
Core Component 3.E.The institution
fulfills the claims it makes for an
enriched educational environment.
Sub-component 1. Co-curricular
programs are suited to the
institution’s mission and
contribute to the educational
experience of its students.
Example: Strategic Plan Link
12
YSU’s 2020 Strategic Plan
The new University-wide
Strategic Plan is organized
around four cornerstones
 Student success
 The urban research university
mission/transition
 Regional impact
 Institutional accountability
and sustainability
Office of Assessment
(OOA)
Two goals of the OOA (to increase campus
engagement in assessment and to
successfully demonstrate progress in
the HLC Academy for the Assessment
of Student Learning) are directly tied
to 2 cornerstones of the 2020 Plan
Accountability and Sustainability
which emphasizes shared
responsibility.
Student Success since it documents
student achievement in curricular and
co-curricular areas.
The new Engagement in Assessment
Portfolio will provide documentation.
Example: Strategic Plan Link
13
YSU’s 2020 Strategic Plan
Theme 2.2.
Satisfaction with nonacademic experiences and
campus life-including the
importance of
participation in service
learning and community
activities
Office of Community
Involvement
The main goal of this unit is
to increase participation
in community activities
as well as meaningful
reflection on the impact of
those activities on
students’ development
Unit Information
14
2. Describe the goals for your unit from the previous
academic/fiscal year
Example (from Office of Campus Planning and Community
Partnerships):




to annually update the Campus Master Plan and undertake such studies as
may be required to implement specific recommendations in accordance
with the University’s annual capital improvement program;
to complete the Campus Roadway Study initiated by the City of
Youngstown and focused upon the major arterial roadways serving the
campus;
to implement improved bus service to the campus in collaboration with the
WRTA and the University’s Alternative Transportation Community; and
to collaborate with Student Government to develop a matrix of student
volunteer requirements and opportunities.
Learning Outcome Focus
15
3. State at least one learning outcome that is relevant
for your unit



Keep to 1 (maybe 2 LO’s)
Keep to 1 focused project (not a laundry list)
What are you interested in?
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational programs,
learning environments, and support services, and it evaluates their effectiveness for student
learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational
achievement and improvement through ongoing assessment of student learning.
Sub-component 1. The institution has clearly stated goals for student
learning and effective processes for assessment of student learning and
achievement of learning goals.
Developing Learning Outcomes
16
Sources for LO:
 A particular project that your unit will be focusing
on in 2011-12 OR
 LO that staff from your unit develop OR
 YSU’s General Education learning outcomes OR
 The Council for the Advancement of Standards in
Higher Education (CAS) General Standards OR
 Learning Reconsidered
Developing Learning Outcomes
17
Developing a learning outcome:
 Framing Question: What do you want
students/staff/faculty to know and be able to do as
a result of interacting with your office or program
you offer?
 Should be SMART:

Specific, Measurable, Attainable, Realistic, Timely
 Two common methods to writing a learning
outcome:
ABCD Model
 SVP Model

Learning outcome material excerpted from GLACUHO Webinar on Writing Learning Outcomes by Jeff Long, May 2010
Developing Learning Outcomes
18
ABCD Model of Learning Outcomes
Audience-target of the outcome
Behavior-verb in future tense
Condition-activity causing learning
Degree of Achievement (optional)
________ ________ ________ ________
Condition
Audience
Behavior
Degree
Developing Learning Outcomes
19
ABCD Model Examples
________ ________ ________ ________
Condition
Audience
Behavior
(Degree-optional)
 As a result of student staff training, student staff
will be able to list (4 of the 5) department’s values.
 Through publicity and training, staff will correctly
complete online timesheets (90% of the time).
 As a result of this workshop, attendees will explain
the components of a well-written assessment
report.
Developing Learning Outcomes
20
SVO Model of Learning Outcomes
Subject = learner
Verb = cognitive process
Object = content/knowledge
Note: removes condition (e.g. specific intervention);
somewhat more like academic learning outcomes.
Examples:
 Students will identify academic resources on campus.
 Faculty will use DARS system in advising.
Developing Learning Outcomes
21
Using Verbs in Learning Outcomes
 Avoid “know,” “understand,” “think,” etc. –how will
you measure?
 Consider the level of learning you expect from your
students/staff/faculty—do you want them to
comprehend? Apply knowledge?
 Verb list that considers the level of learning (aka
Bloom’s Taxonomy)--very helpful!:

http://www.clemson.edu/assessment/assessmentpractices/referencem
aterials/documents/Blooms%20Taxonomy%20Action%20Verbs.pdf
Example: Learning Outcomes
22
Academic Advisors Learning Outcomes Example:
Students can independently seek out resources to
aid in evaluating their progress toward degree
completion
2. Students will be knowledgeable of the curriculum
requirements for their chosen major
1.
Learning Outcome Focus
23
4. How does this learning outcome fit your unit?

Why did you pick this LO?
Example (for fictional Office of Community
Involvement):
This is directly related to our unit’s purpose to
promote the benefit of community involvement
to our students
Learning Outcome Focus
24
5. How does this learning outcome fit in with the YSU
2020 Strategic Plan?
 Consider how the learning relates to the larger goals of the university
HLC Criterion One. Mission
The institution’s mission is clear and
articulated publicly; it guides the institution’s
operations.
Core Component1.A. The institution’s
mission is broadly understood within the
institution and guides its operations.
Sub-component 2. The
institution’s academic
programs, student support
services, and enrollment profile
are consistent with its stated
mission.
HLC Criterion Three. Teaching and Learning:
Quality, Resources, and Support
The institution provides high quality
education, wherever and however its
offerings are delivered.
Core Component 3.E.The institution
fulfills the claims it makes for an
enriched educational environment.
Sub-component 1. Co-curricular
programs are suited to the
institution’s mission and
contribute to the educational
experience of its students.
Learning Outcome Focus
25
Example, Questions 3-5—how do they align?
Learning Outcome
Relation to Unit
Relation to YSU’s
Strategic Plan
As a result of a unit publicity
campaign, students will be more
aware of the importance of
community involvement
This is directly related to our unit’s
purpose, to promote the benefit of
community involvement to our students
Promoting the importance of
community involvement helps to
encourage participation, and the
Regional Engagement Cornerstone
outlines the importance of student
involvement in the community to
support quality learning experiences
and positive community change (in
particular Theme 1, Priority 1).
Staff demonstrate awareness of
professional responsibilities of
students serving in community
organizations
By ensuring our staff are responsive to
the needs of the community
organization with which we
partner…we are well positioned to
expand our community service
opportunities
By maintaining positive and responsive
relationships with our community
partners, we enable students to interact
positively with community partners,
and foster the collaborative
relationships with community
organizations outlined in the Regional
Engagement Cornerstone (in particular
Theme 1, Priority 1).
Assessment and Evaluation of LO Data
26
6. Provide a summary of LO-related data collected for
the previous academic/fiscal year (i.e., explain how
you measured the LO-related activities).



Include methods used and number of individuals assessed
Minimum of one tool, but we recommend no more than two
Have one direct measure to receive an “exemplary” score
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational programs,
learning environments, and support services, and it evaluates their effectiveness for student
learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational
achievement and improvement through ongoing assessment of student learning.
Sub-component 2. The institution assesses achievement of the learning
outcomes that it claims for its curricular and co-curricular programs.
Examples: Direct vs. Indirect Measures
27
Direct Measures
Indirect Measures
Direct evidence of learning
outcome performance:
Self-reported achievement
or proxy evidence:
 Factual knowledge
 Surveys




questions embedded in a
survey
Presentations
Performances
Simulations
Internship supervisors’
evaluations
 Group discussions
 Focus groups
 Exit interviews
 Employment placement
rates
Methods: Both Direct and Indirect
28
Tools may be direct or indirect, depending upon how
they are used and their purposes:
Method
Indirect
Direct
Minute paper after a
workshop on diversity
Perceptions of moral
dilemmas regarding
diversity
Factual question on the
definition of diversity
Telephone calls to a Unit
Caller’s satisfaction with
technology
A qualitative analyses of
callers knowledge of
technology based on their
questions
Survey after a training
Teacher and learner
satisfaction
Factual question on
knowledge of workshop
content
Example: Data Summary
29
Rate of Participation (from Housing and Residence
Life):
Housing & Residence Life participated in the national
EBI Resident Study. 618 of our 745 residents (83%)
of our residents participated. This was double the
rate of participation from the previous year, before
“Love Where You Live” was established.
Example: Data Summary
30
Results (from Housing and Residence Life):
From EBI Resident Study:
 excellent scores and showed marked improvement in
both factor means and national rankings.
 Of 19 factors assessed in the EBI Resident Study, YSU
was ranked in the top 100 of 300 participating schools
in seven factors, including overall program
effectiveness.
 More important than rankings, our improved scores
and participation reflected an increased investment in
our community by our residents.
Assessment and Evaluation of LO Data
31
For questions 7-9, please provide an evaluation of
the 2011-2012 data you collected related to your
unit’s learning outcome.
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational programs,
learning environments, and support services, and it evaluates their effectiveness for student
learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational
achievement and improvement through ongoing assessment of student learning.
Sub-component 3. The institution uses the information gained from
assessment to improve student learning.
Assessment and Evaluation of LO Data
32
7. Based on the data, what is one key strength of
(student, staff, or faculty) learning?
8. Based on the data, what is one important challenge
to (student, staff, or faculty) learning?


Strengths/challenges of learning, not of your unit
You can bring in other sources of information, but be sure that
the conclusions you form are based on the data you’ve
collected
Example: Data Evaluation
33
Key Strength Example (from Housing):
Our positive assessment results tell us that our
campaign is making a difference—students are
more aware of our staff’s efforts to create a more
conducive living space, students are more
intentional about maintaining a strong community
and are more respectful of their fellow residents.
Example: Data Evaluation
34
Challenge (from Housing):
 engaging our residents to remain actively involved
in the residence hall and University community.
 generating new ideas that keep our programs
evolving.
 increasing faculty awareness about our programs
& incorporating more faculty into our programs.
Assessment and Evaluation of LO Data
35
9. What action step has been identified based on this
evaluation of the data? What is the status of this
action step?


Considering the evaluation you conducted in questions 7 & 8,
what could be changed in order to improve the learning
outcome you are examining?
Data may not indicate the need for change; this is fine. Include
evidence for your unit’s conclusion based on the data.
Example: Action Step
36
Action Step Example (from Housing):
The Director of Housing & Residence Life conducted
Speak Out Be Heard meetings in each residence hall
to assess what University Housing could do to
increase the desirability of student housing. Many
of the comments made in those meetings became
improvement projects and several have already
been completed.
Relation to New HLC Criteria
37
10. (OPTIONAL FOR 2012) Please provide an example of
evidence from your unit for the new HLC Criterion Five
(Resources, Planning, and Institutional Effectiveness), Core
Component 5A (The institution engages in systematic and
integrated planning.), Sub-component 2 (The institution
links its processes for assessment of student learning,
evaluation of operations, planning, and budgeting.):



Can help to begin familiarizing yourself with new HLC Criteria
How will assessment activities you are planning impact your
planning and budget decisions?
How does your work help contribute to YSU’s body of evidence for
this criterion?
Example: HLC Criteria
38
Aligning Assessment with Budget Example (from
fictional Office of Community Engagement):
While we didn’t think of it this way before reading
this question, the 2010 survey results made it clear
that more staff and financial resources needed to be
devoted to a campus campaign. When planning for
the 2011-12 fiscal year, funding was allocated
specifically to the student awareness campaign, and
staff responsibilities were arranged to allow for
sufficient attention to outreach activities during the
year.
Use of Data
39
11. Feedback Loop: How and when are both internal and
external stakeholder involved in the assessment process
(e.g., data sharing and recommendation gathering)?



share with groups outside your unit
use active ways rather than passive ways to share findings
seek recommendations from groups inside and outside the unit
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational programs,
learning environments, and support services, and it evaluates their effectiveness for student
learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational
achievement and improvement through ongoing assessment of student learning.
Sub-component 4. The institution’s processes and methodologies to assess
student learning reflect good practice, including the substantial
participation of faculty and other instructional staff members.
Example: Feedback Loop
40
Feedback Loop Example (from fictional Office of
Community Engagement):
We compiled the results from our survey and shared
the information by: meeting monthly with internal
staff to present data and solicit recommendations;
soliciting a story about the results in the Jambar for
students in March; and inviting other community
service-oriented campus units (e.g., University
Honors, WYSU, Alumni Services) to come in person
for a presentation and discussion of the data on
April 17th.
Use of Data
41
12. Considering action steps from previous years
(2008-2011), what has been an impact on student
learning as a result of (one of) those action steps?
 You can refer to previous reports for action steps
 Only one specific example is needed
 In particular, we are interested in changes you made previously
that have since impacted student/faculty/staff learning
HLC Criterion Four: Teaching and Learning: Evaluation and Improvement
The institution demonstrates responsibility for the quality of its educational programs,
learning environments, and support services, and it evaluates their effectiveness for student
learning through processes designed to promote continuous improvement.
Core Component 4.B. The institution demonstrates a commitment to educational
achievement and improvement through ongoing assessment of student learning.
Sub-component 3. The institution uses the information gained from
assessment to improve student learning.
Example: Previous Action Step Impact
42
Impact of previous action steps from fictional Office
of Community Engagement:
The results of the 2010 student survey made it apparent to
us that there was a need to increase student awareness
about the opportunities for and importance of community
service activities as a student. We decided that for our action
step we would undertake a campus campaign to increase
awareness; the campaign was completed in February 2012.
The results of the survey repeated in spring 2012 shows that
students had an increase in the perceived value of
participation—we see that as a positive impact of the
campaign and a change in the students’ learning.
Use of Data
43
13. How did your unit utilize feedback from the OOA
regarding last year’s assessment report?
o If this is your first time completing
a plan or report, answer
“first reporting year”
o This is an important part of
completing a full assessment
cycle/spiral/loop
Assessment process image credit: http://pdx.edu/studentaffairs/assessment-planning-and-practice.
Example: OOA Feedback
44
OOA Feedback Example (from Academic Advisors):
Based on feedback from OOA to develop a LO that
could lend itself to direct method of measure, added
a LO that included:


Students will be advised in a timely manner
Student will correctly identify their advisor (is being
considered for the future)
Conclusion
45
14. Executive Summary: Summarize the key ideas from
your report



one paragraph is fine
a maximum length of one half page
allows us to compile and develop themes from 40+ units
Conclusion
46
15. (Optional) Is there anything else you would like to
share regarding your assessment report and/or is
there any particular area on which you would like
assistance or feedback?


This is a chance to share something that didn’t “fit” with our
questions but is important to your assessment “story”
Or, it is something about which you could use our help or
advice
Example: Additional Information
47
 We would love some examples of rubrics that have
already been developed to measure community
engagement. Do you have some instruments to
share with us
 We would like to use the institutional account for
survey monkey. How do we go about learning more
about using this survey?
Example: Additional Information
48
 We need help in having a forum to share our data
from our prior year’s report. Can you help us set up
a workshop or forum?
 We are not certain how to incorporate a direct
measure with this learning outcome-do you have
any ideas
New Online Reporting Format
49
Online Assessment Report Submission Form:
www.jotform.us/ysuassessment/2012nareport
Thank you for your
participation!
50