Transcript Document

Leveraging Partnerships to
Enhance Assessment
Practices
Beth M. Lingren Clark, Ph.D.
Director: Orientation & First-Year Programs
University of Minnesota-Twin Cities
Outcomes
•
•
•
•
Evolution into assessment framework
Developing assessment plans
Involving key campus partners/stakeholders
Examples of using the data to increase the
effectiveness of measures, themes and
impact institutional projects
Old practices in assessment
• Orientation & First-Year Programs (OFYP) discovered
assessment methods only measured the level of
satisfaction with our program/specific components.
• OFYP asked questions unveiled that participants were
satisfied to very satisfied.
• Satisfaction not outcome orientated or as helpful when
trying to enhance or improve your program.
• OFYP aligned orientation program questionnaires. We
worked with the Office of Measurement Services to help
us identify common questions across all questionnaires.
Old practices in assessment
• Shared data with presenters to
enhance content
• Shared with collegiate representatives
to enhance college components
• Influenced orientation program content
and communications
Needed more information to identify content, make
enhancements/improvements, inform partners to all populations
served
WITH THIS DATA…
TAKING A BROADER
APPROACH
Think about what motivates a study(topic)
and what shapes a study (question or
problem).
“Research is a structured process of
inquiry that draws upon existing
knowledge and theory to advance
general understanding or impact
practices in the field at large.”
-Henscheid, J.M.& Keup, J.R. (2011, p.ix)
Research: Topics/Questions
• How does what we implement impact
retention?
• Is parent involvement important?
• Why do students leave?
• How are we impacting personal and
academic success?
• What key factors predict transfer student
success?
Process: Topics/Questions
•
•
•
•
Who completes online Orientation?
Who attends on campus Orientation?
Who lives in Living Learning Communities
Who has completed the Strengths
assessment?
• What attended Transfer Welcome Day?
• Which International students are coming?
Content: Topics/Questions
• What are the gaps between expectations
and reality of experiences?
• How can we reduce anxieties/concerns?
• How can we increase parent involvement?
• How do we create a sense of belonging?
EVOLUTION TO ASSESSMENT
FRAMEWORK
Where we started…
Welcome Week Committee Structure
Purpose of Committee: (Continued)
– Identify existing instruments to collaborate data collection
– Discuss implementation of the program evaluation
– Discuss attendance tracking at events
Representatives from the following areas: Provost’s
office, Orientation & First-Year Programs, Office of
Measurement Services, Institutional Research, College of
Liberal Arts, Student Activities, Center for Teaching and
Learning, Department of Educational Policy &
Administration, Career Center, Student Leadership &
Engagement
The process
•
Followed cohorts from old program – New Student Weekend (2001 and
2002 cohorts)
•
Welcome Week planning occurred simultaneously with the development
of the First Year Assessment Plan (baseline cohort 2007)
• Did not want to implement WW in isolation
• Support the efficacy of WW
•
Created a mapping processing and aligned with what mattered at the
institution
•
Identified what else existed system-wide to support first year content and
themes
Assessment Cycle
• Identify learning objectives/goals for program
• Map learning objectives/program goals to Student Learning
and Development Outcomes
• Conduct gap analysis
• Change/enhance content based upon gap analysis and
program survey results from previous year (Common
questions)
• Implement program introducing objectives and outcomes to
population
• Evaluate program
• Analyze and share results
• Repeat
How Do We Create a Common
Language of Assessment?
Understanding the assessment framework
• Goal: Initiating undergraduates into Transformative
Education
– Student learning & development outcomes
• Two primary transformations…
– Transforming the campus into the curriculum
– Transforming the undergraduate experience
• Assessing first-year programs
– Have goals we care about
– Know what you know
– Map to goals
How Do We Create a Common
Language of Assessment?
Mapping Outcomes
•
•
Mapping process is a conversation
– Meant to be iterative; refined over time
– Each step informs the next
What we needed to begin mapping
– Schedule and description of
goals/sessions/events
– List of student learning and
development outcomes
– Mapping spreadsheet (essentially a
matrix of sessions/outcomes)
How Do We Create a Common
Language of Assessment?
Mapping Outcomes
•
•
Was done on a large scale (for all of Welcome Week’s content)
Mapping also done on a smaller scale
–
College Day - College of Liberal Arts
The Power of the Common
Question
• We wanted to:
– Create a common language beyond
orientation programming
– Create an intentional plan for content and
evaluative measures
– Implement transformational practices on
campus
– “We wanted to speak a common language”
The Power of the Common
Question (continued)
• We wanted to:
– Create a profile of people completing our
program questionnaires
– Create longitudinal data from one year to
the next as well as within the cohort/year
– Ask more specific questions, AND ask
participants to elaborate on items where
they may not have been satisfied
– Understand the utility of common mapping
DEVELOPMENT OF A FORMAL
FIRST-YEAR ASSESSMENT
PLAN
First-Year
Assessment
Plan
Intent of FYA Plan
• The intent of this process provides data on
student behavior related to:
•
•
•
•
•
•
Matriculation process
Student participation
Student satisfaction
Student success both personally and academically
Longitudinal data utilized
Measure the impact on student retention and other
programs
• The plan helps to inform institution on the
things we were learning about from our
students and their experiences
Involving Campus Partners
•
•
•
•
•
Office of Measurement Services
Undergraduate Education
Collegiate Units
Academic Support Resources
Admissions, Housing, Athletics, Honors,
International
• Office of Institutional Research
• Strengths
• Office for Student Affairs
EXAMPLES OF USING THE
DATA
Matriculation
• Tracking…
– Online Orientation completion
– Orientation attendance
– Document check for international
– Parent attendance
– Welcome Week/Transfer Welcome Day
attendance
– Strengths completion
Participation
Class of 2016
5514
(10th day enrolled)
Commuter
754 13.7%
Residential
4760 86.3%
MN
3493 63.3%
Out of State
1714 31.1%
International
285 5.2%
UHP
572 10.4%
Athletes
139 18.4%
SOC
Band
1077 19.5%
74 1.3%
First-generation
1382 25.7%
NHS Attended OR
5572 103.8% 5501 (10th Day Enrolled)
NHS Parents Attended OR
4791
June OR
3626 67.5%
July OR
1276 23.8%
Attended OR 8/22
129 2.34% (3 non 10th day, 53 international)
Attended OR 8/23
189 3.43%
(4 non 10th day, 76 international)
Attended OR 8/27
307 5.57%
(139 international)
0 Attended Parent OR
2267 41.1%
1 Attended Parent OR
1810 32.8%
2 Attended Parent OR
1370 24.8%
3 Attended Parent OR
66 1.2%
4 Attended Parent OR
1 0.1%
Survey Data
• Pre-Orientation Survey
– Expectations/Welcome messages
– Orientation content (student and parent)
• Common Questions
– Cross college
– Theme/Program content
Survey Data
• Program Evaluations
– Improvement
– Outcomes based
• Check-In Surveys
– Reality of expectations
– Additional support and resources
– Newsletter content
Institutional Data
• National Survey on Student Engagement
– Comparing freshmen and transfer student
experiences
• Student Experience in a Research
University
– Welcome Week and Sense of Belonging
– Transfer Research Grant results
Institutional Data
• Leaver’s Analysis
– Longitudinal data/Collegiate specific
– Importance of Parent/Family Engagement
– @ Home in MN
• Student Retention
• Transfer Grant
– Influence overall experience and engagement
Additional Themes…
• Interactions with faculty are an important part of
the college experience.
• The majority of the students spend between 11-20
hours attending class per week.
• On average, students spend up to 10 hours per
week on social networking (facebook).
• Transfer students have been successful adjusting
to the academic demands and developing effective
study skills but have had more of a difficult time
getting to know their peers and their faculty
Additional Themes…
• Student’s top major anxiety/concerns:
–
–
–
–
–
–
–
–
academic coursework
making friends
financial concerns
deciding on a major
developing skills related to note-taking/studying
time management
stress management
transfer of credits**
Additional Themes…
• The University can feel like a big place so it is
important for students to create community.
–
–
–
–
–
–
Students can do this in their residence hall
Through the Commuter Connection program
In class
At work
Joining a student organization.
Especially important for students who are not from the
Twin Cities area.
Additional Themes…
• New students leave the U for various reasons:
–
–
–
–
–
Financial concerns
Academic
Medical
Family
Other personal issues
Additional Themes…
• At the six to eight week mark, students are:
–
–
–
–
–
seeking more contact with advisers
want help with study skills
more opportunities to socialize**
information on campus libraries**
more interactions with faculty**
Sharing the Data
– College/unit level data is reported to committees
and administrators
– Mapping Process revisited to ensure meeting
goals and desired programmatic outcomes and
University Student Development and Learning
Outcomes
Sharing the Data
– Program evaluation data will be distributed to
content creators and future planning committees
– Data presented to the students (e.g., Advice
from the Class of 2016/Transfer?)
– Annual reports disseminated to leadership and
decision-makers within Undergraduate
Education
Where Do We Go From Here?
•
Utilize the Data
– Use the findings for continuous improvement efforts
• Did we do what we set out to do?
• What can we do better?
• What was successful and should be continued?
• How do we alter content to address new issues?
– Compare findings year to year
• Are students attitudes/perceptions changing?
• Are there new transformational issues?
– Longitudinal Analysis
• Has Welcome Week impacted the student experience?
• Are there correlations between student attendance, satisfaction
retention, graduation rates, etc.? (Utilize orientation database)
Inspiring Our Work
• Influence program content, delivery, and
messaging
• Integrate data results directly into what we share
with students and families
• Share with colleges and units to influence the way
we do our work
• Strengthens partnerships
• Provide evidence to work with units to make
strategic changes
“Educators who work to improve the
experience of college students, especially
students facing transitional periods, are
typically driven by passion for the work.
At their best, these educators combine
this passion with a solid understanding of
how students experience college, what
helps or hinders the students’ progress,
and what outcomes result from their
experience.
-Henscheid, J.M.& Keup, J.R. (2011, p.17)
Resources
http://www.ofyp.umn.edu/en/welcome-week.html
http://www.ofyp.umn.edu/en/survey-results.html
http://www.ofyp.umn.edu/ofypmedia/pdfs/highered
/reports/2011_OFYPAnnual.pdf
http://www.oir.umn.edu/
http://www.oir.umn.edu/surveys/seru
Reference
Henscheid, J.M. & Keup, J.R. (2011).
Crafting and conducting research on
student transitions. Columbia, SC:
University of South Carolina, National
Resource Center for the First-Year
Experience and Students in Transition.
“When done well, quantitative research
has the potential to enhance knowledge
and inform programs, policies and
pedagogies that educators use to serve
students and contribute to their success”
(Henscheid & Keup, 2011, p.84)
Questions/Comments
Beth Lingren Clark, Ph.D.
Director: Orientation & First-Year Programs
[email protected]