Transcript Using Assessment of Undergraduate Research to Improve

Using Assessment of Undergraduate
Research to Improve Programming
and Student Outcomes
Presentation made to SoTE Conference
March 28, 2014
Susan Larson
What is undergraduate research,
scholarship & creative activity (UR)?

Council on Undergraduate Research: An inquiry or
investigation conducted by an undergraduate student that
makes an original intellectual or creative contribution to
the discipline. www.cur.org

Concordia College: Undergraduate research, encompassing all
forms of scholarship and creative activity, is an investigation or
inquiry conducted by a student, under the mentorship of a
faculty member, which contributes to a high-level intellectual
or creative outcome.
Defining UR








Student, process centered  Outcome,
product centered
Student initiated  Faculty initiated
All students Honors students
Curriculum based  Co-curricular fellowships
Collaborative  Individual
Original to the student  Original to the discipline
Multi- or interdisciplinary  Disciplinary based
Campus community audience  Professional audience
- Beckman & Hensel, CUR Quarterly, 2009
Value of UR

Cognitive and intellectual growth




Gains in knowledge and skills
Academic achievement and educational attainment
Professional growth and advancement
Personal growth and development
Assessing the Benefits of UR

Survey of Undergraduate Research Experiences (SURE)




Includes demographic variables, learning gains, and evaluation of
aspects of summer programs.
Students report the greatest learning gains on items
related to the research process, scientific problems, and
lab techniques
Students report learning gains on personal development
measures (e.g., tolerance for obstacles and working
independently)
Summer research sustains an interest in pursuing science

Lopatto (2004), Cell Biol Educ, 270-277.

http://www.grinnell.edu/users/lopatto
Assessing the Benefits of UR

Assessment of undergraduate research opportunity
program (UROP) at Univ. of Michigan



Longitudinal assessment evaluating student retention, academic
performance, engagement, pursuit of graduate and professional
education
Evaluated UROP students and matched control group:
individuals who applied, but did not participate
Significant positive impact of UROP on degree completion
rates (most notably for African American students) and
likelihood of pursuing post-graduate education

Nagda, Gregerman, et al. (1998), The Review of Higher Education, 55-72

http://www.lsa.umich.edu/urop/aboutus/evaluationassessment
Assessing the Benefits of UR

Ethnographic study of summer undergraduate research
(UR) experiences



Initial work done at four liberal arts colleges
Focus groups with students and faculty
Both groups described gains related to process of
"becoming a scientist"


Faculty focused on “professional socialization”
Students more focused on “personal and intellectual
development”

Hunter, Laursen, Seymour (2007). Science Education, 36-74.

http://www.colorado.edu/eer/index.html
Why Should YOU Assess UR?

Evaluate the effectiveness of your program




Determine if your programming is meeting your objectives
Develop your programming
For donors, your administrators, grant writing
Understand the benefits of UR for your students


To improve student learning in your context
To improve faculty mentoring

To broaden participation in UR

But, we don’t have to re-invent the wheel
What Should You Assess?

Student (learning) outcomes






What did you learn in this experience?
How did this affect your educational experience, career, etc.?
Strategies for developing an assessment program
Example of assessment program at my home institution
Program effectiveness
Make use of COEUR

Characteristics of Excellence in
Undergraduate Research
Questions to Ask

What do you want to know?

Should align with the objectives of your program







Student researchers more likely to go to grad school?
Self-confidence and self-efficacy?
Impact on departments/advisors scholarly activity?
Who is the target of your assessment?
Who does the assessing?
What is the quality of the instrument?
How will you use the results?



Benchmarking
Program improvement
Understanding student learning outcomes
Assessment of UR and SoTL

There are great opportunities for SoTL projects related
to undergraduate research

So, consider framing your UR assessment to answer the
questions you need to ask, but also the questions you want to
ask.


E.g., benefits to faculty: often discussed, less often assessed
E.g., how is UR different/similar to other forms of engaged learning at
UVU?
Designing a Sustainable Assessment
Program

State expected outcomes, clear and measurable



Research skills, student development goals
Program goals
Determine methods and criteria to assess outcomes


Direct assessment and/or indirect assessment
Qualitative and/or quantitative assessment


Identify and collect information


Start small, start with successes, minimize the burden
Set priorities, conduct periodic and timely assessments
Use the data!

Share results, use assessment results in planning and progam
evaluation
The Assessment Cycle

Assessment of UR is an on-going process of

Establishing clear, measurable expected outcomes for UR
programs

Providing participants (students, faculty, institutions,
system) with opportunities to achieve those outcomes

Systematically gathering, analyzing, and interpreting
evidence to determine how well UR activities match our
expectations

Using the resulting information to improve UR programs
and initiatives

Maki, Journal of Academic Leadership, 2002
Example of an Assessment Plan for UR:
Concordia College

The assessment plan for the Office of Undergraduate
Research seeks to accomplish the following goals



Provide a count of undergraduate research, scholarship
and creative activities at Concordia, including
demographic data
Assess learning achieved in undergraduate research
experiences
Evaluate the effectiveness of the Office of Undergraduate
Research.
Counting UR

Faculty surveys administered by the Office of Institutional
Research

Faculty report on





# of students they mentored
# of projects they mentored
Funding for the projects
Outcomes of the projects, e.g., presentations, conferences
Survey also distinguishes classroom-based and
individually-mentored research
Other Ways to Count UR

Enrollment in directed research courses
# of students participating in funded travel opportunities
# of students presenting at on-campus symposia or
related events
Course syllabi which note an extensive research
experience
Count of students who engage in off campus research
Make use of department annual reports

OTHERS?





The Challenge and Value of the Count

Do you need to count everyone?


Can we even do it if we tried?
Is counting enough?


Do you need to know what happens to these students?
Do you need to know their demographic characteristics?
Assessment of Student Learning

Students are expected to achieve some of the
following learning goals/objectives.
I.
II.
III.
IV.
V.
VI.
formulate questions in their field
demonstrate ability to understand literature
explain methodology
demonstrate proficiency in the use of the tools and
instruments of the area of study
analyze and interpret literature and/or results as
appropriate to their field of study
demonstrate ability to communicate (writing,
presenting) within the area of study, and
Assessment of Summer Research
Program

Indirect assessment -- student surveys




In-house self-report assessment of learning gains and
confidence
Responses to open-ended question about learning
outcomes
Evaluation of the degree to which students met the goals
in the learning agreement they sign with their mentor at
the start of the summer.
Science students complete Lapatto’s SURE survey
(http://www.grinnell.edu/academic/csla/assessment/sure).
Assessment of Summer Research
Program

Faculty mentor survey


Faculty rate their student’s learning gains and respond to open-ended
questions about learning gains
Faculty and students rate their satisfaction with the
summer program activities.
How Much Did You Gain as a Result of
Your Summer Research Experience?
0= No gain
No
A little Moder Good Great
5=Great gain
gain
ate
Formulating research questions
All students: mean=3.77,
19%
13%
39%
29%
SD=1.09
1st year: mean=3.90, SD=.99
10%
20%
40%
30%
1st yr mentors: mean= 3.78,
11%
22%
44%
22%
SD=.97
Proficiency in the use of the tools and instruments in your area of study
All students: mean=4.13,
13%
29%
58%
SD=.88
1st year: mean=4.80, SD=.42
20%
80%
1st yr mentors: mean=3.78,
44%
33%
22%
SD=.83
Analyzing data & interpreting results
All students: mean=4.45,
10%
19%
35%
32%
SD=.72
1st year: mean=4.00, SD=1.05
10%
20%
30%
40%
1st yr mentors: mean=4.56,
33%
33%
33%
SD=.53
Meeting Learning Agreement Goals
Considering your learning agreement,
rate your learning gains for the goals
specified in the agreement.
All Concordia summer research
students
1st year research students
1st year research student mentors
Mean
S.D.
3.96
.96
4.40
.52
4.11
.86
Open Ended Question

“What were one or two of the most important things you learned
from your research experience?”
 Research skills specific to specific projects
 Understanding of the research process


Understanding of science


E.gs., interpreting data; reading scientific literature; the importance of
a detailed lab notebook.
E.g., understanding how scientists work together; the importance of
previously published findings; appreciation of the differing perspectives
individuals from different disciplines bring to a problem
Personal gains

Patience; honesty; flexibility; independence; the importance of team
work.
Summer Program Activities
Impression of summer activities
Mean
S.D.
3.65
.63
3.88
.71
(very useful/enjoyable = 5)
How useful were the summer program
activities?
How enjoyable were the summer
program activities?



Allowed them to develop some practical skills
Appreciated the chance to interact with other student
researchers
Some felt that the workshops targeted first-time
researchers and took too much time away from the lab
Other Ways to Assess SLOs

Institutional assessment data



Departmental assessment programs
Nationally available survey’s





E.g., NSSE and critical thinking measures employed by the
institution
E.g., SURE and others by Lapotto
E.g., Undergraduate Research Student Self-Assessment
(URSSA), on salgsite.org
Rubrics for scoring posters/talks given by research
students
Reflective essays for students who travel and present UR
Focus groups
Other Ways to Assess Student
Outcomes

Post graduate activities




Alumni survey
Career center survey
Publications and presentations generated by involvement
in UR
Review CUR’s assessment database for ideas

https://www.zotero.org/groups/curassessmentbibliography/items/
Using the Assessment Data

Closing the assessment loop

Lower scores on communicating research


Lower rating on learning gains related to ethics


Added an oral presentation expectation to the summer program
Will enhance summer program workshop on ethics
Information about mentor availability and its impact on the
summer experience

Conversations with mentors about their expectations
Program Evaluation


May want to go beyond assessing the experience of
students to understand if the UR program you offer is
robust and effective
Could be accomplished with the assistance of
COEUR
http://www.cur.org/publications/publication_listings/coeur/
What is COEUR?


A summary of best practices that support and sustain
highly effective undergraduate research environments
The purpose of COEUR is to provide a guide for
those who wish to build, evaluate, and maintain
robust, productive, meaningful and sustainable
undergraduate research programs





Function as a guide for strategic planning
Provide programmatic and institutional benchmarking
against national standards
Outline a structure for UR self-study
Provide a framework for assessment and evaluation of UR
The Characteristics

Campus mission and culture


Administrative support


e.g., UR program office
Research infrastructure


e.g., institutional commitment
e.g., space, instrumentation and equipment
Professional development opportunities

e.g., research leaves, mentorship training
The Characteristics

Recognition


External funding


e.g., internal and external funding for research
Dissemination


e.g., UR in promotion and tenure guidelines
e.g., peer-reviewed publication, exhibition, or performance;
student research conferences
Student-centered issues

e.g., community of student scholars
The Characteristics

Curriculum


Summer research program


e.g., faculty & student compensation
Assessment activities


e.g., integration of teaching and research; student course
credit for research
e.g., assessment of student learning
Strategic Planning
Simple Evaluation Rubrics
CUR’s Characteristic of
Excellence
Campus mission and culture

Institutional commitment

Scholarly faculty

Faculty commitment

Broad disciplinary
participation

Accessible opportunities for
undergraduates

Integration with other
engaging and high-impact
opportunities
Thinking about your own context
successes at your institution
challenges at your institution
What is your marker of
excellence?
EXAMPLE
Administrative
support

Student travel
student funding
EXAMPLE
X# of students traveling to
professional meetings
How do you measure this?
EXAMPLE
Student receiving travel funds fill
out pre-post survey; reported
annually; count %age of students
doing UR who travel to meetings
What evidence do you have
that this component is a
priority at your institution?
EXAMPLE
Increased travel funds from X$ per year
to X$$ per year, student travel award
recipients published in alumni magazine
http://www.cur.org/publications/publication_listings/coeur/
Conclusions

There are many published benefits of UR


But, local, contextualized assessment of UR are important
When undertaking an assessment program


Know what you are assessing and why
Keep it simple and sustainable



Don’t assess everything all the time
Keep your focus on enhancing student learning
Use assessment data and program evaluation to improve
your UR programming

COEUR can be a useful tool for improving the experience you
offer students