Transcript Slide 1

Before & After:
What Undergraduates and Alumni Say About
Their College Experience and Outcomes
Angie L. Miller, NSSE & SNAAP Research Analyst
Amber D. Lambert, NSSE & SNAAP Research Analyst
Becca Houghton, SNAAP Project Coordinator
Assessment Institute
October 30th, 2012
Introduction
 Surveys are a common means of assessment in
higher education
 Student surveys are conducted on a variety of
topics, from student engagement to use of
campus resources to faculty evaluations
 Alumni surveys are used to gather information
about satisfaction, acquired skills, and career
attainment
Research Questions
 Question 1: Are there differences in how
students and alumni perceive aspects of their
institutional experiences and the skills and
competencies that they acquire at their
institutions?
 Question 2: What discipline-specific strengths
and weaknesses exist when exploring students’
reported development of skills and abilities?
What is SNAAP?
Strategic National Arts Alumni Project
 On-line annual survey of arts graduates
 Investigates educational experiences and
career paths
 Provides findings to educators and
policymakers to improve arts training, inform
cultural policy, and support artists
Who is Surveyed?
 Graduates of:
 Arts schools, departments, or
programs in colleges and
universities
 Independent arts colleges
 Arts high schools
 Both graduate and
undergraduate degree
recipients
 All arts disciplines
SNAAP Questionnaire Topics
1.
Formal education and degrees
2.
Institutional experience and satisfaction
3.
Postgraduate resources for artists
4.
Career
5.
Arts engagement
6.
Income and debt
7.
Demographics
SNAAP 2011
Administration Information
 Administered in Fall 2011
 66 participating institutions
 58 postsecondary and 8 high schools
 Over 36,000 total respondents
What is NSSE?
National Survey of Student Engagement
 NSSE gives a snapshot of college student
experiences in and outside of the classroom
by surveying first-year and senior students
 NSSE items represent good practices related
to desirable college outcomes
 Indirect, process measures of student
learning and development
NSSE Purpose
 NSSE annually gathers
valid, reliable
information on the
extent to which
students engage in and
are exposed to proven
educational practices
that correspond to
desirable learning
outcomes.
•
Results indicate how
students spend their
time and what they
gain from college.
NSSE 2012
Administration Information
 Administered in Spring 2012
 546 participating U.S. institutions
 Over 285,000 total respondents
 Each year, experimental item sets
appended at end of core survey
Question 1: Methodology
 Participants from 6 institutions that
participated in both SNAAP11 and NSSE12
 Senior NSSE respondents from arts majors
in corresponding SNAAP participating
programs (n = 222)
 Alumni of undergraduate SNAAP programs
from graduating cohorts of 2001-2010
(n = 593)
Question 1: Respondents
Characteristics
NSSE
SNAAP
Female
72%
61%
U.S. citizen
98%
98%
First Generation status
37%
30%
White
68%
89%
Black
5%
4%
Latino/Hispanic
7%
3%
Asian
1%
5%
Race/Ethnicity*
*SNAAP asks race/ethnicity in a check-all format;
NSSE asks race/ethnicity in a forced-choice format
Question 1: SNAAP items
Question 1: NSSE items
Question 1: Results
Means comparison for overall rating of
institutional experience (4-point scale from
“Poor” to “Excellent”) suggests that alumni give
higher general appraisals
Student Alumni
Mean
Mean
Overall experience
*p<.05; **p<.01; ***p<.001
3.27
3.38
Sig.
*
Effect
size
(d)
.17
Question 1: Results (cont.)
Means comparisons for satisfaction with aspects
of time at institution (4-point scale after
removing “Not Relevant” option) suggests that
alumni give lower specific appraisals for certain
aspects
Academic advising
Student Alumni
Mean
Mean
3.00
2.79
Sig.
**
Effect
size (d)
.22
Career advising
2.79
2.40
***
.39
Opp. for internships
2.68
2.36
***
.31
*p<.05; **p<.01; ***p<.001
Question 1: Results (cont.)
Means comparisons for amount of
institutional contribution to acquired skills
and competencies (4-point scale from “Not at
all” to “Very much”) show a similar pattern,
with alumni giving lower specific appraisals
for certain skills
Student Alumni Sig.
Mean
Mean
Effect
size (d)
Research skills
3.29
3.10
**
.24
Clear writing
3.21
2.95
***
.33
Persuasive speaking
2.96
2.79
*
.20
Project management
3.23
3.03
**
.23
Technological skills
3.25
3.08
*
.21
Financial & business
2.24
1.92
***
.38
Entrepreneurial
2.23
2.00
**
.26
Networking
3.07
2.84
**
.27
*p<.05; **p<.01; ***p<.001
Question 2: Methodology
 Participants from 37 institutions that
participated in NSSE12 and received the
experimental items derived from the
SNAAP survey
 Senior NSSE respondents from all
majors* (n = 23,726)
*Excluding “other” and “undecided” categories
Question 2: Respondents
Characteristics
Female
64%
U.S. citizen
95%
First Generation status
40%
Race/Ethnicity
White
70%
Black
6%
Latino/Hispanic
8%
Asian
5%
Question 2: Respondents’ Majors
Primary Major Field
Arts
8%
Humanities
9%
Biological Sciences
12%
Business
18%
Education
8%
Engineering
10%
Physical Sciences
4%
Professional (other)
14%
Social Sciences
16%
Question 2: Results
 Looking at frequencies for students in the
different majors reporting that their institution
contributed “Very Much” to acquiring certain
skills and competencies reveals interesting
patterns
 Some patterns are expected; others highlight
less apparent similarities among major fields
% Reporting Institution Contributed
“Very Much” to Acquiring…
Artistic Technique
100%
90%
80%
74%
70%
60%
50%
40%
30%
20%
10%
0%
21%
7%
11%
14%
7%
9%
15%
11%
% Reporting Institution Contributed
“Very Much” to Acquiring…
Financial & Business Management Skills
100%
90%
80%
65%
70%
60%
50%
40%
30%
20%
10%
0%
10%
8%
6%
10%
9%
8%
14%
13%
% Reporting Institution Contributed
“Very Much” to Acquiring…
Teaching Skills
100%
90%
77%
80%
70%
60%
50%
40%
30%
20%
10%
0%
33%
28%
17%
19%
27%
15%
26%
19%
% Reporting Institution Contributed
“Very Much” to Acquiring…
Critical Thinking
100%
90%
80%
62%
70%
60%
50%
40%
30%
20%
10%
0%
49%
51%
55%
52%
57%
58%
58%
61%
% Reporting Institution Contributed
“Very Much” to Acquiring…
Creative Thinking
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
62%
58%
46%
55%
59%
63%
57%
61%
57%
% Reporting Institution Contributed
“Very Much” to Acquiring…
Research Skills
100%
90%
80%
61%
70%
60%
50%
40%
30%
20%
10%
0%
40%
60%
48%
48%
47%
54%
57%
64%
% Reporting Institution Contributed
“Very Much” to Acquiring…
Technological Skills
100%
90%
80%
60%
70%
60%
50%
40%
30%
20%
10%
0%
40%
25%
28%
39%
36%
40%
39%
27%
% Reporting Institution Contributed
“Very Much” to Acquiring…
Interpersonal Relations
100%
90%
80%
70%
53%
60%
50%
40%
30%
20%
10%
0%
36%
32%
29%
47%
38%
47%
30%
37%
Discussion
Question 1
 Alumni may be viewing their institutional
experience as a whole through rose-colored glasses
when they think about “the good old days”
 Post-graduation experiences in the workplace may
better enable alumni to reflect on certain aspects of
their time
 Alumni may also learn that they needed to develop
some skills more once they have gained work
experience
Discussion (cont.)
Question 2
 The skills that students acquire while at their
institutions can vary greatly among different
majors
 Many of these differences are expected (and
increase our confidence in the survey
instrument)
 Other patterns link seemingly dissimilar
majors, such as arts and engineering, or
business and education
Limitations
 May not represent ALL students and alumni,
data only available for those participating in
both SNAAP and NSSE (and those receiving
experimental NSSE items)
 Difficulty of surveying alumni: response rates,
tracking accurate contact information
 Relies on self-reported data
Conclusions
 Important institutional information can be gained
through surveying both students and alumni
 Students may be better able to provide
information about affective components of their
experience, while alumni may be better judges of
specific things needed in the workplace
 A variety of perspectives can also be gained
through exploring differences by majors
http://3millionstories.com/
March 7-9, 2013
Nashville, TN
Indiana University
Center for Postsecondary Research
Strategic National Arts Alumni Project
[email protected]
snaap.indiana.edu
National Survey of Student Engagement
[email protected]
nsse.indiana.edu
Angie L. Miller [email protected]
Amber D. Lambert [email protected]
Becca Houghton [email protected]