Transcript Slide 1
Using the National Survey of
Student Engagement
to Assess Educational
Effectiveness at AICAD schools
AICAD Consortium
Meeting
Pratt Institute, NY
June 12, 2007
Agenda
•
Introduction & NSSE overview
•
What can you learn about your students and their
experience from NSSE?
• NSSE Reports
• Benchmarking
• Consortium comparison
• AICAD interests: marketing and institutional improvement
•
NSSE details
• Timeline and Administration
•
Questions – What questions do you have right now?
Introduction Activity:
Assessment at your institution:
What do you want to know about your
students?
Why do you want to know this?
What is the purpose of your assessment
initiative(s)?
To what extent have you used NSSE
data?
What is NSSE?
Student Engagement
The time and energy students devote to
educationally purposeful activities and the
extent to which institutions emphasize effective
practice
Engagement is a reliable predictor of student
learning and personal development
Institution can shape curriculum and resources
for learning to promote engagement
What Really Matters in College:
Student Engagement
Because individual effort and
involvement are the critical
determinants of impact,
institutions should focus on
the ways they can shape their
academic, interpersonal, and
extracurricular offerings to
encourage student
engagement.
Pascarella & Terenzini, How College Affects
Students, 2005, p. 602
Foundations of Student Engagement
Time on task (Tyler, 1930s)
Quality of effort (Pace, 1960-70s)
Student involvement (Astin, 1984)
Social, academic integration
(Tinto,1987, 1993)
Good practices in undergraduate
education (Chickering & Gamson, 1987)
Outcomes (Pascarella, 1985)
Student engagement (Kuh, 1991, 2005)
Good Practices in
Undergraduate Education
(Chickering & Gamson, 1987; Pascarella & Terenzini, 2005)
Student-faculty contact
Active learning
Prompt feedback
Time on task
High expectations
Respect for diverse learning styles
Cooperation among students
NSSE Survey & Results
Survey offers an annual snapshot of
student participation in programs and
activities that institutions provide for their
learning and personal development.
Results provide an estimate of how
undergraduates spend their time and what
they gain from attending college.
NSSE items represent empirically confirmed
‘good practices’; they reflect behaviors
associated with desired outcomes of college.
NSSE 2006 Participating Colleges &
Universities by Carnegie Classification
30%
25%
20%
National
15%
NSSE 2006
10%
5%
0%
DRU-VH DRU-H
DRU
Master-L Master- Master-S Bac-AS Bac-Div
M
Core Survey: NSSE
Research based on
effective educational
practices
Designed and tested for
high validity and
reliability
Relatively stable over
time
High credibility of selfreported data
Over 275,000 students at
600 institutions annually
NSSE Survey Item Organization
Q.1 – Academic
activities
Q.7 – Enriching
educational experiences
Q.2 – Learning mental
activities
Q.8 – Campus
relationship
Q.3 – Reading & writing
Q.9 – Time usage
Q.4 – Homework
Q.10 – Institutional
emphasis
Q.5 – Academic
challenge
Q.6 – Co-curricular
activities
Q. 11 – Gains
Q.12-14 – Satisfaction
NSSE Results
Are diagnostic; to help institutions look
holistically at undergraduate experience
Help pinpoint aspects not in line with
mission, or what institution expects
Identify weaknesses and strengths in
educational program
Help institutions know what to focus on
to improve student learning and success
Questions to answer with NSSE
results
How many hours per week do first-year
students spend studying? Do women study
more than men?
What % of seniors work with faculty members
on activities other than coursework (activities,
committees)? Does this differ by major?
What % of FY and SR spend 0 hours in cocurricular involvements? Is this more than at
peer institutions?
Do FY students work more frequently with
classmates on assignments outside of class
than their counterparts at peer institutions?
Questions to answer with NSSE
results
Do NSSE results match our mission and what we say
about a [INSTITUTION] experience?
Are we meeting our own expectations for having a
supportive campus environment?
Since implementing a new multicultural education
initiative and expanding diversity programming, has
our score on the diversity scale changed?
Are FY who withdraw from the institution different in
terms of engagement than students who are retained?
How are we performing compared to our select peers
(normative benchmarking) or to our institutionally
identified standards (criterion benchmarking)?
NSSE Deliverables
NSSE Institute Information
Institutional Report (August)
Using NSSE Data
Comparison Reports
Accreditation Toolkit
Respondent characteristics
(Demographic Information)
Data Facilitator’s Guide
Means and Frequencies
(item averages and response
percentages)
Benchmarks of Effective
Educational Practice
Additional Reports
(If Applicable)
FSSE Report
BCSSE Combined Report
Data file (student-identified)
Sample NSSE results: Frequency
comparisons
Frequency Comparisons: About how many hours do you
spend in a typical 7-day week participating in co-curricular
activities (organizations, campus publications, student
government, fraternity or sorority, intercollegiate or
intramural sports, etc.)
(1=0 hrs/wk, 2=1-5 hrs/wk, 3=6-10 hrs/wk, 4=11-15 hrs/wk, 5=16-20 hrs/wk, 6=21-25 hrs/wk, 7=26-30
hrs/wk, 8=more than 30 hrs/wk
NSSEville State Selected Peers Carnegie Peers NSSE 2006
COCU 0 hr/wk 190 61% 2539 43% 3,440 52% # 43%
RR01
NSSEville State Selected Peers Carnegie Peers
170 56% 2558 46% 3,341 52%
NSSE 2006
50,704
47%
0 Hours on co-curricular activities = 61% FY vs. 56%
seniors compared to 43% and 46% at Select Peer
Institutions – is this what NSSEville expects??
Sample NSSE results: Mean
comparisons
NSSEville State compared with:
Benchmark
1.
Class
Academic and Intellectual Experiences
h.
Worked with classmates outside of class to
prepare class assignments
Worked with faculty members on activities other
s. than coursework (committees, orientation, student
life activities, etc.)
ACL
SFI
NSSEville State Selected Peers
Carnegie Peers
Effect
Effect
a
a
b
c
a
b
c
NSSE 2006
Effect
a
b
Mean
Mean
Sig
Sizecurrent school
Mean year,Sigabout how
Size often have
Meanyou done
Sig each Size
In your experience
at your institution
during the
of
the following? 1=never, 2=sometimes, 3=often, 4=very often
c
FY
2.11
2.34 *** -.27
2.33 *** -.24
2.36 *** -.28
SR
2.52
2.70 ** -.20
2.65
2.75 *** -.26
FY
1.47
1.55
1.59 ** -.14
1.56
SR
1.83
1.83
1.85
1.81
*
-.15
NSSEville State score on 1h. (working with peers
outside of class) is significantly LESS than
SELECT PEER institutions for FY and Seniors
Benchmark Report
Level of
Academic
Challenge
Student
Faculty
Interaction
Enriching
Educational
Experiences
Active &
Collaborative
Learning
Supportive
Campus
Environment
Sample NSSE Results:
Benchmark Report
NSSEville State Strength –
significantly higher score
for FY and SR on Supportive
Campus Environment
Supportive Campus Environment (SCE)
Benchmark Comparisons
NSSEville State compared with:
NSSEville State
Selected Peers
Carnegie Peers
Effect
Class
First-Year
Senior
Mean
100
First-Year
Sig
Size
57.8
**
.17
55.8
**
.21
a
Mean
60.9
59.7
a
b
c
NSSE 2006
Effect
Mean
a
Sig
b
Size
c
59.4
59.4
75
57.8
59.4
59.1
S e le c te d P e e rs
C a rne gie P e e rs
NS S E 2006
50
25
0
NS S Eville S ta te
Mean
59.1
56.6
First Year Students
60.9
Effect
a
Sig
*
b
Size
c
.16
AICAD Consortium Questions
•Value in
developing
consortium
specific
questions
•Comparison
options
•Potential
Data sharing
•Establish
Core
questions &
others that
rotate in
NSSE Use
“The NSSE data is one among several pieces of
information that is used to organize
discussions about enrollment management,
curricula, retention, and faculty
development.”
—Christopher Cyphers,
Provost, School of Visual Arts
Using NSSE Data
Benchmarking –
longitudinal, criterion,
normative
Context Setting – paint a
picture of the institution
Problem Identificationresults point to things
institutions can do
something about –
almost immediately
Evidence of outcomes &
processes
Mobilize Action - to
change/improve
Provides lexicon for talking
about collegiate quality in
an understandable,
meaningful way
Helps inform decisionmaking
Refocus conversation
about collegiate quality
Making Sense of Data:
Benchmarking
Three Approaches:
Normative - compares your students’ responses
to those of students at other colleges and
universities.
Criterion - compares your school’s performance
against a predetermined value or level
appropriate for your students, given your
institutional mission, size, curricular offerings,
funding, etc.
Longitudinal – year to year comparison of your
students to assess improvement
Benchmarking within Example: 1
AICAD consortium - Writing
Assessment Issue: Insure high quality writing experiences in
the first year. Are we using writing center/tutors effectively?
Relevant NSSE items: 1c, d; 3c,d,e; 11c. Provide student
learning process & outcome indicators
NSSE results: First year students write short papers
comparable to AICAD schools; but fewer med & long papers,
are less likely to prepare 2+ drafts & also report lower gain in
writing than AICAD peer schools.
Interpretation: Benchmarking with AICAD schools indicates
institution is underperforming; what other data might you
gather to assess writing? What first year writing initiatives
might help? What goals might you set for improvement?
Example: 2
Benchmarking – longitudinal
(performance indicators & co-curricular improvements)
Assessment Issue: Maintaining effectiveness and making
targeted improvements with upper division students
Relevant NSSE items: NSSE items identified as key performance
indicators, gains items for seniors (11 a-p); and targeted
improvements in co-curricular experiences (1h,s,t; 6a; 7b,
10f & diversity scale, 1e,u,v; 10c)
NSSE results: Baseline NSSE = 2006, monitor indicators in
2008; assess impact of co-curricular enhancements &
diversity initiatives started in 2006 by comparing NSSE 2006
SR to 2008 SR scores.
Interpretation: Longitudinal benchmarking (2006-2008); could
also benchmark with AICAD schools. Did you meet
performance goals? Did the enhancements have an impact?
Multi year comparison
Using NSSE to “market”
AICAD schools
Demonstrate AICAD consortium and
institutional strengths (items, NSSE
benchmarks) in undergrad program
Use results to show mission
effectiveness
i.e., gains items (11 a-p) & comparison
peers show liberal education gains;
use consortium results to focus on arts
school mission
Provide results to prospective
students and families
Share results with current students,
development office and alumni
Institutional Example: NSSE and
Enrollment Management
The enrollment management area at Meredith has
used NSSE results to help guide the enrollment
marketing strategies. They look closely at trends
and make adjustments to programs and campus
visitation days to ensure that students are more
cognizant of student involvement and engagement
opportunities.
An academic dean reports using NSSE information
when speaking to parents at an admissions event.
"Parents seemed impressed that there was
data to support the points that I was
making about what we say about the
student/faculty relationships and
educational opportunities at Meredith."
Institutional Example:
Hanover College
A detailed summary of NSSE is sent to the
faculty as well as the Admission and
Student Life staffs to ensure the results,
both good and bad, are understood by key
folks on campus. Last year, Admission
requested an additional presentation and
discussion of findings to help them better
understand the strengths of the Hanover
experience and how that impacts student
fit.
Helping Students and Families Focus
on What Matters to Success
Pocket Guide helps prospective
students ask the “right” questions
Good questions to ask of all
schools, not just those that
participate in NSSE
School counselors can request
up to 1000 free pocket guides per
year. Colleges and non-profit
education organizations can
request up to 300 copies free per
year.*
*Request via the NSSE Web site
“A Pocket Guide to Choosing a College:
Are You Asking the Right Questions…”
Connecting NSSE Data to
Accreditation Standards - Example
Accreditation standard: “Demonstrate effectiveness of
student academic and social support services”
Evidence for institutional self study:
Information about availability and student use of tutoring, writing
support, peer study groups, counseling services
NSSE indicates FY & SR believe institution emphasizes spending
time studying and support for student success; 79% seniors
tutored or taught peers; positive correlation between peer
collaboration outside of class, satisfaction and first-year retention
Positive student satisfaction data about support services
Area for improvement - seniors indicate low gains in writing and
completing drafts of papers; institution responds with examination
of writing requirement in senior capstone and targets seniors for
increased use of writing center
NSSE and AICAD
consortium
Consider data sharing agreements
Potential for additional comparison studies; prepare
papers/presentations; examine shared concerns
(retention, outcomes)
Use consortium to explore common concerns
Coordinate survey schedule
Ideas to improve participation rate (incentives,
persuasive to promote that survey is occurring at
other AICAD schools??)
Identify focus for additional questions (up to 20!)
Develop stable core of questions? or change focus? Or a
mix? Rotate new questions in?
Administration Details
What challenges
have you faced in
your NSSE
administrations?
What concerns do
you have about
your next
administration?
Questions about the
details?
NSSE Timelines
May
NSSE/FSSE registration opens
September
NSSE/FSSE registration deadline
NSSE materials due two weeks after
registration confirmation
October
NSSE pop. files, oversample, and
consortium decisions due
December
FSSE materials and pop. files due
Mid-January early February
NSSE administrations open
BCSSE registration begins
(15 mos.)
Mid-March early April
FSSE administration opens
June
NSSE & FSSE administrations close
BCSSE administration begins at many
campuses
August
Institutional Reports sent, including
raw data and printed reports for NSSE,
FSSE, and the prior summer’s BCSSE
BCSSE administration continues
September
BCSSE data and reports sent to
participating institutions
NSSE Administration
Administration Mode
Paper: We need
accurate mailing
addresses, letterhead,
signatures
Web+: 4x the paper
sample, we need e-mail
and mailing addresses
Web: 5x the paper
sample, we need e-mail
addresses
NSSE Administration
Sample Size
Numbers are based on mode and school size
Oversampling can increase sample size or ensure
adequate representation of populations of interest
NSSE Administration
Things that we need from you
Contact persons
Campus Project Manager (required)
Campus Administrative Contact (required)
Auxiliary Contact (optional)
Population File
All First-Year and Senior Students
Accurate mailing and/or e-mail addresses
Institutional letterhead and signature file (Paper
mode only)
NSSE Administration
Things for you to consider
Broad buy-in from others at your institution
(informal word-of mouth)
Web-mode institutions:
Good partnership with
IT department
Consortium
NSSE Administration
Things you to consider (cont.)
Administration Plan
•
•
•
•
•
Follow the IRB rules of Indiana University
Bloomington
Allowed up to 5 institutional contacts
Promotion plan
Incentive programs
Tips to boost response rates
http://nsse.iub.edu/html/tips.cfm
NSSE: Only one step in
assessment process
Step #1: Survey Data
Step #4: Follow-up
• Use results as
benchmarks to
monitor progress
• Faculty & student
focus groups
• Survey students
• Review results
• Develop preliminary
list of strengths and
opportunities for
improvement
Step #3: Action Plan
• Finalize plan
• Share plan with
appropriate groups
• Link to strategic plan
• Implement action
Step #2: Feedback
• Share results with
faculty, administrators
& students
• Identify themes &
priorities
• Design action plan
NSSE in your assessment plan
How often should I administer NSSE?
Every Year: Gives you a snapshot of each class
Every Three Years: Gives you a picture of a cohort at the
beginning and the end of their college experiences
Every Five Years: Works well with most accreditation cycles
(Accreditation and Interim Reports)
Other factors to consider
Establishing a baseline
Costs (using all core surveys)
Additional Surveys/Sources of Data
Time to take absorb results, make changes
Updates for 2007 and 2008
No changes to survey content
Select up to three customized comparison
groups on your reports
Electronic report
delivery
Executive Summary
Report
Pocket Guide Report
45
Institutional Examples: Worcester
Polytechnic Institute
NSSE results showed FY students
were less engaged than seniors
New FY interdisciplinary, inquirybased seminars; better integration
of disciplines; engaging
introductory courses
Associate Dean appointed to Office
for the First Year
Assessment plan in development
with NSSE indicators as key
component
Institutional Example: NSSE &
Assessing General Education goals
Used NSSE items in 11a-p to assess
institutional impact on college-level
competencies (a.k.a., indirect measures of
student learning outcomes)
Undergraduate seniors 2005 NSSE
results confirmed findings from 2004
Most seniors (75%+) reported that KSU
experience had “substantial impact”
(VM+QAB) in 9 or 16 college-level
competencies
KSU rank ordered competencies, showing
connection to mission, and compared to
other master’s instit where KSU was sig.
higher, comparable, sig. lower on
competencies
Institutional Example: Program
Development and Strategic Planning
NSSE results framed a “Sophomore Experience”
2005 = Pace’s 5th year of participation
Concern regarding SP- JR persistence; FY results offers
context for understanding exp. as students enter SP year
Established “SP Experience Working Group” to investigate if
FY exp. carried over in SP year. Focused on low NSSE
score items, conducted focus groups, created sophomore
survey. Led to pilot of “Pace Plan” (mentoring), includes
Career Exploration Course, Sophomore Kick-Off Day
NSSE also used in strategic indicators, Accred, NCATE,
AACSB, Faculty Development/Colloquia, items used by
offices (Technology, Multicultural Affairs), studies
performed by Enrollment Mngmt.
NSSE suite
The “NSSElings”
The Faculty Survey of Student Engagement (2003)
The Beginning College Survey of Student Engagement
(2004)
Additional Surveys
The Law Student Survey of Student Engagement
The College Student Experiences Questionnaire
The College Student Expectations Questionnaire
The High School Survey of Student Engagement*
The Community College Survey of Student Engagement*
*Not administered by the Center for Postsecondary Research
FSSE
Faculty perceptions of how often their students
engage in different activities
Importance faculty place on various areas of
learning and development
Nature and frequency of interactions faculty have
with students
How faculty members organize class time
FSSE Instrument: Survey
Options
Course-based (default)
Responds to questions based on one particular
undergraduate course section during taught
during the current academic year
Typical-student
Responds to questions based on the typical firstyear student or senior taught during the current
academic year
New!
Beginning College
Survey of Student Engagement – BCSSE
Designed for entering firstyear students as a
companion to NSSE
Measures:
pre-college academic and
co-curricular experiences
expectations for
educationally purposeful
activities during college
BCSSE Instrument
Three pilots: ‘04, ‘05, and ‘06
2004 pilot with 28 schools,
15,890 students
2005-2006 pilots with 80
institutions, 39,986 students
Study effect of students’
background on NSSE scores
Use to examine gap between
expectations and engagement
Registration now open
Creative Campus initiative and
Experimental questions developed by Arts
Consortium in NSSE 2007
1 )In your experience at your institution during
the current school year, about how often have
you done each of the following? (Very Often,
Often, Sometimes, Never)
a) Attended an art exhibit, play, dance, music,
theater, or other performance
b) Talked about an art exhibit, play, dance,
music, theater or other performance with
other students, friends, or family
c) Participated as an artist, performer, or crew
member in an art exhibit, play, dance, music,
theater, or other performance
d) Used your experiences and interest in the
visual and performing arts in class discussions
or assignments
e) Explored a new subject area as a result of
your attendance or participation in the arts.
2) To what extent has your experience at this
institution contributed to your knowledge,
skills, and personal development in the
following areas? (Very much, Quite a bit,
Some, Very little)
a) Developing an understanding and
enjoyment of an art exhibit, play, dance,
music, theater, or other performance
b) Developing a commitment to be involved in
3) In a typical 7-day week, about how
many hours do you spend in arts
experiences on or off campus? [0, 15, 6-10, 11-15, 16-20, 21-25, 26-30,
more than 30]
4) How many arts courses (art, music,
theater, dance) have you taken since
coming to college? [0, 1, 2, 3, 4 or
more]
5) What motivates you to attend arts
events (art, music, theatre, dance)?
(Select all that apply)
a) Class requirement
b) Encouragement from friends
c) Enthusiastic professor
d) Personal interest
e) Proximity
f) Quality of the facility
g) Opportunity to meet people
h) To be involved on campus
i) Promotion of the event on campus
j) Quality of the event
Creative Campus initiative and
Experimental questions developed by Arts
Consortium in NSSE 2007
6) What keeps you from attending arts events
(art, music, theatre, dance)? (Select all that
apply):
a) Expense
b) Parking availability
c) Lack of awareness of events
d) Difficulty getting tickets
e) Coursework demands
f) Facility is rundown
g) Location is not convenient
h) Job demands
i) No one to go with
j) Extracurricular activities/meetings
k) Lack of quality events
l) Limited interest in arts events
7) Select the three (3) places you perceive to be
the most active spaces on campus
a) sports venues (football stadium, basketball
arena)
b) public plaza (main campus quadrangle or
green)
c) arts venues (performing arts center,
museums/galleries, rehearsal halls, art
studios)
d) student union
e) dining hall
8) Where have you attended arts events (art,
music, theatre, dance) while attending this
institution? Select all that apply.
a) Theater buildings (performing arts center,
recital hall)
b) Museums/galleries
c) Coffee shop or restaurant
d) Large concert venues (stadiums, arenas,
band shell, etc)
e) Random places, such as street
performances
f)\ Residence halls
g) Arts festivals
h) Off-campus; downtown or in the local
community
9) To what extent have arts experiences at this
institution contributed to your abilities in the
following areas? [very much, quite a bit,
some, very little]
a) Thinking critically and analytically
b) Thinking imaginatively or creatively
c) Understanding fundamental concepts in
my major
d) Communicating clearly and effectively
e) Working effectively with others
f) Learning effectively on your own
g) Taking intellectual risks
Discussion and Questions
Jillian Kinzie
Associate Director, NSSE
Institute
Indiana University Center for
Postsecondary Research
1900 East 10th Street
Eigenmann Hall, Suite 419
Bloomington, IN 47406-7512
Ph: 812-856-5824
Fax: 812-856-5150
Web site: www.nsse.iub.edu
E-mail: [email protected]