Transcript NSSE Ohio Drive-In

Using National Studies
of Student Engagement
to Support Institutional
Change
Nathan Marti, CCSSE
Todd Chamberlain, NSSE
FAIR Conference
June 23, 2004
What Really Matters in College:
Student Engagement
“The research is unequivocal:
students who are actively
involved in both academic and
out-of-class activities gain more
from the college experience
than those who are not so
involved.”
Ernest T. Pascarella & Patrick T.
Terenzini, How College Affects Students
Lessons from the Research
 What
matters most to desired
outcomes is what students do, not
who they are
A
key factor for student learning is
the quality of effort students
devote to educationally purposeful
activities
National Survey of
Student Engagement
(pronounced “nessie”)
Community College
Survey of Student
Engagement
(pronounced “sessie”)
College student surveys that
assess the extent to which
students engage in educational
practices associated with high
levels of learning and
development
Penetrating NSSE/CCSSE
Findings
 Overview of Reports
 Start Broad, then Dig Deeper
 Identify Significant Items
 Focus on What is Important to
YOUR Institution’s Priorities
Customized Institutional Report
Respondent
Characteristics
National
benchmarks
Institutional data
Means and
frequencies
Subpopulations
Comparisons by
Carnegie/size,
national
Respondent
Characteristics
Respondent Characteristics:
Does This Represent Your Campus?
Response Rate
Sampling Error
Student Characteristics




Gender
Race/Ethnicity
Class Level
Enrollment Status
Comparisons by Carnegie/size,
national and consortium when
relevant
Benchmarks: High Level
Views of the Data
Supportive
Campus
Environment/
Support for
Learners
Enriching
Educational
Experiences/
Student Effort
Level of
Academic Challenge
StudentFaculty Interaction
Active &
Collaborative
Learning
NSSE/CCSSE developed five indicators, or benchmarks, to
represent the multi-dimensional nature of student engagement
at the institutional, sector and national levels
NSSE 2001 Institutional Benchmark Report
Nesseville State University
The NSSE survey, The College Student Report, measures student engagement in many important activities that research
studies show are positively related to learning and personal development. Forty-one questions from the survey are assigned to five
clusters of similar activities and conditions to make up the national benchmarks of effective educational practice. The benchmarks
are created on 100-point scales to make it easier to compare performance within and across sectors and institutional types.
These benchmarks are: (1) level of academic challenge, (2) active and collaborative learning, (3) student interactions with
faculty members, (4) enriching educational experiences, and (5) supportive campus environment. The NSSE benchmark analysis is
based on more than 105,000 randomly selected students at 467 four-year colleges and universities that participated in the spring of
2000 or 2001. The students represent a broad cross-section of first-year and senior students from every region of the country. The
institutions are similar in most respects to the universe of four-year schools. More detailed information about the benchmarks can
be found in the national report that was sent with this mailing and on the NSSE website at www.iub.edu/~nsse.
Benchmark
Report
This report summarizes your institution’s performance in these five areas of effective educational practice. Your institution’s
benchmark scores are presented and compared to schools in your consortium, your Carnegie Classification, and the NSSE national
norms. Page 4 provides some additional information, including a standard score that represents the magnitude of the difference
between your institution's score and the respective comparison group, and page 5 presents a table of National and Carnegie
classification deciles against which you can gauge the relative performance of your institution on each of the benchmarks.
Level of Academic Challenge
Level of Academic Challenge Items:
Nesseville State
AAUDE
Doc/Res-Extensive
National
80
70
Preparing for class (studying, reading, writing,
rehearsing, and other activities related to your
academic program)
Number of assigned textbooks, books, or book-length
packs of course readings
Number of written papers or reports of 20 pages or
more
60
Number of written papers or reports of between 5 and
19 pages
50
Number of written papers or reports of fewer than 5
pages
40
Coursework emphasizes: Analyzing the basic elements
of an idea, experience or theory
Coursework emphasizes: Synthesizing and organizing
ideas, information, or experiences into new, more
complex interpretations and relationships
30
Coursework emphasizes: Making judgments about the
value of information, arguments, or methods
20
First-Year
Senior
Coursework emphasizes: Applying theories or
concepts to practical problems or in new situations
Benchmark Scores
Institution
First-Year
Senior
Consortium
Nesseville State
AAUDE
57.3
57.3
54.9
56.2
Carnegie Classification
Doc/ResExtensive
51.4
54.6
All NSSE Institutions
National
52.9
56.6
Worked harder than you thought you could to meet an
instructor's standards or expectations
Campus environment emphasizes spending significant
amounts of time studying and on academic work
Mean
Summary
Report
Frequency
Distribution
Report
Means & Frequency Reports
Look at the Items that Make Up
Each Benchmark
Which Items Have Significantly
Higher/Lower Responses than
Comparison Groups?
Practical Significance: Identify
Standardized Effect Sizes
greater than .2
Activity: Highlight Key Findings
Review Benchmark and Means
Summary Report
Small groups by benchmark
Examine a benchmark
Which items differ?
Digging Deeper: Using the Data
 Identify
important subpopulations
 Determine the outcome that matters
 Determine the factors that influence the
outcome
Principles for Data-Driven
Learning-Centered Change
1. Get the ideas right
 Focus on a real problem
(e.g., persistence, raising
expectations, success in
major field courses)
 Concentrate on effective
educational practices
Characteristics of
Educationally Effective
Colleges
Organizational culture valuing
 High expectations
 Respect for diverse talents
 Emphasis on early years of
study
Characteristics of
Educationally Effective
Colleges
Curriculum
 Coherence in learning
 Synthesizing experiences
 Integrating education and
experience
 Ongoing practice of learned
skills
Characteristics of
Educationally Effective
Colleges
Instruction
 Active learning
 Assessment and feedback
 Collaboration
 Adequate time on task
 Out-of-class contact with
faculty
2. Get grass roots buy-in
 Leaders endorse, but don’t
dictate
 Structures not (nearly) as
important as relationships
 Validate pockets of quality
 The 10% rule
2. Get grass roots buy-in
Examples
 Ask deans about their concerns
 Focus groups
 Get students “engaged” in the
improvement effort (Illinois State,
Oregon State)
 Faculty version of NSSE survey
3. Keep the stakes and
volume low
 Avoid “winners” & “losers”
 Suspend disbelief
 Denial management
 Go public later than sooner
4. It’s the culture (stupid)
 Culture is (almost) always
(at least) part of the
problem
 Focus on “reculturing” and
“revisioning”
 Use familiar (or at least
understandable) language
5. Think and act systemically
 Link innovations and change
efforts from different parts
of the campus (e.g., Greater
Expectations, Gen Ed reform,
SOTL, NSSE, service
learning, diversity)
Presenting NSSE/CCSSE
Findings
 Potential Audiences?
 Consider Audiences Before
Administration—It May
Influence Decisions
 Internal Use & External
Reporting
 Be Strategic
Internal Audiences
Internal Sharing of NSSE 2003 Results
%
President
80
Administrative Staff
68
Department Chairs
59
Academic Advisors
51
Faculty
71
Governing Board
34
Students
32
Other (web site, fact book, etc.)
20
Source: NSSE 2003 Report Card
External Audiences
External Sharing of NSSE 2003 Data
%
Accreditation Agencies
31
State Agencies
11
Media
13
Web Site
25
Prospective Students
13
Parents
13
Alumni
13
Other
No External Disclosure
Source: NSSE 2003 Report Card
7
35
Ways to Communicate Results






Customize Report to Reach Campus Audiences
Organize NSSE data by colleges, depts,
programs
Focus on items of interest to stakeholder and
communicate understandable percentages
Use data to start discussions
Use data to correct misunderstandings
Assemble and distribute detailed information on
method and credibility of survey
Activity: Connecting Findings
with Relevant Audiences
What was the most interesting
finding?
Who needs to know about this?
How can you let them know?
National Context
Putting Your Results in Context:
 How
good is good enough?
 What would it mean to have 80% of
your students indicate that they are
satisfied with the college?
Sampling Methods
Sampling Consistency Across
Colleges:
 CCSSE
takes a stratified random
sample of courses
 NSSE takes random samples of firstand fourth-year students
 Results are representative of the
populations from which they are drawn
 Over-sampling for targeted populations
Comparing Your Results
Sampling Consistency Across Colleges:
 Because
results are comparable across
institutions, institutions can situate their
results
 Anything on the survey can potentially be
benchmarked
Benchmarking
Two Approaches:
 Normative - compares your students’
responses to those of students at other
colleges and universities.
 Criterion - compares your school’s
performance against a predetermined
value or level appropriate for your
students, given your institutional
mission, size, curricular offerings,
funding, and so forth.
Criterion Referenced Approach

Most valued activities
Ask faculty & staff what is most
valued in institution, department,
then present relevant data

Eliminate “Nevers”
Work on reducing or eliminating
reports by students of never doing
specific engagement activities
Assessment Purposes


Accountability
Improvement
Activity: Identify Appropriate
Benchmarks
Internal Differences: How do
part- and full-time students
differ?
External Examples: How does
your institution differ from similar
colleges?
Principles for Good Practice in
Undergraduate Education
(Chickering & Gamson, 1987)
 Student-faculty contact
 Active learning
 Prompt feedback
 Time on task
 High expectations
 Respect for diverse learning styles
 Cooperation among students
Lessons from the Research
 What
matters most is what
students do, not who they are
A
key factor is the quality of
effort students expend
 Educationally
effective
institutions channel student
energy toward the right activities
Principles For Promoting
Student Engagement

Intentionality - Not leaving
serendipity to chance

Alignment (mission,
curriculum, student
experiences)

Collaboration (pull many
levers)

Assessment and feedback
to guide/document
improvement
How is Engagement
Measured?
 Switch
benchmarks
 How would you measure this
construct?
Psychometric Findings: Reliability
 Both
instruments have been shown to
be reliable and valid
 Good construct reliability: benchmarks
items measure the same construct
 Structural equation models have been
used to demonstrate that there is
strong consistency across
subpopulations (gender & part- v. fulltime) and across years
Psychometric Findings: Validity
Psychographic Data
 NSSE
data used to form clusters of
students
 http://mypage.iu.edu/~chuzhao/air03dpi.
pdf
 Psychographic data consistently was
more effective at predicting outcomes
than demographic data
The Florida Opportunity
 How
does engagement relate to
outcome?
 Persistence
 Goal Attainment
Activity: Expectations and
Desires in Student
Engagement
 How
many students responded often or
very often?
 What would you like to see?
 How did they respond?
 How can you increase engagement?
Questions and Suggestions
 Questions
for NSSE and CCSSE
 Suggestions for NSSE and CCSSE