Transcript tmplate
Session for NSSE “Veterans”
Regional NSSE User’s Workshop October 2005
Shimon Sarraf, Research Analyst Center for Postsecondary Research, Indiana University Bloomington
Overview
A Quick Survey Reports Reviewed Benchmarks Explained 2005 NSSE Dataset Details Future NSSE Developments Discussion & Questions
“NESSIE”
Goals
Two-way conversation Share your experiences Good preparation for Shimon’s second and third session
A Quick Survey to Get Things Started….
Years of Participation in NSSE Background of attendees
The NSSE Reports: Respondent Characteristics
A quick snapshot of your institution Data quality: Confirming if sample is representative Response Rate and Sample Error
2005 NSSE Response Rates by Carnegie Classification
30 20 10 0 100 90 80 70 60 50 40 33 17 58
Doc-Extensive
32 16 53 36 7 77 39 72 49 14
Doc-Intensive Masters
Mean Min Max 3
Bac-Gen Bac-LA
79
The NSSE Reports: Respondent Characteristics
What is
Sampling Error
?
Assumes random sampling An estimate of the margin likely to contain your "true" score, for example: If 60% of your students reply "very often" and the sampling error is ± 5%, it is likely that the true value is between 55% and 65%.
More respondents --> smaller sampling error
2005 NSSE Sampling Error by Carnegie Classification
30 25 20 15 10 9 5 0 4 2
Doc-Extensive
8 4 2
Doc-Intensive
14 5 2
Masters
Mean Min Max 6 2
Bac-Gen
26 5 2
Bac-LA
11
The NSSE Reports: Means Comparison
Means, statistical significance and effect sizes
1.
Academic and Intellectual Experiences
a.
Asked questions in class or contributed to class discussions b.
Made a class presentation c.
d.
e.
f.
Prepared two or more drafts of a paper or assignment before turning it in Worked on a paper or project that required integrating ideas or information from various sources Included diverse perspectives (different races, religions, genders, political beliefs, etc.) in class discussions or writing assignments Come to class without completing readings or assignments g.
Worked with other students on projects
during class
h.
Worked with classmates
outside of class
to prepare class assignments i.
j.
Put together ideas or concepts from different courses when completing assignments or during class discussions Tutored or taught other students (paid or voluntary) k.
Participated in a community-based project (e.g. service learning) as part of a regular course
Variable
CLQUEST CLPRESEN REWROPAP INTEGRAT DIVCLASS CLUNPREP CLASSGRP OCCGRP INTIDEAS TUTOR COMMPROJ
NSSE 2005 Means Comparison Report NSSEville State University
Bench mark
ACL ACL
Class
NSSEville compared with:
NSSEville Selected Peers
Effect Mean Sig a Size b Mean
Master's
Sig a Effect Size b Mean
NSSE 2005
Effect Sig a Size b Mean In your experience at your institution during the current school year, about how often have you done each of the following? 1=never, 2=sometimes, 3=often, 4=very often
FY SR FY SR FY SR 2.71
3.02
2.24
2.66
2.45
2.32
2.55
2.82
1.95
2.57
2.48
2.32
*** *** *** * .19
.22
.40
.11
2.84
3.15
2.30
2.93
2.69
2.55
*** *** ** *** *** *** -.15
-.16
-.07
-.32
-.25
-.24
2.86
3.16
2.28
2.88
2.65
2.51
*** *** *** *** *** -.18
-.18
-.26
-.21
-.20
FY SR ACL ACL FY SR FY SR FY SR FY SR ACL ACL FY SR FY SR FY SR 2.98
3.31
2.60
2.89
1.72
1.86
1.39
1.63
2.75
2.74
2.14
2.31
2.36
2.43
2.41
2.82
2.86
3.19
*** *** .15
.15
2.65
2.64
2.19
2.28
2.30
2.37
2.34
2.80
*** * * * .11
.11
.07
.08
2.49
2.88
1.72
1.87
1.40
1.54
*** * .14
.11
3.08
3.36
*** -.13
2.76
2.83
2.01
2.05
2.42
2.58
2.39
2.76
** *** *** ** *** -.11
.17
.35
-.08
-.16
2.54
2.91
1.68
1.90
1.56
1.80
* .08
*** *** -.21
-.19
3.08
3.37
*** * -.12
-.08
2.77
2.83
2.03
2.08
2.40
2.52
2.43
2.77
** *** *** -.11
.15
.31
* -.09
2.57
2.93
1.72
1.94
1.54
1.77
* *** *** -.08
-.19
-.15
The NSSE Reports: Means Comparison
What is
Statistical Significance
?
Helps you answer the question, “How likely is it that the difference between my average student and the average student at [comparison group] is due to chance?
Significance determined by standard alpha values of p<.05, .01, or .001
The NSSE Reports: Means Comparison
Potential problem: As N becomes large, almost everything becomes statistically significant How do we identify truly significant differences?
This is a question of …
practical significance
The NSSE Reports: Means Comparison
What is
Effect Size
?
Practical significance
of the mean difference ES=mean difference/standard deviation .2 is often considered small, .5 moderate, and .8 large (but rare!) For example, while the difference in the means is statistically significant, the difference is so nominal that it doesn’t warrant further attention
Mean, N, SEM, SD, p value, effect size
The NSSE Reports: Detailed Statistics
Mean Standard Error of the Mean a Standard deviation b NSSE 2005 Detailed Statistics NSSEville State University First-Year Students Number of respondents Significance c
NSSEville compared with:
Effect size d
NSSEville compared with:
CLQUEST CLPRESEN REWROPAP INTEGRAT DIVCLASS CLUNPREP CLASSGRP OCCGRP INTIDEAS TUTOR COMMPROJ ITACADEM EMAIL FACGRADE FACPLANS FACIDEAS FACFEED WORKHARD FACOTHER OOCIDEAS DIVRSTUD DIFFSTU2 MEMORIZE ANALYZE SYNTHESZ EVALUATE APPLYING READASGN 2.71
2.24
2.45
2.98
2.75
2.14
2.36
2.41
2.60
1.72
1.39
2.73
3.16
2.54
2.25
1.78
2.69
2.55
1.48
2.62
2.52
2.76
2.96
3.12
2.82
2.78
2.99
3.47
2.86
2.28
2.65
3.08
2.77
2.03
2.40
2.43
2.57
1.72
1.54
2.61
3.06
2.62
2.16
1.86
2.76
2.63
1.63
2.73
2.60
2.77
2.85
3.09
2.87
2.84
2.99
3.31
2.55
1.95
2.48
2.86
2.65
2.19
2.30
2.34
2.49
1.72
1.40
2.72
2.99
2.43
1.99
1.70
2.60
2.46
1.44
2.68
2.71
2.85
2.88
3.05
2.81
2.74
2.95
3.32
2.84
2.30
2.69
3.08
2.76
2.01
2.42
2.39
2.54
1.68
1.56
2.57
3.01
2.62
2.18
1.82
2.73
2.64
1.62
2.69
2.52
2.70
2.90
3.04
2.81
2.82
2.96
3.23
.02
.02
.03
.02
.02
.02
.02
.02
.02
.02
.02
.03
.02
.02
.02
.02
.02
.02
.02
.03
.02
.02
.02
.02
.02
.03
.03
.02
.02
.01
.02
.02
.02
.01
.02
.02
.01
.02
.01
.02
.02
.02
.02
.01
.02
.02
.02
.02
.02
.01
.02
.01
.02
.02
.02
.02
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.01
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.00
.82
.77
.97
.79
.85
.76
.81
.81
.79
.82
.73
1.01
.80
.88
.85
.74
.80
.83
.81
.97
.82
.75
.83
.76
.85
1.03
.96
.83
.82
.74
1.01
.84
.89
.79
.84
.86
.79
.84
.72
1.01
.84
.87
.86
.76
.84
.86
.86
.89
.81
.78
.86
.73
.87
1.00
.94
.86
.84
.78
.97
.77
.86
.74
.81
.85
.79
.83
.82
1.05
.85
.86
.89
.79
.84
.85
.85
.96
.85
.80
.83
.84
.86
1.01
.97
.87
.84
.78
.98
.78
.87
.74
.82
.85
.80
.84
.81
1.05
.85
.86
.89
.78
.84
.85
.85
.97
.86
.80
.84
.84
.86
1.02
.97
.88
1,329 1,327 1,324 1,324 1,326 1,324 1,327 1,327 1,232 1,232 1,232 1,233 1,230 1,232 1,232 1,233 1,212 1,210 1,209 1,211 1,212 1,212 1,207 1,207 1,206 1,207 1,206 1,197 3,004 3,001 2,999 2,999 2,998 2,994 3,001 3,002 2,835 2,835 2,832 2,834 2,833 2,835 2,836 2,833 2,792 2,795 2,789 2,792 2,790 2,790 2,781 2,781 2,779 2,783 2,783 2,771 19,955 19,949 19,939 19,933 19,932 19,911 19,948 19,948 19,150 19,160 19,156 19,160 19,155 19,158 19,159 19,159 18,968 18,964 18,965 18,960 18,956 18,959 18,859 18,849 18,835 18,838 18,847 18,754 47,746 47,729 47,688 47,691 47,678 47,644 47,705 47,718 45,820 45,844 45,828 45,850 45,839 45,834 45,841 45,829 45,387 45,391 45,381 45,380 45,366 45,375 45,096 45,068 45,049 45,064 45,074 44,895 .000
.000
.371
.000
.001
.053
.031
.012
.000
.991
.573
.846
.000
.000
.000
.004
.000
.005
.150
.034
.000
.009
.013
.004
.714
.136
.198
.000
.000
.009
.000
.000
.841
.000
.006
.304
.010
.067
.000
.000
.000
.004
.009
.114
.047
.000
.000
.006
.969
.035
.016
.000
.628
.100
.172
.000
.000
.133
.000
.000
.324
.000
.071
.428
.191
.787
.000
.000
.000
.001
.001
.003
.001
.000
.000
.000
.008
.726
.000
.113
.065
.020
.873
.000
.19
.40
-.03
.15
.11
-.06
.07
.08
.14
.00
-.02
.01
.19
.13
.31
.10
.01
.05
.04
.16
.10
.12
.10
.05
-.07
-.19
-.09
.08
-.15
-.07
-.25
-.13
-.01
.17
-.08
.03
.08
.05
-.21
.15
.17
-.09
.08
.11
.01
-.05
.04
.25
-.05
-.05
-.11
-.16
-.08
.00
.06
.07
-.18
-.04
-.21
-.12
-.03
.15
-.05
-.02
.04
.01
-.19
.12
.12
-.09
.10
.05
-.05
-.07
.00
.16
-.09
-.09
-.10
-.18
-.13
-.08
-.01
.12
The NSSE Reports: Detailed Statistics
What are
Confidence Intervals
?
CI = Mean +/- 2SEM Multiplying the SEM by 2 creates a margin around the sample mean that is 95% likely to contain the true population mean.
More respondents smaller standard error of the mean (SEM), more precise estimate Higher standard deviation less precise estimate greater SEM,
Counts and percentages for each response option
The NSSE Reports: Frequency Distributions
The NSSE Reports: Frequency Distributions
Tip: Consider merging response options to create dichotomous variables (1/0)
Frequently
= often + very often
Substantial
= quite a bit + very much
The NSSE Reports: New Features
Selected Peer Group Including or Excluding your targeted oversample
5 Benchmarks of Effective Educational Practice
Level of Academic Challenge Active and Collaborative Learning Student Faculty Interaction Enriching Educational Experiences Supportive Campus Environment
Level of Academic Challenge
Challenging intellectual and creative work is central to student learning and collegiate quality. Institutions promote high levels of achievement by setting high expectations for student performance.
11 items include: Preparing for class Reading and writing Using higher-order thinking skills Institutional environment emphasizes academic work
Active and Collaborative Learning
Students learn more when they are more intensely involved in their education. Collaborating with others prepares students to handle practical, real-world problems.
7 items include: Asking questions in class Making presentations Working with other students on projects Discussing ideas from readings or classes with others
Student Interactions with Faculty
Interacting with faculty show students first-hand how to think about and solve practical problems. Teachers become role models and mentors for learning.
6 items include: Discussing assignments with a professor Talking about career plans with faculty member or advisor Getting prompt feedback on academic performance Working with a faculty member on a research project
Enriching Educational Experiences
Students need learning opportunities that complement the goals of the academic program. Provide opportunities to integrate and apply knowledge.
11 items include: Experiencing diversity Using technology Participating in internships Culminating senior experience
Supportive Campus Environment
Students perform better and are more satisfied at colleges that are committed to their success. Does institution cultivate positive working and social relationships among different groups on campus?
6 items include: Helping students achieve academically Helping students cope with non-academic responsibilities Quality of relationship between student and peers, faculty, and administrative personnel
Benchmarks of Effective Educational Practice
How are benchmark scores calculated?
1.
2.
3.
Items are converted to a 100-point scale: [(response value – 1)/(total # of response values – 1)]*100 Part-time students' scores are adjusted on four Academic Challenge items.
Student-level scores are created for each group of items by taking the mean, as long as 3/5ths of the items were answered.
4.
Institutional benchmarks are the weighted averages of the student-level scores.
Benchmark Report
Level of Academic Challenge
75 Challenging intellectual and creative work is central to student learning and collegiate quality. Colleges and universities promote high levels of student achievement by emphasizing the importance of academic effort and setting high expectations for student performance 65 55 45 Nesseville Consortium Carnegie National First-Year 52.5
52.4
51.8
53.4
Nesseville 52.5
56.3
Senior 56.3
55.6
54.9
57.0
Level of Academic Challenge Items:
Preparing for class (studying, reading, writing, rehearsing, etc. related to academic program) Number of assigned textbooks, books, or book-length packs of course readings Number of written papers or reports of 20 pages or more; number of written papers or reports of between 5 and 19 pages; and number of written papers or reports of fewer than 5 pages Coursework emphasizing analysis of the basic elements of an idea, experience or theory 52.4
National information, or experiences into new, more complex interpretations and relationships 51.8
53.4
Coursework emphasizing the making of judgments about the value of information, arguments, or methods 55.6
57.0
concepts to practical problems or in new situations Working harder than you thought you could to meet an instructor's standards or expectations Campus environment emphasizing time studying and on academic work
Benchmarks of Effective Educational Practice
Benchmark recalculation reports ( 04 & 05 ):
Driven by new calculation process that began for the 04 administration Multi-year Comparisons EEE: not comparable because of response set change SFI: comparable by removing “research” item
Benchmarks of Effective Educational Practice
Institutions can use the student-level scores to: Investigate what groups are more engaged than others on your campus.
Institutional subgroups (i.e., programs, departments) Student sub-groups (i.e., gender, race) Incorporate scale scores into predictive models of student outcomes (retention, g.p.a., satisfaction)
Measurement Scales
Satisfaction
General Satisfaction Satisfaction plus Quality of Campus Relationships
Campus Environment
Environmental Emphases Quality of Campus Relationships
Gains Factors
Personal/Social General Education Practical Competence
Deep Learning Activities
Higher-Order Learning
activities that require students to utilize higher levels of mental activity than those required for rote memorization (2b,c,d,e)
Integrative Learning
activities that require integrating acquired knowledge, skills, and competencies into a meaningful whole (1d,e,i,p,t)
Reflective Learning
activities that ask students to explore their experiences of learning to better understand how they learn
NSSE Scalelets
Course Challenge Writing Active Learning Collaborative Learning Course Interaction Out-of-Class Interaction Gains (academic, personal, vocational) Varied Experiences Information Technology Diversity Support for Student Success Interpersonal Environment
2005 NSSE Dataset Details
What do you need to know to match your Institutional Report numbers? “smpl05” (sampling type) use 1 (base sample), 2 (standard oversample), 3 (requested oversample) values… and 4 (targeted oversample) if “targetos” equals 1 “inelig” exclude all ineligible respondents use those with values of 1 for “inelig”
Future Developments
Customized Report Engine Archiving of reports and datasets online Integrating new Carnegie classification NSSE Knowledge Base
Discussion and Comments
Shimon Sarraf
Research Analyst Indiana University Center for Postsecondary Research 1900 East 10th Street Eigenmann Hall, Suite 419 Bloomington, IN 47406 Ph: 812-856-2169