Transcript Evaluating Academic Challenge Beyond the NSSE

Evaluating Academic
Challenge Beyond the NSSE
Jim Purcell, Associate Vice Chancellor
Oklahoma State Regents for Higher Education
Legacy of Assessment

Letter from the Six Nations to
William & Mary College

We are convinc'd, therefore, that you mean to do us Good by
your Proposal; and we thank you heartily. But you, who are
wise, must know that different Nations have different
Conceptions of things; and you will therefore not take it amiss,
if our Ideas of this kind of Education happen not to be the
same with yours. We have had some Experience of it.
Several of our Young People were formerly brought up at the
Colleges of the Northern Provinces; they were instructed in all
your Sciences; but, when they came back to us, they were
bad Runners, ignorant of every means of living in the Woods,
unable to bear either Cold or Hunger, knew neither how to
build a Cabin, take a Deer, or kill an Enemy, spoke our
Language imperfectly, were therefore neither fit for Hunters,
Warriors, nor Counselors; they were totally good for nothing.

We are, however, not the less oblig'd by
your kind Offer, tho' we decline
accepting it; and, to show our grateful
Sense of it, if the Gentlemen of Virginia
will send us a Dozen of their Sons, we
will take care of their Education;
instruct them in all we know, and make
Men of them.

Knowing is half the Battle


Know Thyself


GI Joe Cartoon Character and Action Figure
Socrates
“Every new adjustment is a crisis in selfesteem.”
Assessment is a Mirror

Assessment is a reflection of reality –
not reality
Now we see but a poor reflection
as in a mirror. Now I know in part;
then I shall know fully . . .
1 Corinthians 13:12
JoHari Window
Known by Self
Known by Others
Unknown by Others
Joseph Luft and Harry Ingham
Open
Area
Hidden
Area
Unknown by Self
Blind
Area
Unknown
Area
JoHari Window
Known by Self
Known by Others
Unknown by Others
Joseph Luft and Harry Ingham
Open
Area
Hidden
Area
Unknown by Self
Blind
Area
Unknown
Area
Georgia College & State University
Student Engagement


Notional value: We like the idea, but do not
take action.
Cost for caring about student engagement:

Investment of time, study, money, and risk.



What are we doing?
Is it working?
Important to take risk: Pause to think day.
The NSSE first Reading






Asked the questions that were at the
core of the institutional mission
Sense of Place
Sense of Activity
Sense of Interaction
Provoked faculty interest
Enabled institutional conversation
Selected NSSE findings
Asked questions in class or
contributed to class discussions.
National
Liberal Arts
Georgia College
1st-year 1st-year
1st-year
2.75
1= never
2.81
3.00
2= sometimes
Seniors
3.06
From NSSE 2000
3= often
Seniors
Seniors
3.16
3.30
4= very often
Selected NSSE findings
Worked with other students on
projects during class.
National
Liberal Arts
Georgia College
1st-year
1st-year
1st-year
2.42
2.28
1= never
2= sometimes
Seniors
2.49
From NSSE 2000
2.44
3= often
Seniors
Seniors
2.43
2.64
4= very often
Selected NSSE findings
Worked with other students on
assignments outside of class.
National
Liberal Arts
Georgia College
1st-year
1st-year
1st-year
2.39 2.31
1= never
2= sometimes
Seniors
2.71
From NSSE 2000
2.61
3= often
Seniors
Seniors
2.76
2.83
4= very often
National Survey of Student Engagement
Senior College Activities
Character of Mental Activ ities
2.59
2.63
2.57
2.35
Memorizing facts, ideas or methods from
your cour
3.04
3.07
3.1
3.25
Analyzing the basic elements of an idea,
experien
2.69
2.78
2.82
Synthesizing and organizing ideas, information,
o
3.07
2.62
2.68
2.69
2.88
Making j udgments about the v alue of information, a
3.06
3
3.01
3.09
Applying theories or concepts to
practical proble
1
Red Indicates Significance at the .001 lev el
Scale is 1=v ery little, 2=some, 3=quite a bit, 4=v ery much
2
GCSU
National
3
Master's I&II
Liberal
National Survey of Student Engagement
Senior College Activities
Weekly Activ ities
3.3
3.29
3.43
Preparing for class (studying, reading, writing, l
1.48
1.54
1.65
1.79
Working for pay on campus
GCSU
Master's I&II
National
Liberal
3.78
Working for pay off campus
3.18
2.14
1.52
1.65
1.77
Participating in co-curricular activ ities (organiz
3.62
3.65
2.25
2.63
2.75
2.87
3.07
Relaxing and socializing (watching TV, partying, e
Prov iding care for dependents liv ing with you
2.04
1.42
1
2
2.36
2.79
3
Red Indicates Significance at the .001 lev el
Scale is 1 = 1 or fewer hours/week, 2=6-10 hours/week, 3=11-15 hours/week, 4=16-20
hours/week, 5=21-25, 6=26-30, 7=more than 30
4
Faculty Response?




Bogus: The survey was not administered
correctly
Bad Survey: The survey failed to measure
student engagement
Bad Students: We deal with a different type
of student than other schools
Bubba’s problem: The results do not apply
to me and my colleagues in my department
A few months later . . .
National Survey of Student Engagement
Senior students –by GCSU Academic Schools
Statistically significant responses at .05 level or higher
Used e-mail to communicate with an instructor or
other students. (Sig. .018)
A&S (2.80), Bus (2.90) , Ed (3.40), HS (3.40)
Made a class presentation. (Sig. .015)
A&S (2.87), Bus (2.69) , Ed (3.42), HS (3.07)
Worked with other students on projects during
class. (Sig. .001)
A&S (2.33), Bus (2.55) , Ed (3.32), HS (2.60)
National Survey of Student Engagement
Senior students –by GCSU Academic Schools
Statistically significant responses at .05 level or higher
Participated in a community-based project as part of a
regular class course. (Sig. .000)
A&S (1.47), Bus (1.25) , Ed (2.21), HS (2.53)
Used an electronic medium to complete or discuss a
project. (Sig. .044)
A&S (2.27), Bus (2.17) , Ed (2.90), HS (2.53)
Discussed grades/assignment with instructor. (Sig. .014)
A&S (3.13), Bus (2.52) , Ed (3.00), HS (2.73)
National Survey of Student Engagement
Senior students –by GCSU Academic Schools
Statistically significant responses at .05 level or higher
Number of assigned textbooks, books, or book
length packs of course reading. (Sig. .005)
A&S (3.24), Bus (2.90) , Ed (2.79), HS (3.80)
Number of papers over 20 pages. (Sig. .004)
A&S (1.50), Bus (1.50) , Ed (1.25), HS (2.00)
Number of papers under 20 pages. (Sig. .022)
A&S (2.72), Bus (2.62) , Ed (3.40), HS (3.20)
National Survey of Student Engagement
Senior students –by GCSU Academic Schools
Statistically significant responses at .05 level or higher
Course work emphasizes synthesizing and organizing
ideas. (Sig. .005)
A&S (2.83), Bus (2.28) , Ed (2.70), HS (3.27)
Course work emphasizes making judgments about
the value of information. (Sig. .027)
A&S (2.79), Bus (2.31) , Ed (2.55), HS (3.20)
Preparing for class (studying, reading, etc) (Sig. .017)
A&S (3.77), Bus (2.86) , Ed (2.95), HS (4.29)
National Survey of Student Engagement
Senior students –by GCSU Academic Schools
Statistically significant responses at .05 level or higher
Contributed to acquiring a work related skills. (Sig. .007)
A&S (3.13), Bus (2.83) , Ed (3.45), HS (3.73)
Contributed to analyzing quantitative problems. (Sig.033)
A&S (2.69), Bus (3.00) , Ed (2.40), HS (3.20)
Contributed to the welfare of your community. (Sig. .005)
A&S (2.07), Bus (2.21) , Ed (2.65), HS (3.07)
A few more months later . . .
NSSE survey
Level of Academic Challenge
Active and Collaborative Learning
Student Interaction with Faculty
Enriching Educational Experience
Supportive Campus Environment
Evaluating Academic Challenge
Beyond the NSSE


Stephen L. Payne, Karynne L. M. Kleine, Jim
Purcell, and Ginger Rudeseal Carter
http://www.springeronline.com/sgw/cda/front
page/0,11855,4-40406-70-356121470,00.html
Kuh (2001) strongly encouraged institutional leaders to
adopt multi-faceted, multi-year, data-collection strategies
that capitalize on what contrasting assessment
approaches do best.
Academic Challenge
Eleven items are incorporated in the academic
challenge dimension of the NSSE.
Many items relate strongly to objectives on Bloom’s
and other learning taxonomies and others involve the
amount of time or effort that students must have on
the course.
Focus Group Questions
Do students at particular institutions discount the
importance of aspects of academic challenge that the
NSSE tries to measure?
Is the NSSE missing aspects of academic challenge that
faculty and students at particular institutions believe
would be helpful for improving academic challenge there?
How might an institution with a specific mission,
academic culture, and set of characteristics examine its
degree of academic challenge and move toward more
comprehensive assessment and improvement of this
academic challenge?
Focus Groups





Twenty-nine faculty
Fifty-four students
Main and branch campus students were interviewed.
Transcriber
Asked questions concerning:



conceptions of academic challenge,
the appropriateness of the academic challenge items on the NSSE,
and what might be done to improve academic challenge at this
institution.
Academic Challenge Questions
Students Faculty
Amount of time spent preparing for class
Number of assigned textbooks, books, or book-length packs of
course reading
Number of written papers or reports of 20 pages or more between 519; of fewer than 5 pages.
Analyzing the basic elements of an idea, experience, or theory such
as examining a particular case or situation in depth considering its
components
Synthesizing and organizing ideas, information, or experiences
new, more complex interpretations and relationships
Making judgments about the value of information, arguments, or
methods such as examining how others gathered and interpreted
data and assessing the soundness of their conclusions
Applying theories or concepts to practical problems or in new
situations
About how often have you worked harder than you thought you
could to meet an instructor’s standards or expectation?
To what extent does your institution emphasize spending
significant accounts of time studying and on academic work?
+
+-
-
+
-
-
+
+
+
+-
+
+
+
+
-
+
-
What is Academic challenge?
Students

Initial reactions to the concept of academic
challenge were frequently negative –




Emphasized the extra time and effort as well as the
initial uncertainty or confusion associated with such
courses
Students later mentioned rewards of
academically challenging courses.
Synthesis and, application of learning content
Faculty teaching style and student interactions
were a concern
Off-campus students


Stressed stronger concern for career
entry and success
Reacted less favorably than other
students to academic challenge in
courses, particular in core courses of
those connected to their majors
What is Academic challenge?
Faculty
Significant minority of faculty emphasized lack
of student preparation to enable student
engagement.
“Basics” limiting student opportunities for
higher-order learning and academic challenge
Need for strategies to challenge students.
Need improved resources for faculty
development
What is Academic challenge?
Faculty
Academic challenge requires a recognition of
different student learning styles and offering
multiple paths
Learning outcomes become more important
Need more and improved assessment using the
NSSE and other indicators
Proposal for expanded
assessment



Accepted by the Academic Governance
Committee of the University Senate of
GC&SU (2004)
Ask to devote time to developing this
internal assessment instrument
Used NSSE measures for benchmarks
for quality enhancement plan
Ideas for local research related
to the NSSE discussions



Critical thinking, reading, and writing
are at the core of academic challenge,
but these important concerns are not
handled the existing NSSE items
Several academic challenge items
stress student effort
Items are not linked to learning
outcomes
Ideas for local research related
to the NSSE discussions



Study relationships between student perceptions
of learning and actual outcome measures of
affective, creative, aesthetic, and leadership
development particular programs
Development of items that permit distinction
between experiences of courses
Distinguish between academic challenge and
experiences that are merely highly competitive,
demanding, or even intimidating.
What other interesting tidbits from
this study?
Students have less interest and motivation for
learning in core courses they perceive as having
less direct career relevance.
What roles of responsibilities might faculty and
students assume for improved learning
outcomes?
Establishing both high academic rigor in their
courses as well as a high level of faculty-student
relationships out of class seems to be a
contradiction to many faculty.
What other interesting
tidbits from this study?


Students placed greater
responsibility on faculty in their
learning experience.
Faculty feel that assignments can
result in more meaningful learning
outcomes than students might
realize at this point in their lives
Closing Remarks




We should design educational
experiences that are empowering
Measure quality of encounters
Ensure encounters are
transformational
We should desire positive
restlessness