Transcript Undergraduate scholarship and other High Impact Practices

Undergraduate Scholarship and
other High Impact Practices:
Assessing the Outcomes,
Not the Content.
Norman Jones
Director of General Education & Curricular Integration
Utah State University
[email protected]
Assessment of teaching in 2030 will
probably not look like it does now. The
delivery of “content” will not be our
job. It will be to prepare students for
the human jobs that can’t be done by
machines, jobs requiring personal
interrelations and critical thinking.
The Future of Higher Ed
“Degrees and other postsecondary credentials
can’t simply be defined by the amount of time a
student spends in classrooms or labs. Rather,
degrees must represent well-defined and
transparent learning outcomes. In short, students
should get credit for what they know and what
they can do. And all learning should count ― no
matter how, when or where it was obtained.”
Jamie Merisotis, President, Lumina Foundation
The traditional assessment method is
to measure progress incrementally,
using quizzes and tests, measuring
individual performance against the
average of group performance, over a
fixed period of time, calibrated in
multiple units of seat time.
 “Current assessment practice, for the most
part, rests on faculty-established goals,
developed independently at each institution,
for what graduates should know and be able to
do.”
 Whether or not graduates attain these goals is
then investigated on average by using various
methods to examine the performance of
representative samples of students.”
Peter T. Ewell, “The Lumina Degree Qualifications Profile (DQP):
Implications for Assessment.”
http://www.learningoutcomesassessment.org/documents/EwellDQPop1.pdf
These methods do not ask about
developing individual
proficiencies, or mastery, or the
possibility of learning that occurs
outside the classroom.
We Must Teach Toward Proficiency:
“a set of demonstrations of
knowledge, understanding and skill
that satisfy the levels of mastery
sufficient to justify the award of an
academic degree.”
Degree Qualification Profile, 2.0
http://www.luminafoundation.org/publications/DQP/DQP2.0-draft.pdf
In order to assess for proficiency
we have to rethink what we are
asking students to learn. Is it
content, or the application of
content?
 “Assignments or examination questions designed to
determine proficiency in particular DQP competencies,
consequently, must require students to generate a
product of some kind—a research paper, an oral
presentation, a dance performance, a translation of a
text from one language to another, an engineering
design.”
 “Merely identifying a “correct” answer from a set of
posed alternatives is not a production task. Because the
assessments associated with DQP competencies
require students to directly demonstrate mastery, the
assessment really is the competency from an
operational standpoint.”
Ewell, “DQP…Asssessment” 12.
WE MUST
Decrease emphasis on content. They
have libraries in their pockets.
Create opportunities for practice,
linking content to a need to know.
Emphasize acquisition, deployment,
and communication of knowledge
through experience.
High Impact Practices
develop their proficiencies,
more than their book
learning.
Pedagogical High Impact
Practices that Reinforce
Proficiencies
 First-Year Seminars and Experiences
 Undergraduate Research
 Writing-Intensive Courses
 Capstone Courses and Projects
Institutionally Organized
Experiences that Reinforce
Proficiencies:
 Common Intellectual Experiences
 Learning Communities
 Diversity/Global Learning
 Internships
 Service Learning
HIP Assessments
 There are “Process” assessments – based
on course design
 There are “Outcomes” assessment –
based on demonstrated proficiency
 In “Process” assessments we
identify the target outcomes and
design ways for students to
prepare and practice for them.
 In "Outcomes” assessments,
students produce evidence of
proficiency.
At USU, we built Gen Ed
template rubrics that identify
the proficiencies we expect a
course to deliver
WITHOUT PRECISELY
DEFINING CONENT.
Breadth Life Science Rubric
Knowledge:
Understand how the
enterprise of science works
(i.e., erecting testable
hypotheses, refining
hypotheses, reproducible
results, etc.)
Learn the key laws, concepts
and processes that govern
biological systems.
The student who The student who
attains
is approaching
proficiency
proficiency
Can apply the
basic structure
and methodology
of scientific
enterprise.
Knows the key
laws and
concepts, and is
able to apply
them to novice
problems.
Can articulate
the basic
structure and
methodology of
scientific
enterprise.
Knows the key
laws and
concepts and can
articulate them.
The student who
lacks proficiency
Is unable to
articulate the basic
structure and
methodology of
scientific enterprise.
Does not know the
key laws and
concepts beyond
memorization.
Utilize quantitative
methods to collect,
analyze and interpret
scientific information
Can interpret, read,
understand and explain a
graph, table, or
quantitative series of data,
and apply that
understanding to a
problem.
Can interpret, read and
understand a graph,
table, or quantitative
data.
Is not able to interpret,
read or understand a
graph, table, or
quantitative series of data.
Evaluate the credibility
of various sources of
information about
science-related issues.
Can assess the credibility
of sources of scientific
information, and critique
source as it applies to a
scientific issue.
Can assess credible
sources of scientific
information, and can
articulate why they are
credible.
Cannot assess credible
sources for scientific
information, or unable to
determine credibility of
sources.
Use written or visual
communication to
demonstrate
knowledge of scientific
findings.
Can write and/or
illustrate knowledge of a
scientific idea or concept
clearly, comprehensively,
and concisely.
Can write and/or
illustrate knowledge of
a scientific idea or
concept.
Is unable to convey
knowledge.
Examine the
relationship of the
science learned to
societal issues (such as
sustainability, etc...)
Can apply science
concepts and societal
issues to the greater
question of the course.
Can articulate the
relationship between
science concepts and
societal issues.
Is unable to recognize the
links between social
issues and scientific
findings.
These template rubrics require that
achieving proficiency in the
outcomes be intentionally designed
into the course. Using active verbs,
we demand demonstration that
students know, understand and are
able to do generic things.
The template rubrics establish the
hierarchies of proficiency in the
subject, and give form to the
grading rubrics used to assess the
demonstrations of competence.
HOW DO YOU
ASSESS AN
EXPERIENCE THAT
HAS A PROFICIENCY
OUTCOME?
We have to think about the “take
away” from the experience. If we
focus on outcomes, there is no
precise question like “Who is buried
in Grant’s tomb?” There are no
points to be lost for incorrect
answers.
We want to understand the
student’s engagement with
practice. Has he or she
demonstrated proficiencies in the
discipline through use of the tools
of the discipline?
We have to think about the
demonstration of proficiencies
within the outcomes, both in
courses and in degrees. They
build together, reaching apogee in
the capstone project.
Building Pathways to Increasing
Proficiency
As we refine our templates and
enact them in course rubrics, we can
begin to map the way student
practice builds proficiency over the
curriculum. That means we can
envision appropriate pathways to
degrees.
USU, HIST 4990: Senior
Capstone Common Rubric
Used by all capstones in the
History degree to assess
proficiencies.
Learning
Student frames
historical
questions in a
thoughtful,
critical manner.
Excellent
mastery
Good mastery
Some mastery
The paper
addresses a
significant
historical question
that is clearly
stated. The
question’s
significance is
satisfactorily
demonstrated; the
student is
conscious of the
role of
periodization in
forming the
question; the
question is of
manageable scope
and logically
formulated.
The paper
addresses a
significant
historical question
that is clearly
stated. The
student makes an
effort to
demonstrate
significance and
to employ
periodization.
Question is of
manageable
scope, posed with
minimal logical
flaws in question
framing.
The paper
addresses a
historical question
that can be
identified with
some difficulty.
Significance of
question unclear;
minimal grasp of
periodization;
serious logical
lapses in question
framing.
Minimal
mastery
Significance of
question not
demonstrated;
question of
inappropriate
scope or
illogically
presented; no
grasp of
periodization.
No mastery
No identifiable
historical
question.
Student
employs a
range of
primary
sources
appropriate to
the informing
thesis of the
paper.
Makes
thorough use
of all relevant
online and
print databases
to identify
primary source
literature; all
available
primary
sources
identified. All
sources in
bibliography
thoroughly
used in text.
Makes good
use of relevant
online and
print
databases;
some gaps in
primary source
base. A few
sources in
bibliography
not fully used.
Makes some
use of online
or print
databases;
significant
gaps in source
base; paper
based on only
a few of cited
sources.
No evidence of
using
databases to
establish
source base;
source base
very limited.
Major sources
unknown or
not employed.
Little evidence
that author has
used works
listed in
bibliography.
No evidence of
using
databases;
sources
entirely
insufficient
and
inappropriate
to paper topic.
Student
evaluates
and
analyzes
primary
sources.
Shows
thorough
awareness
of origins,
authors,
contexts of
all primary
sources;
consciously
employs
verification
strategies as
needed.
Shows
Offers
some
partial
awareness evaluation
of contexts of primary
of primary
sources;
sources;
spotty
employs verification.
some
verification
strategies.
Offers little
Is not
to no
aware of
evaluation
need to
of primary evaluate or
sources; no
verify
verification. sources.
Through assessing the 7
degree outcomes, we are
asking if students have
sufficient mastery to be
granted a degree. The senior
thesis is the artifact used as a
demonstration.
We are “preparing students to tackle
nonstandard, unscripted problems and
questions. Unscripted problems, by
definition, are those where ‘right
answers’ are not known and where the
nature of the problem itself is likely
uncertain at best, and often actively
contested.”
Carol Geary Schneider in Ewell, “DQP…Asssessment,” 25.
 Our only real assessment audience
for improvement is ourselves. It
reassures us that we can honestly say
we are educating our students.
 Is a fancy metric needed?
 No. Proof of proficiency is.
 Process assessment + Proof of
outcomes performance = evidence
of proficiency.
 An Intentional curriculum, taught
by intentional faculty to intentional
learners.
 Through it, every students’
proficiency is established.