Transcript No Slide Title

How Do We Really Know What Our
Students Are Learning?
Teaching and Assessing for Deep Learning
Douglas Eder
[email protected]
Southern Illinois University Edwardsville
Asteroid Impact Assessment
Assumptions & Expectations?
Assessment makes
learning visible
Assessment is the ongoing,
systematic examination of student
learning and the learning
environment.
Direct vs. Indirect Assessment


Direct assessment acquires evidence
about student learning and the learning
environment: Exams, projects, logs,
portfolios, observations....
Indirect assessment acquires evidence
about how students feel about learning
and their learning environment:
Surveys, questionnaires, interviews,
focus groups, reflective essays....
Assessment vs. Evaluation


Assessment focuses on the
student and the learning
environment.
Evaluation focuses on the
professor and the teaching
performance.
Assessment Questions Answers
Assessment Asks Questions
Bad questions take
just as much
energy to answer as
good questions do.
What is the purpose of college?
UTEP Goals, 2001-05
Learning & Teaching
(President’s Page)
Students will acquire tools to:
 meet lifelong intellectual,
ethical, and career challenges,
and
 be leaders.
The Basic Assessment Question
What do we want to know
about ourselves?
The Basic Assessment Question
Amplified
Are we doing what we say
we’re doing?
 Are we doing what we
ought to be doing?

* * * * *

Just what do we think
we’re doing?
Good Assessment
Asks Good Questions



What is your job description for your first
year students? For your second year
students? How do they differ?
Do we have a curriculum or a set of courses?
Which of these do our students experience?
How much and what kinds of writing do our
students do? What kinds of math? Are these
relevant to our curricular goals?
It’s human nature to respond to
subtle cues in the environment
The Scholarship of Teaching
Discovery
Integration
Application
Teaching
==================
Public and peer reviewed
Reflective
Six Principles of Scholarship
--Glassick et al.
1. Clear goals -- we identify the question
2. Adequate preparation -- we see the task
3. Appropriate methods -- we can do the task
4. Significant results -- we do care
5. Effective presentation -- we can escape
6. Reflective critique -- we can improve
Principle #1
Clear
Goals
What’s
the
quickest
way to
solve a
maze?
The First and Only Goal:
To teach for
long-term
retention of
information
and
application to
new situations
--after Halpern & Hakel
Deep (expert) learning is not
a course function.
It is a curriculum function.
Colleges and
Universities
are Systems
Cognitive Neuroscience
The more of the brain
--sensory, motor, and
association pathways-involved during the
learning process, the
more effective is the
learning.
P
ssssssssssss
ssssssssssss
ssssssssssss
ssssssssssss
ssssssssssss
s s
s s
s
s
s
s s
s s
P
s s
s
s
s s
s
s s
s
s
s
s s
s
s s
s s
s s
s s
s
s
s s
s
s
s s
s s s s s s s Ps s s s
s
s
s
s
s
s
ssssssssssss
P
s
s
s
s
s
s
s
s
ssssssssssss
Why do students come to
“classroom”?
Why do students come to
“classroom”?
Why should students come to
“classroom”?
Annotated Word Journal
Read the assigned text and write one word that captures the
essence of what you’ve read and summarizes your response
to it.
____________________
Explain why you chose that word and how it provides, in a
capsule, your summary of the reading.
____________________________________
____________________________________
The Importance of Goals
“Cheshire Puss...,” asked Alice, “would
you tell me, please, which way I ought
to go from here?”
“That depends a good deal on where you
want to get to,” said the Cat.
“I don't much care where—” said Alice.
“Then it doesn't matter which way you
go,” said the Cat.
Principle #2
Adequate
Preparation
Goals and Objectives
for Students


Goals express what we want our
students to be.
Objectives describe what we want
our students to do.
.....
• Objectives are indicators of goals.
To be assessment friendly,
objectives should:
Focus on students
 Make the learning goals visible
(serve as indicators)
 Describe behaviors or products
(doing, making) that can be
captured by assignments.

UTEP Core Curriculum Objectives?







Communications: To enable students to communicate effectively in
clear and correct prose in a style appropriate to the subject, occasion,
and audience.
Mathematics: To develop a quantitatively literate college
graduate…able to apply basic mathematical tools in the solution of
real world problems.
Natural Sciences: To enable the student to understand, construct and
evaluate relationships…and to understand the bases for constructing
and testing theories.
Humanities, Visual & Performing Arts: To expand students’
knowledge of the human condition and human cultures….
US History, Political Science: To expand students’ knowledge of the
origin and evolution of US & Texas history.
Social & Behavioral Sciences: To increase students’ knowledge of
how social and behavioral scientists discover, describe, and explain
[behavior].
Institutional Option: To develop [in students] the critical thinking
tools necessary to become effective learners.
An Assessment Question:
How Do You Know...
...that students walk out your
door looking like you want
them to? What behaviors have
they exhibited or products
have they produced? What are
the indicators for your goals?
Principle #3
Appropriate
Methods
Some Assessment Ways and Means












Assessment days and centers
Case studies
Classroom assessments
Completion and retention
studies
Content analyses
Debates
Direct observations
Focus groups
Graduate success
Internships and service learning
Interviews (including
videotapes)
Exams for certification and
licensure









Matrices
Performances
Portfolios of several kinds
Projects (Primary Trait
Analysis)
Questionnaires and surveys
(Direct and telephone;
employer, alumni, and student
attitude and satisfaction)
Reflective essays
Study and activity logs
Tests
(Locally-developed and
standardized)
Transcript analyses
If all you have
is a hammer,
everything
starts to look
like a nail.
Course or Curriculum
Alignment Grid
What are
your students
focusing on?
The Minute Paper
1. What is the most important
thing you learned here so far?
2. What is your most important
unanswered question from our
time together?
Primary Trait Analysis...
...is a very
strong link.
What We’re Really Looking for is:
Way Better than Good Enough
Good Enough
Not Good Enough
------------------------------That's Enough!
Principle #4
Significant
Results
Five Conditions That Foster
Deep Learning
[Halpern & Hakel, Marchese, DeWinstanley & Bjork, Shpancer,
National Research Council]
1. Engaging students actively
2. Practicing retrieval and presentation in
varied environments
3. Linking new learning to prior learning
4. Applying learning to new situations that
students care about
5. Receiving timely and relevant reinforcement
What are the characteristics of…




first year writing --- Pink
second year writing --- Yellow
third year writing --- Green
fourth year writing --- Blue
Cognitive Stages of Knowing
(Piaget, Perry, Kitchener, Baxter-Magolda -- Summarized by
Haynes
Absolute
Transitional
Method of authority
Answers come from
research
Uncertainty when
authority is
unavailable
Uncertainty when
answers are
unknown
Strategy: Gain control
of information
Strategy: Debate and
research
Cognitive Stages of Knowing
(Piaget, Perry, Kitchener, Baxter-Magolda -- Summarized by
Haynes
Independent
Contextual
Many valid interpretations
Evidence in context
All opinions are good,
including one’s own
Some opinions are
better than others
Strategy: Introspection
Strategy: Values are
important
Cognitive Stages of Knowing
(Piaget, Perry, Kitchener, Baxter-Magolda -- Summarized by
Haynes
Absolute Transitional Independent
%1st
%2nd
%3rd
%4th
%5th
68
47
11
2
--
32
52
83
80
--
0
1
5
16
57
Contextual
0
0
1
2
12
Staged Writing & Thinking...
...takes students from one cognitive level
to the next through incremental, not
additive, [writing/speaking] assignments.
e.g.,
Description/Narration
Analysis
Comparison/Contrast
Integration
Highest Priority Questions
The three highest priority questions that I
would like to look into regarding student
educational performance at this institution
are:
1.________________________________
2.________________________________
3.________________________________
What is Good [Effective]
Critical Thinking?
• Identifying
• Exploring
• Prioritizing
• Revisioning
--Wolcott & Lynch
Critical Thinking in a Writing Environment:
Grading and Assessing
Traits:
Critical Thinking (after Wolcott & Lynch)
1. Identifying
2. Exploring
3. Prioritizing
4. Revisioning
Writing
5. Consistent focus on topic or issue
6. Claims founded upon evidence
7. Language appropriate for the audience
8. Appropriate writing mechanics
9. Scholarly bibliographic support
Scoring:
27-23
22-16
15- 9
3
2
1
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
Exceeds expectations
Meets expectations
Does not meet expectations
Critical Thinking in a Writing Environment:
Grading and Assessing
Traits:
Critical Thinking (after Wolcott & Lynch)
1. Identifying
2. Exploring
3. Prioritizing
4. Revisioning
Writing
5. Consistent focus on topic or issue
6. Claims founded upon evidence
7. Language appropriate for the audience
8. Appropriate writing mechanics
9. Scholarly bibliographic support
Scoring:
27-23
22-16
15- 9
3
2
1
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
Exceeds expectations
Meets expectations
Does not meet expectations
Critical Thinking in a Writing Environment:
Levels of Performance
Exploring [2]
3 Probes alternatives and presents primary and secondary evidence in support.
2 Recognizes alternatives and acknowledges existence of evidence in support.
1 Does not recognize that alternatives may exist; ignores conflicting evidence.
Prioritizing [3]
3
2
1
Consistent focus on topic or issue [5]
3
2
1
Critical Thinking in a Writing Environment:
Grading and Assessing
Traits:
3
Critical Thinking (after Wolcott & Lynch)
1. Identifying
_x_
2. Exploring
___
3. Prioritizing
___
4. Revisioning
___
Writing
5. Consistent focus on topic or issue
_x_
6. Claims founded upon evidence
___
7. Language appropriate for the audience ___
8. Appropriate writing mechanics
___
9. Scholarly bibliographic support
_x_
Score = 19
Scoring:
27-23
Exceeds expectations
22-16
Meets expectations
15- 9
Does not meet expectations
2
1
___
_x_
_x_
___
___
___
___
_x_
___
___
_x_
_x_
___
___
_x_
___
___
___
Critical Thinking in a Writing Environment:
Grading and Assessing
Traits:
Critical Thinking (after Wolcott & Lynch)
1. Identifying
2. Exploring
3. Prioritizing
4. Revisioning
Writing
5. Consistent focus on topic or issue
6. Claims founded upon evidence
7. Language appropriate for the audience
8. Appropriate writing mechanics
9. Scholarly bibliographic support
[Mean score = 19.2
or
3
2
1
12
11
8
3
13
15
16
12
5
4
6
15
22
14
16
4
6
5
12
9
15
16
3
4
5
11
8
2.1 / 3  2.8 / 4]
QUESTION: What part(s) of the curriculum
deserves special attention?
Assessment is...
...more than finding out how
many students passed and
what the average score is on
a test of competence.
Construct an effective
3- or 4-point rubric for
some aspect of writing
or critical thinking.
Pay special attention to
the level identified as
“acceptable.”
An Important Lesson
from the Farm
A pig doesn’t get
any fatter merely
by weighing it.
Closing Thought
The enemy of the good
is the perfect.