Transcript AACSB Assurance of Learning Standards - UPR-RP

Anchoring Assessment in its Place:
Reviving, Surviving, and Even Thriving
Douglas Eder
[email protected]
Southern Illinois University Edwardsville
Strongest Links:
The Scholarship of Teaching
•
•
•
•
•
•
Clear Goals
Adequate Preparation
Appropriate Methods
Significant Results
Effective Presentation
Reflective Critique
Asteroid Impact Assessment
Assumptions & Expectations?
Assessment vs. Evaluation
•
Assessment focuses on the
student and the learning
environment.
•
Evaluation focuses on the
professor and the teaching
performance.
Direct vs. Indirect Assessment
•
•
Direct assessment acquires evidence
about student learning and the learning
environment: Exams, projects, logs,
portfolios, observations....
Indirect assessment acquires evidence
about how students feel about learning
and their learning environment:
Surveys, questionnaires, interviews,
focus groups, reflective essays....
We Give Grades, Don’t We?
Aren’t grades
(by themselves)
enough?
NO!
Grades may reflect many things besides
student mastery of course objectives:
Verbal ability
Participation
Cooperation
Extra credit
Attendance
Effort
Criterion Performance vs. Value Added
Myths regarding student evaluations
When used alone, grades are no
longer regarded as adequate
indicators of student learning.
 Professor autonomy: Grades in one
course or section may be recorded
using a different standard than
grades in another.
 General accountability: Much of the
current public annoyance with higher
education comes from a lack of skill
and content mastery by students who
have received good grades.
Why Assessment Won’t Go Away
• Limits on money
• Shift in the public view of Higher
Education
* * * * *
Result: We must demonstrate quality of
outcome and return on investment.
Through assessment educators meet
responsibilities to themselves, to
students, and to the public.
Cognitive Neuroscience
The more of the brain
--sensory, motor, and
association pathways-involved during
learning, the more
effective the learning
process.
Deep (expert) learning is a
curriculum function.
It is not a course function.
Universities
are systems
Assessment Asks
Good Questions
Bad questions take
just as much
energy to answer as
good questions.
The Basic Assessment Question
What do we want to know
about ourselves?
The Basic Assessment Question
Amplified
•
Are we doing what we say
we’re doing?
• Are we doing what we
ought to be doing?
* * * * *
•
Just what do we think
we’re doing?
Focus on the Students
It’s not so much what the
professors do that matters.
It’s what the students do.
. . . . .
So, how do we know what
students do?
Good Assessment
Asks Good Questions
•
•
•
What is your job description for your first
year students? For your second year
students? How do they differ?
Do we have a curriculum or a set of courses?
Which of these do our students experience?
How much and what kinds of writing do our
students do? What kinds of math? Are these
relevant to our curricular goals?
Directed Paraphrase
In 1-3 clear, concise sentences, please
write your definition of deep learning
(in a discipline or in general education).
Construct a definition that would make
sense to your students regardless of
their years in school or academic
disciplines.
Outline
•
•
•
•
•
•
Clear goals -- we identify the question
Adequate preparation -- we see the task
Appropriate methods -- we can do the task
Significant results -- we do care
Effective presentation -- we can escape
Reflective critique -- we can improve
Principle #1
Clear
Goals
What’s
the
quickest
way to
solve a
maze?
What do you think are the most
desirable characteristics for new
employees to have?
•
•
•
•
•
Self motivation -- training is
expensive
Business etiquette -- ability to
represent
Written communication -- can explain
a complex topic simply
Oral communication -- group and
individual
Disciplinary competence is assumed
What Are Your Goals
for Your Students?
What are your “Big Rocks”
for your students? What do
you want your students to
look like when they walk out
the door after spending time
under your supervision?
What do you think your new
graduates should look like when
they walk out your door?
•
•
•
•
•
Goal
Goal
Goal
Goal
Goal
Student goals are not necessarily
what the faculty pictures for them.
The mission of the Black Walnut Inn
is:
Firstly
That guests experience accommodations and
hospitality that meet the highest standards
expected by the discerning traveler, and that this
will encourage the novice Bed and Breakfast
guests to repeat their choice.
Secondly
That the owners experience a relaxed lifestyle
that includes friendly relationships with their
guests, a workload that retains enthusiasm, and a
monetary return that avoids financial concerns.
The Importance of Goals
“Cheshire Puss...,” asked Alice,
“would you tell me, please, which
way I ought to go from here?”
“That depends a good deal on where
you want to get to,” said the Cat.
“I don't much care where—” said Alice.
“Then it doesn't matter which way you
go,” said the Cat.
Approximate Analogy
INSTRUCTIONS: Complete the analogy
below.
1. Assessment is to teaching as:
______________ is to _______________.
Explain:
__________________________________
__________________________________
__________________________________
Approximate Analogy
INSTRUCTIONS: Complete the analogy
below.
1. F2F learning is to e-learning as:
_______________ is to _______________.
Explain:
__________________________________
__________________________________
__________________________________
It’s human nature to respond to
subtle cues in the environment
The Importance of Goals
If you don’t watch
where you’re going,
you’re liable to wind
up somewhere else.
--attributed to Yogi Berra
Principle #2
Adequate
Preparation
Thinking About Learning
What do we know about
how mammals learn?
They must see the bar.
The bar must be steady.
There must be a fish.
A Good Rule for Practicing
Medicine and Everything Else
Diagnose first
Prescribe second
Four Barriers to Institutional
Improvement
•
•
•
•
Amnesia
Fantasia
Inertia
Nostalgia
--after Shulman
Why do we insist on:
•
•
•
measuring with a
micrometer?
when we mark it with
chalk?
and cut it with an axe?
----after Peter Ewell
SIUE Assessment Mantra
Assessment
monitors
student
learning;
It does not evaluate
faculty teaching.
The Basic Assessment Question
What do we want to
know about
ourselves?
Angelo’s Rule
Caveat Assessor:
If you don’t want to
know, don’t ask.
How Assessment Causes Change
What is measured
is valued;
What is ignored
doesn’t exist.
Goals and Objectives
for Students
•
•
Goals express what we
want our students to be.
Objectives describe what
we want our students to
do.
• Objectives are .....
indicators of goals.
Objectives to be assessed...
...must be written so that
they specify visible student
behaviors or products.
An Assessment Question:
How Do You Know...
...that students walk out
your door looking like
you want them to? What
behaviors have they
exhibited or products
have they produced?
What are the indicators
for your goals?
Principle #3
Método
Apropiado
Whatever Assessment Is...
...It’s Multiple Measures
Over Time.
If all you have
is a hammer,
everything
starts to look
like a nail.
Some Assessment Ways and Means
•
•
•
•
•
•
•
•
•
•
•
•
Assessment days or centers
Case studies
Classroom assessments
Completion and retention
studies
Content analyses
Debates
Direct observations
Focus groups
Graduate success
Internships and service
learning
Interviews (including
videotapes)
Exams for certification and
licensure
•
•
•
•
•
•
•
•
•
Matrices
Performances
Portfolios of several kinds
Projects (Primary Trait
Analysis)
Questionnaires and surveys
(Direct and telephone;
employer, alumni, and
student attitude and
satisfaction)
Reflective essays
Study and activity logs
Tests
(Locally-developed and
standardized)
Transcript analyses
An Effective Consequence of the
Basic Assessment Question
Assess the things
that matter within
your university
culture.
--Pat Hutchings
Course or Curriculum
Alignment Grid
What are
your
students
focusing
on?
Curriculum Alignment:
An Alternative View
Assessment “Autopista”
A. What are you already doing
and what opportunities could
you pursue in order to do
assessment?
What Ways and Means
(p.11) might you use?
Cautions and Obstacles to
Assessment Initiatives
Assessment Barrera
B. What resource$ do you
need?
C. What or who is standing
in the way of using
assessment to monitor
student learning?
Principle #4
Significant
Results
The Main Thing
is to keep the
Main Thing
the Main Thing.
How do you know that your students
can do there what they do here?
The Flight Simulator...
...combines
teaching,
learning, and
assessing into a
real task.
Because
assessment is
embedded in
teaching and
learning, it is
largely
transparent.
The Academic Simulator...
...is a
reflective,
scholarly
engagement
between
student and
dedicated
professor
that results in
a visible
product or
behavior.
Authentic Assessment is...
• appraisal
of a real task
• in a real or simulated
environment
• under supervision
The closer any simulation is to the
real thing, the better the appraisal.
Traits of the Senior Assignment (SRA)
(SIUE’s Academic Simulator)
•
•
•
•
•
•
Embedded in learning and
teaching
Matches departmental goals
Departmentally owned
Viewed by the faculty
High stakes assessment
Liberal education component
High Stakes Assessment
STUDENT LEARNING IS “AUTHENTICALLY”
ASSESSED IN PUBLIC ENVIRONMENTS
• Civil Engineering: Solid waste reduction, campus seismic
•
•
•
•
•
•
analyses, campus water run-off, water treatment, dam repair
Chemistry: Student presentations and defenses before external
referees, including Nobel Laureate Roald Hoffmann
Psychology: Presentations and defenses at annual meeting of
Midwest Psychology Association in Chicago
Art & Design: Mexica95, Mexica96, ... Mexica02 exhibitions
Speech Communication: Academic/Student Affairs
connections by studying academic advising, minority scholarship
use, conflict resolution resident halls, University Speech Center
Liberal Studies: American Sign Language teaching protocol for
college lectures
Nursing: Community health outreach in Mexico City
Direct Curriculum Feedback
from Senior Assignment Findings
• Chemistry: Added writing and speaking at junior level;
•
•
•
•
•
increased sophomore writing emphasis.
Psychology: Increased statistics, hypothesis forming and
testing; shifted teaching emphasis toward research design.
Biology: Added student presentations; upgraded
instruction in computer use, speech, graphics, and statistics.
Mathematics & Statistics: Increased writing and
computer use; emphasized mathematical experimentation.
Music: Increased collaborative learning by emphasizing
ensemble performance opportunities.
Business: Increased analytical writing, critical thinking,
ethical understanding, working in teams.
Primary Trait Analysis...
...is a
very
strong
link.
What We’re Really Looking for is:
Way Better than Good
Enough
Good Enough
Not Good Enough
------------------------------That's Enough!
The Faculty Sets the Standards
What is Good [Effective]
Critical Thinking?
• Identifying
• Exploring
• Prioritizing
• Revisioning
--Wolcott & Lynch
Critical Thinking in a Speaking Environment:
Grading and Assessing
Traits:
Critical Thinking (after Wolcott & Lynch)
1. Identifying
2. Exploring
3. Prioritizing
4. Revisioning
Speaking (after Martell)
5. Voice and pace
6. Body language, mannerisms
7. Professionalism, audience awareness
8. Use of media
9. Handling of questions
Scoring:
27-23
22-16
15- 9
Exceeds expectations
Meets expectations
Does not meet expectations
3
2
1
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
__
Critical Thinking in a Speaking Environment:
Levels of Performance
Exploring [2]
3 Probes alternatives and presents primary and secondary evidence in support.
2 Recognizes alternatives and acknowledges existence of evidence in support.
1 Does not recognize that alternatives may exist; ignores conflicting evidence.
Prioritizing [3]
3
2
1
Use of media [8]
3
2
1
Critical Thinking in a Speaking Environment:
Grading and Assessing
Traits:
Critical Thinking (after Wolcott & Lynch)
1. Identifying
2. Exploring
3. Prioritizing
4. Revisioning
Speaking (after Martell)
5. Voice and pace
6. Body language, mannerisms
7. Professionalism, audience awareness
8. Use of media
9. Handling of questions
Score = 19
Scoring:
27-23
Exceeds expectations
22-16
Meets expectations
15- 9
Does not meet expectations
3
2
1
_x_
___
___
___
___
_x_
_x_
___
___
___
___
_x_
_x_
___
___
___
_x_
___
___
_x_
_x_
___
___
_x_
___
___
___
Critical Thinking in a Speaking Environment:
Grading and Assessing
Traits:
Critical Thinking (after Wolcott & Lynch)
1. Identifying
2. Exploring
3. Prioritizing
4. Revisioning
Speaking (after Martell)
5. Voice and pace
6. Body language, mannerisms
7. Professionalism, audience awareness
8. Use of media
9. Handling of questions
[Mean score = 19.2
or
3
2
1
12
11
8
3
13
15
16
12
5
4
6
15
22
14
16
4
6
5
12
9
15
16
3
4
5
11
8
2.1 / 3  2.8 / 4]
QUESTION: What part(s) of the curriculum
deserves special attention?
Assessment is...
...more than finding out
how many students
passed and what the
average score is on a
test of competence.
Choose one objective and...
...write a 3- or 4- point
standard or rubric that
represents either
qualitative or
quantitative
assessment of
learning.
The Minute Paper
1. What is the most important
thing you learned in this
session?
2. What is your most
important unanswered
question from our time
together so far?
Principle #5
Effective
Presentation
The Minute Paper
1. What is the most important thing
you learned in this session?
2. What is your most important
unanswered question from our
time together so far?
Institutional Effectiveness
A comparison of results
achieved to goals intended.
--Ewell, 1985
A Picture of Accountability
What To Assess?
1
2
3
 Incoming  Curriculum Teaching & Learning
Student
Processes

5
4
 Outgoing Student  Outcome
--after Vars
The Three Curricula
•
•
•
The one in the catalog
The one the professors teach
The one the students experience
Three Parts of a Public
Assessment Report
•
What have you learned?
• What are you going to do about it?
• When and how are you going to do it?
Administrivia Requirement
One hour
Per department
Per year
Assessment Reports
•
•
•
Who writes them?
Who reads them?
What happens to them?
Attracting the Audience
Is Important
Principle #6
Middle States,
AACSB,
Baldrige,
NCATE, ABET,
and others all
require:
Reflective
Critique
An Important Lesson
from the Farm
A pig doesn’t
get any fatter
merely by
weighing it.
What Next?
What is the weakest link?
•
•
•
•
Missing or
inappropriate
feedback
Little or no
faculty
involvement
Does not
address goals of
curriculum
Strictly norm
based
Guiding Principles for Assessment
[Angelo, 1997]
•
•
•
•
•
•
Actively engages participants
Addresses what matters most
Operates at key points and
times
Embeds in normal activities
Focuses on processes as well
as products
Provides timely, relevant
feedback to those who need it
Closing Thought
The enemy of the
good is the
perfect.
Assessment vs. Evaluation
•
Assessment focuses
on the student and the
learning environment.
•
Evaluation focuses on
the professor and the
teaching performance.
Examples of Things Students are
Qualified to Evaluate
•
Perceived fairness
The professor tested over what I was expected to
learn.
•
Communication effectiveness
The professor communicated in language and
with examples that I could understand.
•
Perceived challenge
I felt stretched by the challenges of this course.
•
Availability
The professor was available to answer my
questions during office hours or through ecommunication.
•
Timeliness
The professor began and ended class on time.
Examples of Things Students are
Not Qualified to Evaluate
• Professor’s
preparation
The professor was prepared for
class.
• Currency
of information
The professor was up-to-date in
his/her field.
• Relevance
The content of this course is
relevant to the
discipline.
Some Areas of Ambiguity
•
Expectations
This class was what I expected.
•
Degree of difficulty
This class challenged me at an
appropriate level.
•
Organization
The professor was organized.
•
Engagement/interactivity
This course engaged me to learn well.
End-of-semester
evaluations are
one way for
professors to
receive formative
feedback and
correct errors.
Dangerous “Question #9”
Compared with other
instructors and other courses
at this level, I rate this
instructor/course overall as:
_____
• Higher ratings go to
easier graders.
Myths?
•
•
•
•
Students learn more from
highly rated professors.
Student ratings and
trained observer rating
agree.
Students rate style and
enthusiasm.
Student ratings truly
reflect instructional
quality.
• Profs who teach larger
classes get lower
ratings.
• Rating scales influence
ratings
• Ratings are stable
Evaluations:
Summative and Formative
Evaluations:
Summative and Formative
Asking students
what they “liked”
or “valued” about
their
classes...was far
less productive
than asking them
what they had
specifically
gained.
--Linda Nilson, Putting Real
Value in Student
Evaluations, POD
presentation
Slides by Friday
También, web recurso:
http://www.siue.edu/
~deder/assess/index.html
with links to major
assessment sites
worldwide
Regla de 15/35
In any organization:
•
15% leaders, opinion shapers
(Includes dreamers)
• 35% willing followers, early
adopters
---------------------------------------------------• 35% reluctant followers, late
adopters
• 15% curmudgeons
15% Curmudgeons
•
5% already busy and
productive
Leave them alone
• 5% reasoned skeptics
Take them to lunch
• 5% just plain mean
Leave them alone
Don Farmer’s Advice
“I know this bothers you
and you don’t want to
think about it. But...
...please
don’t
prevent us
from
thinking
about it.”
To get faculty buy-in...
Start small.
Use pilot projects to capture
the 15% leaders + 35%
early adopters.
Fertilize...
...then get out
of the garden.
We have never been asked...
...to assess
everything all at once.
7 Principles of Good Practice in
Effective Undergraduate Education
•
•
•
•
•
•
•
Encourages contact between students
and faculty
Develops reciprocity and cooperation
among students
Encourages active learning
Gives prompt feedback on performance
Emphasizes time on task
Communicates high expectations
Respects diverse talents and ways of
learning
My Goals
•
•
•
•
•
•
•
Facilitate a conversation about
education goals
Provide some assessment techniques
Outline assessment organization for
accountability
Give some materials for you to keep
Give some time to think
Give some time to share
Enjoy
The Minute Paper
1. What is the most important
thing you learned in this
session?
2. What is your most
important unanswered
question from our time
together so far?
Most Important Thing
•
•
•
•
•
Assessment is not the same thing as
evaluation of faculty = 5
Goals and objectives are different = 5
Assessment has realistic steps and
guidelines = 3
The focus of program assessment is on
student learning = 3
Assessment does not have to be
complicated = 2
Important Unanswered Question
•
•
•
•
•
•
How do we operationalize assessment? = 2
I need more help to identify indicators = 2
How do I do assessment in a large class? = 1
Teaching vs. learning -- when I assess learning
I find out much about my teaching = 1
I have questions about qualitative assessment
because of terms like “adequate” and
“appropriate” = 1
I need self confidence = 1
Important Unanswered Question
•
•
•
•
•
•
•
How do we apply specific techniques? = 1
How to apply assessment techniques to
administrative departments = 1
Is this all possible? = 1
How does one write an assessment report? = 1
How to use assessment for improvement? = 1
What are valid program objectives?
Assessment may take away the most
important educational goals = 1
Bloom’s Taxonomy
LEVEL
SOME COGNITIVE BEHAVIORS
•
Evaluation
•
Synthesis
Analysis
Application
Understanding
Knowledge
language,
Appraisal of an Analysis
or Synthesis
Assembly of Application
Disassembly of Application
Use of Understanding
Management of Knowledge
Memorization of facts,
concepts, principles, theories
•
•
•
•
Analytical Minute Paper:
Analysis via Bloom’s Taxonomy
SIUE 2003 NFO
Knowledge Understanding Application
Define
Identify
Identify
Identify
Relate
Relate
Repeat
Describe
Describe
Describe
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Explain
Locate
Tell
Tell
Tell
Tell
Tell
Tell
Tell
Tell
Tell
Tell
Apply
Apply
Give example
Give example
Illustrate
Illustrate
Interpret
Analysis
Differentiate
Differentiate
Differentiate
Synthesis
Evaluation
Analytical Minute Paper:
Analysis via Bloom’s Taxonomy -- UPR
Knowledge Understanding Application
Analysis
Synthesis
Identify
Describe
Apply
Differentiate Assemble
Explain
Apply
Assist
Explain
Give examples
Explain
Explain
Explain
Explain
Clarify
Evaluation
Annotated Word Journal
Read the assigned text and write one word that
captures the essence of what you’ve read and
summarizes your response to it.
____________________
Explain why you chose that word and how it
provides, in a capsule, your summary of the
reading.
______________________________
_______________________________
Assessing Group Effectiveness
1. What specific goal(s) is this group trying to
accomplish? Please list the goals(s) in your
priority order. Do you think the group basically
agrees on the contents of this list?
2. What activities has the group specifically chosen
to undertake or assign in order to achieve its
goals? Which activities, if any, are particularly
effective?
3. Does each group member have specific --even
unique-- responsibilities that help the group
attain its goal(s)? List all group members by
name and their individual responsibilities.
Assessing Group Effectiveness
4. Do you find the work of your group stimulating
and worth your time? How many hours per week
do you spend working with this group? In the
table below, enter the percentage of these hours
spent in each category of effectiveness.
Very Effective
Effective
Marginally Effective
Not Effective
5. Does this group have the resources (e.g.,
organization, communication, leadership, talents,
time) to achieve its goals? What additional
resources are needed for real effectiveness?
Time Log
How do students use the 90%
of their time not in the
classroom?
RSQC2
•
•
•
•
•
Recall
Summarize
Question?
Connect
Comment
Assessment of Self Confidence
1. Set and explain
learning goals
2. Determine most
effective pedagogy
3. Use the pedagogy
4. Motivate students to
engage
5. Establish standards of
performance
6. Use Primary Trait
Analysis
7. Apply the feedback
Very Somewhat Not very Not at all
Very Somewhat Not very Not at all
Very Somewhat Not very Not at all
Very Somewhat Not very Not at all
Very Somewhat Not very Not at all
Very Somewhat Not very Not at all
Very Somewhat Not very Not at all
Very Somewhat Not very Not at all
Very Somewhat Not very Not at all
Approximate Analogy
INSTRUCTIONS: Complete the analogy below.
1. Assessment is to teaching as:
_______________ is to _______________.
Explain:
__________________________________
___________________________________
___________________________________
Select an idea about assessing
student learning
•
•
•
•
•
•
Don’t filter or censor for now.
Why do I care about this question?
Who else cares?
Who might benefit?
Who might be inconvenienced?
Upon reflection, do I want to
proceed?
Possible Benefits
•
•
•
•
•
•
•
•
More effective student learning?
More efficient teaching?
Better use of time?
Number of people benefit?
Improved finances?
Potential rise in status?
Improved cheer and good will?
________________________?
Possible Costs
•
•
•
•
•
•
•
•
Confusion and administrivia?
Extra work?
Extra time?
Inconvenienced people and
committees?
Drain on finances?
Potential loss of status?
Resistance and grumpiness?
________________________?
Benefit/Cost Ratio
Reckon the benefit/cost ratio:
Benefits
Costs
Is the result greater or
lesser than 1.0?
Rethink Benefits and Costs
In order to reap potential general education
assessment benefits, are might we give up
some old things so that some promising new
things might grow?
• Are there any additional questions bearing
on the concept that this conference has
raised?
• Which of these questions, if any, remain
unresolved?
•
Dealing creatively with
[human] obstacles
An Old Gaelic Blessing
May those who love us, love us,
And those that don’t love us,
may God turn their hearts,
And if he doesn’t turn their
hearts, may he turn their
ankles,
So we’ll know them by their
limping.