Transcript No Slide Title

Assessment
Is like a dancer’s mirror.
It improves one’s ability to see and
improve one’s performance.
Alexander Astin
1993
ASSESSMENT . . .
“a rich conversation
about student learning
informed by data.”
-- Ted Marchese -AAHE
Assessment of Individual
Student Development
Assessment of basic skills for use in
advising
•
•
Placement
Counseling
Periodic review of performance with
detailed feedback
End-of-program certification of
competence
•
•
•
Licensing exams
External examiners
CLAST
Key Results of Individual
Assessment
 Faculty
can assign grades
 Students
learn their own
strengths and weaknesses
 Students
become selfassessors
A Second Look
Across
students
Across
sections
Across
courses
 Where
 What
is learning satisfactory?
needs to be retaught?
 Which
approaches produce the
most learning for which
students?
Group Assessment
Activities
•
•
•
•
•
•
•
Classroom assignments, tests,
projects
Questionnaires for students,
graduates, employers
Interviews, focus groups
Program completion and placement
Awards/recognition for graduates
Monitoring of success in graduate
school
Monitoring of success on the job
Use of Results of Group
Assessment
•
Program improvement
•
Institutional and / or state
peer review
•
Regional and / or national
accreditation
Outcomes Assessment
The process of providing credible
evidence of the processes
and outcomes of higher education
undertaken for the purpose of
improving programs and services
within the institution.
Banta, T. W.
Some Purposes of
Assessment
1. Students learn content
2. Students assess own strengths
3. Faculty improve instruction
4. Institutions improve programs/services
5. Institutions demonstrate accountability
Assessment of Learning
1. Faculty agreement on outcomes and
performance standards
2. Evidence in syllabi and assignments
that outcomes are taught
3. Collective faculty review of group
performance
4. Use of findings to improve instruction
and curriculum
Most Faculty Are Not Trained as
Teachers
Faculty Development
Can Help Instructors:
 Write clear objectives for student learning in
courses and curricula
 Individualize instruction using a variety of
methods and materials
 Ask questions that make students active
learners
 Develop assessment tools that test higher
order intellectual skills
Taxonomy of Educational
Objectives
(Bloom and Others, 1956)
Cognitive domain
categories
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Sample verbs for outcomes
Identifies, defines, describes
Explains, summarizes, classifies
Demonstrates, computes, solves
Differentiates, diagrams, estimates
Creates, formulates, revises
Criticizes, compares, concludes
Organizing for Assessment
Goal
Course Measure Findings Uses
Write
Portfolio
Speak
Speech
Think
Test
Find
Information
Project
Planning for Learning and
Assessment
1. What
2. How
3. How will
general
would you
you help
outcome
know it
students
are you
(the
learn it?
seeking?
outcome)
(in class
if you saw
or out of
it? (What
class)
will the
student
know or
be able to
do?)
4. How could
5. What are 6. What
you
the
improvemeasure
assessments
each of the
ment
might be
desired
findings?
based on
behaviors
assesslisted in #2?
ment
findings?
Faculty and Staff
Development

Focus faculty and student affairs
professionals on improving learning in and
outside class
 Attend conferences together
 Study literature on student learning
 Provide workshops on teaching and learning
 Provide resources (e.g., grants, summer
salary, release time)
Direct Measures of Learning
Assignments, exams, projects, papers
Indirect Measures
Questionnaires, inventories, interviews
- Did the course cover these objectives?
- How much did your knowledge increase?
- Did the teaching method(s) help you
learn?
- Did the assignments help you learn?
Select or Design
Assessment Methods
1. Match with goals
2. Use multiple methods
3. Combine direct and indirect measures
4. Combine qualitative and quantitative
measures
5. Consider pre - post design to assess
gains
6. Use built-in points of contact with
students
Outcomes Assessment
Requires Collaboration

In setting expected program outcomes

In developing sequence of learning
experiences (curriculum)

In choosing measures

In interpreting assessment findings

In making responsive improvements
Barriers to Collaboration
in the Academy
1
2
3
4
Graduate schools prepare
specialists
Departments hire specialists
Much of our scholarship is
conducted alone
Promotion and tenure favor
individual achievements -interdisciplinary work is harder to
evaluate
Change the Focus
from
TEACHING
to
LEARNING
Barr & Tagg
1995
Functions of the Work of the
Professoriate
 Scholarship
of Discovery
 Scholarship
of Integration
 Scholarship
of Application
 Scholarship
of Teaching
Ernest Boyer - 1990
Good assessment is
good research . . .





An important question
An approach to answer the
question
Data collection
Analysis
Report
-Gary R. Pike (2000)
Campus Interest in
Assessment
WHAT WORKS in….




increasing student retention?
general education?
use of technology in instruction?
curriculum in the major?
Measures of Critical
Thinking
1. Academic Profile (ETS)
2. Collegiate Assessment of Academic Proficiency
(ACT)
3. California CT Dispositions Inventory (Faciones)
4. California CT Skills Test (P. Facione)
5. Cornell CT Test (Ennis & Millman)
6. Tasks in CT (ETS)
7. Reflective Judgment Inventory (King & Kitchener)
8. Watson Glaser CT Appraisal (Psych Corp)
Collegiate Assessment of Academic
Proficiency
(CAAP from ACT)
Individual Modules
 Reading
 Writing
(MC and Essay)
 Mathematics
 Science Reasoning
 Critical Thinking
COLLEGE BASE
Competences
Interpretive
Reasoning
S
U
B
J
E
C
T
S
English
Strategic
Reasoning
English
Subject Clusters
Adaptive
Reasoning
Interpretive
Reasoning
Math
Writing Skills
Science
Social
Studies
Reading/Literature Skills
Reading Critically
Reading Analytically
Understanding Literature
Are Standardized Tests
the Answer?
 Not
available in many fields
 Do not measure all that is taught
 Usually assess knowledge, not
performance
 May be standardized on
unrepresentative norm group
 Provide few, if any, subscores
 Do not indicate why scores are low
Start with Measures You
Have
 Assignments
in courses
 Course exams
 Work performance
 Records of progress through
the curriculum
Primary Trait Scoring
Assigns scores to attributes (traits) of a task
STEPS

Identify traits necessary for success in
assignment
 Compose scale or rubric giving clear
definition to each point
 Grade using the rubric
Can Develop a Research
Paper
1.
2.
3.
4.
5.
6.
Narrows and defines
topic
Produces
bibliography
Develops outline
Produces first draft
Produces final draft
Presents oral
defense
Outstanding
Accept- Unaccept
able
-able












Bibliography
Outstanding – References current,
appropriately cited, representative,
relevant
Acceptable – References mostly current,
few citation errors, coverage adequate,
mostly relevant
Unacceptable – No references or
containing many errors in citation
format, inadequate coverage or
irrelevant
Mapping Course Outcomes
to Program Outcomes
Outcomes
Course 1
Course 2
Course 3
1





2
3



4
5

6

7


Sophomore Competence in Mathematics
(Multiple choice responses & supporting
work)
Score
3
2
1
0
Criterion
Clear conceptual understanding, consistent
notation, logical formulation, complete
solution
Adequate understanding, careless errors,
some logic missing, incomplete solution
Inadequate understanding, procedural errors,
logical steps missing, poor or no response
Problem not attempted or conceptual
understanding totally lacking
Ball State University
Journal Evaluation
1.
2.
3.
Entries accurately and
vividly record objective
observations of site
experiences (events,
people, actions, setting)
Entries convincingly record
subjective responses to site
experience (thoughts,
emotions, values,
judgments)
Entries effectively analyze/
evaluate your experiences
(find insights, patterns,
meaning, causes, effects)
Well
done
Satisfactory
Unsatisfactory
Assessment in Sociology and
Anthropology
Focus groups of graduating students
 Given
a scenario appropriate to the discipline,
a faculty facilitator asks questions related to
outcomes faculty have identified in 3 areas:
concepts, theory, methods.
 2 faculty observers use 0-3 scale to rate each
student on each question
 GROUP scores are discussed by all faculty

Murphy & Goreham
North Dakota State University
Methods of Assessment

Paper and pencil tests
 Individual or group projects
 Portfolios
 Observation of practice
 Observation of simulated practice
 Analysis of case studies
 Attitude or belief inventories
 Interviews and focus groups
 Surveys
Assessing Student Growth
The Portfolio - Some Examples of Content







Course assignments
Research papers
Materials from group projects
Artistic productions
Self-reflective essays (self-assessment)
Correspondence
Taped presentations
Student Electronic Portfolio
 Students
take responsibility for
demonstrating core skills
 Unique individual skills and
achievements can be emphasized
 Multi-media opportunities extend
possibilities
 Metacognitive thinking is enhanced
through reflection on contents
- Sharon J. Hamilton
IUPUI
PRINCIPLES OF UNDERGRADUATE
LEARNING (PULs)
1.
2.
3.
4.
5.
6.
Core communication and quantitative
skills
Critical thinking
Integration and application of
knowledge
Intellectual depth, breadth, and
adaptiveness
Understanding society and culture
Values and ethics
Approved by IUPUI Faculty Council
May 1998
ePort Goals
 Promote
clearer understanding among
faculty and students of how the
curriculum supports increasing mastery
of the PULs.
 Contribute
to assessment of student
learning of the PULs at multiple levels of
aggregation.
 Support
student engagement with the
PULs over their entire undergraduate
experience.
PULs Levels of Competence

Introductory: What all undergraduate students
should know and be able to do within the first
26 credit hours.

Intermediate: What all undergraduate
students should know and be able to do within
the first 56 credit hours.

Advanced: What all baccalaureate recipients
should know and be able to do in their major or
profession or academic program.

Experiential: Connecting curricular and cocurricular learning
Background Knowledge Probe
(Pre-Test – Indirect Measure)
1.
ARCHAEOLOGY
A. Have never heard of this
B. Have heard of it, but don’t really know
what it means
C. Have some idea what it means, but not
too clear
D. Have a clear idea what this means and
can explain it
- Classroom Assessment
Angelo and Cross
Fast Feedback
(at end of every class)




Most important thing learned
Muddiest point
Helpfulness of advance reading
assignments for day’s work in class
Suggestions for improving class /
assignments
Bateman & Roberts
Graduate School of Business
University of Chicago
Student Suggestions for
Improvement





Install a portable microphone
Increase type size on
transparencies
Leave lights on when using
projector
Don’t cover assigned reading in
detail
Provide more examples in class
Outcomes of Classroom
Assessment
1. Classes more student centered
2. Focus on learning, not teaching
3. Students take ownership for learning
4. Faculty look for data to make
improvements continuously
- Jann Freed
Assessment Update - 1999
In a Comprehensive
Assessment Program...
INVOLVE
 Students
 Faculty
 Student Affairs
Staff
 Administrators
 Graduates
 Employers
Involve Students
1. Set learning expectations in recruiting
2. Communicate learning outcomes in
orientation
3. Involve student leaders in promoting learning
4. Involve students in evaluating
courses/curricula
5. Let students know their recommendations
are used.
Student Advisory Council at
Montevallo
A way to provide continuous student
assessment
Student Recommendations
1
2
3
4
5
Develop a statement of expected
ethical behaviors for students
Add a second research course with
lab
Increase comparative psychology
Add terminals for statistics lab
Increase opportunities for research,
writing, and speaking
Guidance from Alumni

Alumni surveys emphasized that graduates
valued skills in writing, speaking, working
collaboratively, and information literacy

Now the Faculty Senate’s General Education
Committee has developed 5 learning
elements, at least 3 of which must be
integrated in any course approved for general
education
-Michael Dooris
Penn State University
Junior Course in Professional
Communication
Teams devise promotional event for nonprofit agency
Campaign theme and rationale
 Logo and other visual materials
 Event outline

Faculty, agency representative, peers
evaluate the plan
- A. J. Johnson
Alverno College
Assessment in Fine Arts
—
Portfolio review and/or audition
for every student every semester
by
Panel of faculty, students, community
representatives, staff or faculty outside fine
arts
Results - Students creativity,
conceptualization, technique have
improved
Alec Testa
— Eastern New Mexico University
—
Involving Employers
Combination of survey and focus groups
for employers of business graduates




Identified skills, knowledge, personality attributes
sought by employers
Encouraged faculty to make curriculum changes
Motivated student to develop needed skills
Strengthened ties among faculty, students,
employers
- Kretovics & McCambridge
Colorado State University
Colorado State University
College of Business
Curriculum changes based on employer
suggestions:

1 credit added to Business Communications for team
training and more presentations
 Ethics & social responsibility now discussed in intro
courses
 New Intro to Business course emphasizing career
decision-making
 More teamwork, oral & written communication,
problem-solving in Management survey courses
- Kretovics & McCambridge
Authentic Assessment
At
Southern Illinois University - Edwardsville

Business - Case Study Analysis with Memo
 Education - Professional Portfolio
 Psychology - Poster on Research Project
 Engineering - Senior Design Project
 Nursing - Plan of Care for Patient
Responses to Assessment
At
Southern Illinois University - Edwardsville
Business - More case studies and research
 Education - More practice in classroom
management
 Psychology - Curriculum change in statistics
 Engineering - More practice in writing and
speaking
 Nursing - Simulation lab with computerized
patients
Virginia Commonwealth
University
1. First-year English students wrote 1
response / week for 15 weeks
2. Faculty-student affairs teams read
essays
3. Sample of writers interviewed 18
months later
Responses to findings:
Central advising center, new advising
handbook, multicultural workshops
2004-2005
The Year of Accountability
. . . Peter Ewell
Assessment Update
September 2005
National Center for Public Policy
in
Higher Education
Measuring Up 2004-5-State Pilot:
•National
Adult Literacy Survey
•Licensure and grad admissions scores
•College Learning Assessment
(or ACT Work Keys)
3 Themes Common to 4 Reports
1.
2.
3.
Accountability is now about the
outcomes of student learning
Accountability is more focused on
serving the public interest
Accountability now emphasizes public
disclosure
. . . Peter Ewell
Assessment Update
September 2005
External Accountability Measures
are Summative rather than
Formative
and Lead to
Compliance
or
Creative LOCAL Initiatives
The Future

Need for evidence of accountability will
increase
 More faculty will recognize benefits of
assessment
 More electronic assessment methods will be
developed
 More sharing of assessment methods will
take place
 Faculty will learn more about learning and
student learning will improve