Transcript No Slide Title

U.S. Survey Reveals
Small Impact of
Outcomes Assessment on Student Learning
Presented at the
31st EAIR Forum
Vilnius, Lithuania
24 August, 2009
By
Trudy W. Banta
Professor of Higher Education
and
Senior Advisor to the Chancellor for
Academic Planning and Evaluation
Indiana University-Purdue University Indianapolis
355 N. Lansing St., AO 140
Indianapolis, Indiana 46202-2896
tbanta@ iupui.edu
http://www.planning.iupui.edu
Designing Effective Assessment:
Principles & Profiles of Good Practice
Trudy W. Banta
Elizabeth A. Jones
Karen E. Black
Jossey-Bass (Wiley) 2009
© TWBANTA-IUPUI
Profiles
Invited over 1000
Received 146
Selected 49 for use in full
Categorized all 146 and published
Web sites
© TWBANTA-IUPUI
Outline for Profiles
Background and Purpose
Methods over ? Years
Resources Required
Findings
Use of Findings
Impact of Using Findings
Success Factors
Web sites
© TWBANTA-IUPUI
Plan
Improve
Culture of
Evidence
Implement
Evaluate
© TWBANTA-IUPUI
~ Organization ~
of
Principles & Profiles
Planning
Implementing
Improving & Sustaining
- Building a Scholarship of Assessment
Banta & Associates
Jossey-Bass 2002
© TWBANTA-IUPUI
Types of Institutions Represented





Doctoral and Research Universities
Private
7
Public
16………..47%
Master’s
13………..26%
Baccalaureate
7…..……14%
Associate
5………..10%
ACE
1……..…..2%
© TWBANTA-IUPUI
Planning Profiles
Brigham Young University
Campus Wiki for degree learning outcomes
USMA at West Point
Interdisciplinary teams assess 10
mission-related goals for learners
Kennesaw State University
2008 CHEA Award for linking assessment
with planning, program review, faculty
development
© TWBANTA-IUPUI
USMA @ West Point
6 Developmental Domains
1.
2.
3.
4.
5.
6.
Intellectual
Physical
Military
Social
Moral-ethical
Human spirit
© TWBANTA-IUPUI
USMA @ West Point
Intellectual Domain
10 Goals (write, speak, think; engineering,
math, info tech)
A. Stated learner outcomes
1. Standards
a. Rubrics developed by faculty
© TWBANTA-IUPUI
USMA @ West Point
Interdisciplinary Goal Teams
use
•
•
•
•
•
Curriculum-embedded direct measures
of learning
Student surveys (fr., sr.)
Graduate survey (3 years after)
Employer surveys
Employer focus groups
© TWBANTA-IUPUI
USMA @ West Point
Use of Assessment Findings
Review of core curriculum
Changes in warranted areas:
History
English
Engineering
Information Technology
© TWBANTA-IUPUI
As a catalyst
for outcomes assessment
accreditation is powerful
regional
disciplinary
© TWBANTA-IUPUI
Engaging faculty
in taking a second look
at student work
for outcomes assessment purposes
may require extra pay
© TWBANTA-IUPUI
Standardized tests
of generic skills
(e.g., writing, critical thinking)
used by just 8%
always supplemented
© TWBANTA-IUPUI
At the University of Tennessee
CAAP
Academic Profile (now MAPP)
COMP (like CLA and withdrawn
by 1990)
College BASE
© TWBANTA-IUPUI
In TN We Learned
1)
2)
3)
4)
No test measured 30% of gen ed skills
Tests of generic skills measure
primarily prior learning
Reliability of value added = .1
Test scores give few clues to guide
improvement actions
© TWBANTA-IUPUI
An Inconvenient Truth
.9 = the correlation between SAT
and CLA scores of institutions
thus
81% of the variance in institutions’
scores is due to prior learning
© TWBANTA-IUPUI
More use of RUBRICS
locally developed
VALUE from AAC&U
© TWBANTA-IUPUI
Better Ways to Demonstrate
Accountability
Measures of Learning
Standardized tests in major fields
Internship performance
Senior projects
Electronic portfolios
External examiners
© TWBANTA-IUPUI
Student Electronic Portfolio
•
•
•
•
Students take responsibility for
demonstrating core skills
Unique individual skills and
achievements can be emphasized
Multi-media opportunities extend
possibilities
Metacognitive thinking is enhanced
through reflection on contents
- Sharon J. Hamilton
IUPUI
© TWBANTA-IUPUI
Accountability Report
 85%
achieve Outstanding ratings in
writing as defined . . .
 78%
are Outstanding in applying
knowledge and skills in internships
 75%
are Outstanding in delivering an
oral presentation
© TWBANTA-IUPUI
For External Credibility

Collaborate on rubrics

Use employers as examiners

Conduct process audits
© TWBANTA-IUPUI
Build Assessment into Valued
Processes
1.
2.
3.
4.
5.
6.
7.
8.
9.
Assessment of learning
Curriculum review and revision
Survey research
Program review
Scholarship of Teaching & Learning
Evaluation of initiatives
Faculty development
Promotion & tenure
Rewards and recognition
© TWBANTA-IUPUI
Impact of Using Assessment Findings
NOT
measured in
student learning gains
?WHY?
© TWBANTA-IUPUI
Group Assessment Has Failed to
Demonstrate Institutional Accountability
•
•
•
•
•
42% less than 2 years old
Focus on improvement at unit level
Rare aggregation of data centrally
Too few faculty involved
HE scholars focused on K-12 assessment
© TWBANTA-IUPUI