Transcript Measuring Learning—Using Results Richard J. Shavelson Stanford University Inside Higher Ed Audio Conference March 31.

Measuring Learning—Using
Results
Richard J. Shavelson
Stanford University
Inside Higher Ed Audio Conference
March 31. 2010 @ Noon EST
[email protected]
Overview
 Assessment landscape: A hot issue that
doesn’t seem to go away
 Differing demands for measuring
learning and accounting for outcomes:
Conflict of cultures
 Differing purposes for learning
assessment: Tools being used
 Using assessment results: “Culture of
Evidence” and “Learning Organization”
2
Landscape: Close Encounters!
 A NAEP for higher education—NSF
question in early 1990s
 One (multiple-choice) assessment fits
all—response of small liberal arts
college to North Central Accreditation
 Report card for all colleges—NY State
Education Department
3
Landscape: Spellings’ Commission
on the Future of Higher Education
 Recognized stature
of U.S. higher
education... BUT
 “Higher Education
must change from a
system primarily
based on reputation
to one based on
performance”
 Urged “creation of
robust culture of
accountability and
transparency”
 Student achievement
inextricably
connected to
institutional success
 Achievement
measured by value
added
 Relative
effectiveness of
campuses
published in
league tables
Landscape: Higher-Education
Professional Organization’s Assessment
 Identify seven areas for improvement (affordability,
learning, secondary education assistance, increased
accountability, internationalize student experience,
increase opportunities life-long learning and
workforce training)
 Agree that there is a need to:
 Improve learning
 Increase accountability for educational outcomes
 Most extreme response:
 Voluntary System of Accountability (VSA) to:



Demonstrate accountability and stewardship to the public
Measure educational outcomes to identify effective educational
practices
Assemble information that is accessible, understandable, and
comparable
 College Portrait system of indicators
 High Stakes: Publish comparative results of student
achievement and value added
Landscape: Academics’ Reaction
 One size does not fit all—Commission failed to
recognize diversity of higher education institutions
 Outcomes vary by academic major
 Sole focus on “cognitive” outcomes (declarative
and procedural knowledge) too limiting—need
individual and social responsibility outcomes
 Intrudes on academic culture where faculty
responsible for curriculum, teaching and assessment
 Higher education system too complex for simple
quantitative measurement
 “Horse-race” comparisons of colleges and
universities at best misleading and at worse have
perverse effect on teaching and learning
Landscape: Toward A Common
Ground
 Rejection of standardized assessment of
learning and accountability in higher
education:
 Ignores good and bad assessment
practices
 Is undemocratic saying only a few superior
students in a narrowly defined discipline can
succeed instead of identifying, modeling, and
making a set of intellectual skills and
practices accessible to all students
 Denies higher education’s responsibility
to provide a coherent curriculum
 Is wrong minded when argue that
campuses share no common standards
Landscape: Common Ground For
Higher Education—Generic and Majors
 Think critically
 Reason analytically
 Connect apparently disparate pieces of
information
 Explore others’ knowledge claims
 Justify own knowledge claims with
evidence and examples
 Take another’s perspective and act
accordingly
Landscape: Macro View of
Motivation
“Massification”
Internationalization
Diversification of Missions
Diversity of Students
Cost
Dilution of Curriculum
Lack of transparency
Cultural Conflict between Policy Makers + Public
v. Higher Education
 Distrust of Diverse Student Body—”Standards”








9
Landscape: Beyond The US
Accountability
United States
Accreditation
Performance
Reporting
Assessment
of Learning
Elsewhere
College
Rankings
Int’l
Rankings
Accreditation
Academic
Audit
Research
Assessment
Exercise
OECD
AHELO
Academic
Assessment
Assessment
of Learning
10
Landscape: President Obama?
 Community colleges… community
colleges… etc.
 Assistant Secretary Kanter supports
AHELO in concept (no $)
 Higher education (assessment and
accountability) policy?????
11
Cultural Conflict Vs. Transparency
Accountability Triangle
Accountable To Whom?
Academy
Policy
Makers
Clients
12
LEARNING ASSESSMENT
PURPOSES
 Formative function—

Improving teaching and learning




Faculty constructed
Faculty buy in
Linked to curriculum
Assuring the public


Institution is being accountable
Trust us
 Summative function—

Assuring the public



Transparency—Issue is how given conflict of cultures and lack of
trust?
External oversight credibility given broken contract
Improving teaching and learning



Signaling function
Benchmark performance—how good is good enough?
Feedback for improvement of teaching and learning
13
Learning Assessment Examples
 Formative focus—Summative
background:
 Portfolios
 Capstone courses and projects
 Performance assessments
 Summative focus—Formative
background:
 Pencil & paper measures of critical thinking
etc. in general and in subject areas (e.g.,
CAAP, MAPP—now ETS® Proficiency Profile)
 Collegiate Learning Assessment
14
Using Assessment Information:
Not So Easily Done
 Case studies of six “exemplary” campus
learning assessment systems
 Major findings—vastly different:
 Visions of learning assessment systems
 Levels of maturity
 Levels of capacity (assessment leaders
essential if not lead top down)
 Levels of administrator and faculty buy in
 Levels of student engagement and feedback
 Levels of impact
15
Assessment Of Learning
Outcomes
Flourishing/
Thriving
Knowledge &
Reasoning in
Disciplines
Broad Abilities
 Critical Thinking
 Analytic Reasoning
 Problem Solving
 Communication
Transformational
Learning
Responsibility




Identity
Resilience
Emotional Competence
Perspective Taking &
Acting
College
Environmental
Ecology
 Linked formative
and summative
assessment of
learning
 Strong internal
learning assessment
capacity
 External learning
assessment for
signaling and
benchmarking (but
not external
reporting)
16
A Vision Of Learning
Assessment & Accountability
 Culture of evidence—
evidence fed back up and
down system from
president to students
 Learning organization—
organizational structures in
place to:





Feedback information
Assess gap between goals
and current performance
“Experiments” to try
alternative improvements
Repeat cycle gauging
improvement
Appropriate incentives
17
17
Thank You!
18