Transcript No Slide Title

Can Institutions Really Be Compared
Using Standardised Tests?
Presented at the
2008 Meeting of the
European Association for Institutional Research
Copenhagen, Denmark
August 2008
By
Trudy W. Banta
Professor of Higher Education
and
Senior Advisor to the Chancellor for
Academic Planning and Evaluation
Indiana University-Purdue University Indianapolis
355 N. Lansing St., AO 140
Indianapolis, Indiana 46202-2896
tbanta@ iupui.edu
http://www.planning.iupui.edu
Standardized tests
of
generic skills
cannot provide valid
comparisons
of institutional quality.
© TWBANTA-IUPUI
Standardized tests do have a place
in assessing learning.
Higher education must be accountable
© TWBANTA-IUPUI
My History
Educational psychology
Program evaluation & measurement
Performance funding in Tennessee
1990 USDOE effort to build a national
test
1992 Initiated evidence-based culture
at IUPUI
© TWBANTA-IUPUI
Group Assessment Has Failed to
Demonstrate Institutional Accountability
•
Focus on improvement at unit level
•
Rare aggregation of data centrally
•
Too few faculty involved
•
HE scholars focused on K-12 assessment
© TWBANTA-IUPUI
Now We Have
the
Press to Assess with a Test
© TWBANTA-IUPUI
2006
Commission on the Future of
Higher Education
 We
need a simple way to compare
institutions
 The
results of student learning
assessment, including value added
measurements (showing skill
improvement over time) should be . . .
reported in the aggregate publicly.
© TWBANTA-IUPUI
OECD’s AHELO
for
10 HEIs from 3-4 countries
1.
2.
3.
4.
Generic skills (CLA)
Disciplines (Engineering and
Economics)
Value added
Contextual information indicators
© TWBANTA-IUPUI
Two-Pronged Strategy
in Washington
1.
Pressure accreditors
2.
Voluntary System of Accountability
- NASULGC
- AASCU
© TWBANTA-IUPUI
Voluntary System of Accountability
Report Scores in
critical thinking, written communication,
analytic reasoning
using
CAAP
MAPP
CLA
© TWBANTA-IUPUI
Collegiate Assessment of Academic
Proficiency
(6 independent modules)
•
•
•
•
•
•
Reading
Writing Skills
Writing Essay
Mathematics
Science
Critical Thinking
© TWBANTA-IUPUI
Measure of Academic
Proficiency & Progress
•
•
•
•
•
•
•
•
Humanities
Social Sciences
Natural Sciences
College – Level Reading
College – Level Writing
Critical Thinking
Mathematics
Total Score
© TWBANTA-IUPUI
Collegiate Learning Assessment
Performance and Analytic Writing Tasks
measuring
•
•
•
•
Critical Thinking
Analytic Reasoning
Written Communication
Problem Solving
© TWBANTA-IUPUI
Standardized tests
CAN
initiate conversation
© TWBANTA-IUPUI
Advantages
of standardized tests of generic skills
promise of increased reliability & validity
norms for comparison
© TWBANTA-IUPUI
Limitations
of standardized tests of generic skills
cannot cover all a student knows
narrow coverage, need to supplement
difficult to motivate students to take
them!
© TWBANTA-IUPUI
TN = Most Prescriptive
(5.45% of Budget for Instruction)
1.
2.
3.
4.
5.
Accredit all accreditable programs
Test all seniors in general education
Test seniors in 20% of majors
Give an alumni survey
Demonstrate use of data to improve
© TWBANTA-IUPUI
(25)
(25)
(20)
(15)
(15)
___
100
At the University of Tennessee
CAAP
Academic Profile (now MAPP)
COMP (like CLA and withdrawn
by 1990)
College BASE
© TWBANTA-IUPUI
In TN We Learned
1)
2)
3)
4)
No test measured 30% of gen ed skills
Tests of generic skills measure
primarily prior learning
Reliability of value added = .1
Test scores give few clues to guide
improvement actions
© TWBANTA-IUPUI
VSA Instrument
Reviews Revealed
Available tests of generic skills –
1) Lack test – retest reliability
2) Lack content validation
(except with ACT/SAT)
3) Have no evidence that score gain is
due to college experience
© TWBANTA-IUPUI
An Inconvenient Truth
.9 = the correlation between SAT
and CLA scores of institutions
thus
81% of the variance in institutions’
scores is due to prior learning
© TWBANTA-IUPUI
How Much of the Variance in Senior
Scores is Due to College Impact?
• Student motivation to attend that
institution (mission differences)
• Student mix based on
• age, gender
• socioeconomic status
• race/ethnicity
• transfer status
• college major
© TWBANTA-IUPUI
How Much of the Variance in Senior
Scores is Due to College Impact?
(continued)
•
Student motivation to do well
• Sampling error
• Measurement error
• Test anxiety
 College effects
______
19 %
© TWBANTA-IUPUI
Threats to Conclusions
Based on Test Scores
1.
2.
3.
4.
5.
6.
Measurement error
Sampling error
Different tests yield different results
Different ways of presenting results
Test bias
Pressure to raise scores
- Daniel Koretz
“Measuring Up”
Harvard U. Press - 2008
© TWBANTA-IUPUI
Student Motivation
• Samples of students are being tested
• Extrinsic motivators (cash, prizes) are
used
We have learned:
• Only a requirement and intrinsic
motivation will bring seniors in to do
their best
© TWBANTA-IUPUI
Concerns About Value Added
•
•
•
•
•
Student attrition
Proportion of transfer students
Different methods of calculating
Unreliability
Confounding effects of maturation
© TWBANTA-IUPUI
Recent University of Texas
Experience
30 – 40% of seniors at flagships earn
highest CLA score (ceiling effect)
flagship campuses have lowest value
added scores
© TWBANTA-IUPUI
Other Recent Critiques
Cross-sectional design incorporates
too many confounding variables
~ OR ~
Freshman scores generally are NOT
randomly drawn from same
population as senior scores
© TWBANTA-IUPUI
Word from Measurement Experts
Given the complexity of
educational settings, we may never be
satisfied that value added models can
be used to appropriately partition the
causal effects of teacher, school, and
student on measured changes in
standardized test scores.
- Henry Braun & Howard Wainer
Handbook of Statistics, Vol. 26: Psychometrics
Elsevier 2007
© TWBANTA-IUPUI
Better Ways to Demonstrate
Accountability
1. Performance Indicators
Access, social mobility
Diversity
Workforce development
Economic development
Engaging student experience
© TWBANTA-IUPUI
Better Ways to Demonstrate
Accountability
2. Measures of Learning
Standardized tests in major fields
Internship performance
Senior projects
Electronic portfolios
External examiners
© TWBANTA-IUPUI
AAC&U Essential Learning Outcomes
Knowledge of human cultures
and physical and natural world
Intellectual and practical skills
(writing, thinking, team work)
Personal and social responsibility
Integrative learning
© TWBANTA-IUPUI
Expensive Alternatives ?
 Agreement
on outcomes
 Agreement
on standards of
achievement
 Peer
review
© TWBANTA-IUPUI
Accompanying Benefits
•
•
•
•
Teach faculty how to develop better
classroom assessments
Involve faculty in using results to
improve learning
More collaboration across disciplines
and institutions
Closer ties with community
© TWBANTA-IUPUI