Transcript league tables as policy instruments: the political economy of accountability in tertiary

league tables as policy
instruments:
the political economy of
accountability in tertiary
education
Jamil Salmi and Alenoush
Saroyan
CIEP, 18-20 June 2006
Lexus-Nexus index on
rankings
19
9600
198185
198690
199195
200106
Asia/Pacific:
0
0
6
18
27
Middle East/Africa:
0
0
0
1
9
Europe:
0
0
2
24
68
N. & S. America:
0
3
17
23
68
The rankings
business
A ranking of league tables
September 10, 2005
outline of the
presentation
• typology of rankings
• a world of controversies
• do rankings measure quality?
• policy implications
typology of rankings:
how is it done?
• statistical indicators
– produced by institutions
– publicly available
• survey of “stakeholders”
– employers
– professors
– students
• combination of both
typology of rankings:
what does it apply to?
• entire institution or specific
program
• gives a global score or
measures several dimensions
separately
• research or teaching / learning
cluster of indicators in
league tables as
measures of quality
•
•
•
•
•
•
•
beginning characteristics
learning inputs- staff
learning inputs- resources
learning outputs
final outcomes
research
reputation
who prepares the
ranking?
• A = government agency (Ministry of Higher
Education, Higher Education Commission,
University Grants Council, etc.)
• B = independent organization / professional
association / university
• C = newspaper / magazine / media
• D = accreditation agency
• I = International ranking (IA, IB, IC and
ID linking the international dimension to
the type of institution conducting the
ranking)
ranking systems in 2006
Region
Eastern Europe and Central
Asia
East Asia and Pacific
Latin America and the
Caribbean
National and International Ranking
System
Poland (C), Slovakia (B), Russia (B), Ukraine (B)
Australia (B), China (B, IB), Hong Kong (C),
Japan (C), New Zealand (A), Thailand (A)
Argentina (D)
Middle East and North
Africa
North America
Canada (C), United States (C)
South Asia
India (D), Pakistan (A)
Sub-Saharan Africa
Nigeria (A)
Western Europe
Germany (B/C), Italy (C), Netherlands (A),
Spain (B), United Kingdom (A, B, IC)
outline of the
presentation
• typology of rankings
• a world of controversies
a thin line between
love and hate
a thin line between
love and hate
• disagreement with principle
(“Anglo-Saxon” exercise)
a thin line between
love and hate
• disagreement with principle
(“Anglo-Saxon” exercise)
• criticism of methodology
a thin line between
love and hate
• disagreement with principle
(“Anglo-Saxon” exercise)
• criticism of methodology
• boycotts
boycotts
boycotts
• Asiaweek
• US News and World Report
• McLeans
a thin line between
love and hate
• disagreement with principle
(“Anglo-Saxon” exercise)
• criticism of methodology
• boycotts (Asiaweek, USA)
• court actions (New Zealand,
Holland)
outline of the
presentation
• typology of rankings
• a world of controversies
• do rankings measure quality?
But do they measure
quality?
– quality a moving target;
Illusive definition
– mutlidimensional construct;
unidimensional score
(subjective weights to
indicators)
– theoretical justification of
measures and methodology
– empirical support for
indicators
other shortcomings
• methodological flaws
– lesser emphasis on outcome
indicators
• few meaningful indicators to
assess teaching quality
– one size fits all: general
disregard for non-research
universities and nonuniversity institutions

encourages universities
to adjust method of data
reporting
and the winner is …
the Anglo-Saxon factor
the “English” factor in the 2005 rankings
SJTU
THES
68 of top 100
60 out of top 100
US
UK
Canada
Australia
N.Z.
HK
Singapore
India
53
11
4
51
31
3
12
1
3
2
2
outline of the
presentation
• typology of rankings
• a world of controversies
• do rankings measure quality?
• policy implications
usefulness of
rankings?
• for the Government?
• for the institutions?
• for the public?
government use of
rankings
• Pakistan case
government use of
rankings
• Pakistan case
– promoting a culture of accurate
and transparent information
government use of
rankings
• Pakistan case
– promoting a culture of accurate
and transparent information
– promoting a culture of quality
from the viewpoint
of institutions
• sensitive to factors that affect
their rankings (benchmarking)
• goal setting for strategic
planning purposes
• forming strategic alliances
applying public
pressure
• Provão
applying public
pressure
• Provão
• France
applying public
pressure
• Provão
• France
• Colombia
conclusion:
divisive or helpful?
conclusion:
divisive or helpful?
conclusion:
divisive or helpful?
• rankings are here to stay
• useful for prospective students
• useful in the absence of an
established evaluation and/or
accreditation system
• useful for benchmarking, goalsetting and self-improvement
purposes
• useful to conduct a healthy debate
on issues and challenges
• useful to promote a culture of
accountability
principles of an
appropriate
ranking instrument
• compare similar institutions
• better to focus on program than on
entire institution
• better to rank by indicator than
wholesale (Germany – Pakistan)
• better to focus on results rather
than inputs (labor market
outcomes, publications, patents)
• better if used for self-improvement
purposes
• better to advertise results publicly
than to keep them secret