Transcript Democracy Theory and Practice A cross-disciplinary literature classification project

Democracy Theory and
Practice
A cross-disciplinary
literature classification project
Donna Schenck-Hamlin, Bill Schenck-Hamlin,
Elizabeth Westmoreland, Kevin Keatley
Institute for Civic Discourse and Democracy
Begun as information support to Institute for Civic Discourse and Democracy
GOAL: Aggregate useful scholarship to promote interdisciplinary democracy research
What type of democracy? W of S
title phrases 1945 - 2007
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Social democracy
American democracy
Liberal democracy
Industrial democracy
Deliberative democracy
Direct democracy
Athenian democracy
Economic democracy
Local democracy
Modern democracy
Participatory democracy
Socialist democracy
Chinese democracy
Political democracy
German democracy
Cosmopolitan democracy
Popular democracy
African democracy
Western democracy
Corporate democracy
969
451
274
197
162
161
137
129
110
84
74
61
60
39
29
27
24
23
22
17
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Contemporary democracy
European democracy
Real democracy
Canadian democracy
Environmental democracy
Technological democracy
Militant democracy
Nuclear democracy
Proletarian democracy
Military democracy
Green democracy
Basic democracy
Concordant democracy
Utilitarian democracy
13
11
11
9
7
7
7
4
3
3
2
2
1
1
Developed a database of over 21,000 references for ICDD’s website from various
sources: other databases, websites, individual contributions.
QUESTION:
Can we develop a set of useful descriptors covering most democracy
research that would optimize searching?
Objectives of the Project
1. Improve database searching through
value-added subject descriptors
2. Test the reliability and validity of the
descriptors coded by 4 persons
3. Map the emphases of democracy
scholarship through a quarter-century,
1980-2005
Introduced 3 Levels of Descriptors
• CATEGORIES = areas of democracy
scholarship sought by ICDD researchers
• TOPICS = public issues associated with
democracy in a given document
• DOMAINS = geographic level of interest
addressed in a given document
CATEGORY: “Politics and Government” – about current affairs in specific states
TOPIC: “Social Equity” –employing collective resources to promote societal welfare
DOMAIN: “National” – focus on specific nations
Initial Set of ICDD-Developed
Categories, Topics, and Domains
1 Deliberation Facilitation Methods Models
2 Politics and Government
3 Online/Internet Communication E-democracy
4 Citizen Participation Civic Engagement Citizenship
Enfranchisement
5 Democracy Education
6 Evaluation
7 Media and Democracy
8 Political/Philosophical Theory
9 Public Opinion
11 Social Influences
12 Economic Influences
13 Organizational Democracy
14 History
10 Undetermined 1 – Does not apply to democracy
15 Undetermined 2 – Need more info. to determine
16 Undetermined 3 – Fits none of our descriptors
17 Arts and Letters
100 Environment
110 Technology
120 Science
130 Health
140 Foreign Relations
150 Poverty
160 Conflict Resolution Mediation
170 Pluralism Identity Multiculturalism Gender Race
Relations
180 Development
190 Policy
200 Planning
210 Agriculture
220 Security Safety
230 Religion
240 Social Equity
1100 Community domain
1110 Regional domain
1120 State domain
1130 National domain
1140 International domain
1150 Global domain
1160 Rural domain
1170 Urban domain
Methodology
1. Downloaded records from Web of Science*
into Refworks title = “*Democracy*” 1980-2005
= 18,834 but w/o redundant book review titles
= 7,276
2. Content Variables = 40 coder-applied
descriptors: 17 democracy categories + 15
topics + 8 domains
3. Sampling Procedure – stratified sample,
stratification variable = “year”
4. Sampling Fraction – 30% or 2,170 titles
* Over 10,000 international scholarly journals indexed; humanities, social
sciences, sciences
Sampling Fraction – compared the distribution
of selected descriptors across coders
between two halves of two randomly
selected years. If the two pairs had similar
distributions the sample size was
sufficiently large. 30% was determined to
be an adequate sample fraction.
Sampling Procedure – Randomly selected
records representing 30% of a given year’s
citations w/ “Democracy” in the title
Checking the reliability of our
classification system
• Answers the question: How likely is it that
a team of catalogers will use the same
descriptor (e.g. “history”) for a given
document (e.g. “Athenian Democracy”)?
• If inter-coder reliability scores are low, that
suggests that
– coders don’t have a shared understanding of
the meaning of the descriptor “history” or
– don’t agree on its application to some titles
2 reliability scores are used, but
each have a problem
1. RCA or % Agreement (agreements/total titles
reviewed)
Too liberal: If most of the time the descriptor
“history” was rarely applied by either coder in
the title set (they agreed that it got a “0” for
nearly every title they reviewed), results are
artificially high. What do you really know about
their shared understanding of the term “history”
if it was rarely applied by them?
2. Kappa score (agreement is weighted to
account for the quantity of non-selected
variables)
Too conservative: If there was only a
small number of titles out of 100 selected
by the two coders as “history”, say 8, and
they only agreed on 4 titles out of the 8,
you get a very low score, despite the
coders “agreement” that 90 titles had
nothing to do with history.
Improving Reliability
• Taking a conservative approach,.4 kappa
is “adequate”, but >.59 kappa is excellent.
• Catalogers consult with one another on
disagreements, visit the documents again,
then fine-tune the descriptor definition or
the descriptor itself.
• Over time, scores can improve as both the
coders and the schema itself (the terms
and their definitions) improve.
INITIAL INTER-CODER RELIABILITY SCORES FOR CATEGORIES
“Category” (a primary category of democracy literature)
RCA
Kappa
Politics and Government
.79
.52
Political/Philosophical Theory
.87
.57
History
.90
.59
Social Influences – needs work
.87
.18
Economic Influences
.92
.46
Citizen Participation, Civic Engagement, Citizenship,
Enfranchisement – needs work
.91
.36
Organizational Democracy
.96
.62
Democracy and Education
.98
.79
Arts and Letters
.98
.64
Deliberation, Facilitation, Methods, Models
.98
.52
Public Opinion
.98
.52
Media and Democracy
.99
.76
Online/Internet Communication, E-democracy
.99
.81
INITIAL INTER-CODER RELIABILITY SCORES FOR TOPICS
“Topic” (a public issue addressed in relation to democracy)
RCA
Kappa
Pluralism, Identity, Multiculturalism, Gender, Race Relations
.94
.55
Social Equity – needs work
.93
.39
Foreign Relations
.94
.45
Development
.94
.42
Security Safety
.95
.43
Policy – needs work
.95
.32
Religion
.99
.79
Science
.99
.69
Environment
.98
.40
Health
.99
.58
Conflict Resolution Mediation
.98
.43
Technology
.98
.40
Poverty – needs work
.98
.24
Planning – needs work
.98
.35
Agriculture
.99
0
INITIAL INTER-CODER RELIABILITY SCORES FOR DOMAINS
“Domain” (a geographical level of focus for the article)
RCA
Kappa
Community
.97
.51
Regional needs work
.99
.20
State needs work
.98
.33
National
.83
.61
International
.90
.55
Global
.99
.48
Rural
.99
0
Urban needs work
.99
.31
Results for Reliability and Validity
1. Coding Reliability – The majority of
descriptors achieved adequate to
excellent inter-coder reliability
2. Content Validity – 2% of articles could
not be classified based on our category
system (Undetermined 1,2,3)
To move on: – make adjustments to
descriptors that need more work, or
drop them entirely
Given a classification system
that was adequate in
describing the 30% sample of
democracy articles,
what trends are shown
over 25 years?
Number of titles in each category of
democracy research
Politics and Government
845
39%
Political/Philosophical Theory
513
24%
History
313
14%
Social Influences
261
12%
Economic Influences
238
11%
Citizen Participation, Civic Engagement, Citizenship,
Enfranchisement
228
10%
Organizational Democracy
151
7%
Democracy and Education
102
5%
Arts and Letters
75
3%
Deliberation, Facilitation, Methods, Models
58
3%
Public Opinion
51
2%
Media and Democracy
46
2%
Online/Internet Communication, E-democracy
41
2%
Politics and Government
• On the Prospects of Democracy in Russia
• Direct Democracy - Politicoeconomic
Lessons from Swiss Experience
• The Spanish Transition to Democracy
seen through the Spanish Database Isoc
• Presidential Democracy in America Toward the Homogenized Regime
Political/Philosophical Theory
• Does democracy reveal the voice of the
people? Four takes on Rousseau
• The crisis of modern democracy in
theoretical reflection.
• Can democracy promote the general
welfare?
• Democracy, merit, and presumptive virtue
Citizen Participation, etc.
• Democracy in our backyards - A study of
community involvement in administrative
decision making
• Participation, decentralization, and civil
society - Indigenous rights and democracy
in environmental planning
• Viewpoint: Implications of participatory
democracy for public land planning
Number of titles in each public issue
topic of democracy research
Pluralism, Identity, Multiculturalism, Gender, Race Relations
153
7%
Social Equity
147
7%
Foreign Relations
147
7%
Development
97
4%
Security Safety
80
4%
Policy
74
3%
Religion
66
3%
Science
41
2%
Environment
38
2%
Health
36
2%
Conflict Resolution Mediation
36
2%
Technology
32
1%
Poverty
32
1%
Planning
30
1%
Agriculture
1
< 1%
Pluralism, Multiculturalism, etc.
• Group dominance and the myth of racial
democracy: Antiracism attitudes in Brazil
• Citation Classic - the Politics of
Accommodation - Pluralism and
Democracy in the Netherlands
• The Modernization of Castes and the
Indianization of Democracy - the Case of
the Lingayat
Social Equity
• Democracy, inequality, and inflation
• On social democracy and the Kibbutz
• Does liberte equals egalite? A survey of
the empirical links between democracy
and inequality with some evidence on the
transition economies
• Mergers, concentration, and the erosion of
democracy
Number of titles in each domain of
democracy research
National
762
35%
International > 1 nation-state e.g. NAFTA countries
310
14%
Community
78
3%
28
1%
State subdivision of a nation
27
1%
Urban
22
1%
14
<1%
2
<1%
Global encompasses more than geopolitical entities e.g. biosphere
Regional >1 state within a nation e.g. Prairie Provinces
Rural
Qualifications about trends in this
data
• Actual frequencies for some descriptors
were small
• Increases in W of S indexed journals could
affect trends in “democracy” titles
• Much democracy research is moving from
journal to Internet publication, so how
representative is W of S?
Emphases and trends in
democracy titles over 25 years
• Categories: “Politics and Government” make up less
than half of the literature; many other cross-disciplinary
categories make up the rest.
• Topics: “Pluralism Identity Multiculturalism Gender Race
Relations” leads other topics in total quantity, even more
than “Foreign Relations”, which is tied with “Social
Equity” in total quantity.
• Domain: “National” has been the primary geographic
focus.
Where do we go from here?
• Adjust our category scheme with advice
from subject field experts
• Run end-user tests for utility of category
browsing
• Apply category scheme to remainder of
the online Refworks database