Transcript SCHOLARLY IMPACT METRICS AN OVERVIEW JOHAN BOLLEN – [email protected] INDIANA UNIVERSITY SCHOOL OF INFORMATICS AND COMPUTING CENTER FOR COMPLEX NETWORKS AND SYSTEMS RESEARCH OAI8 - June 2013

SCHOLARLY IMPACT
METRICS
AN OVERVIEW
JOHAN BOLLEN – [email protected]
INDIANA UNIVERSITY
SCHOOL OF INFORMATICS AND COMPUTING
CENTER FOR COMPLEX NETWORKS AND SYSTEMS RESEARCH
OAI8 - June 2013
OAI8 - June 2013
SCIENCE: IDEAS NOT BRICKS
Science and scholarly communication matters.
1)
Economic and cultural value is enormous, and rests on
considerable investments of
1)
2)
3)
4)
2)
Capital
Infrastructure
Human resources
Education
Outcomes: ideas and information
1)
2)
Not the amount of paper pulp produced, number of bricks
laid, metal forged, tractors built, fields plowed
It’s largely about the ideas and how they are communicated,
BUT:
1)
2)
OAI8 - June 2013
Not all ideas matter equally
Not all ideas should be communicated
SCIENCE AS A GIFT
ECONOMY
Gift economy:
-
services and good are shared freely without implicit or explicit
expectation/agreement of reciprocation
-
“economy of abundance, not scarcity”
-
found in some societies
Science is a little like that:
-
information is shared as freely as possible through publications
-
information is perishable (half-life of good idea)
-
reward for sharing is essentially a social phenomenon:
“esteem”, “prestige”, “influence”
OAI8 - June 2013
IMPACT ~ PUBLICATION
Scholarly outcomes and ideas are traditionally perceived to be mainly
shared through the peer reviewed literature, aka publications
-
An entire industry has emerged to support this modus operandi
-
Not universal, has not always been that way, might not always be this
way, but presently dominant
Our ideas of scholarly impact is now strongly tied to scholarly
publications
-
Ideas that impact or influence fellow scholars reach them via peerreviewed publications
-
Influence and impact is thus expected to be expressed through the
medium of peer-reviewed publications
-> Citation data has become de facto currency of impact or influence:
•
When one scholar cites the work of another, this is deemed recognition of
their influence
•
Measuring impact from citations
OAI8 - June 2013
CITATION DATA
OAI8 - June 2013
CITATION NETWORKS
The map equation M. Rosvall , D. Axelsson , and C.T. Bergstrom, European Journal of Physics, 178, 13–23 (2009)
Maps of random walks on complex networks reveal community structure Martin Rosvall*,† and Carl T. Bergstrom*, PNAS 105(4), 1118-1123
OAI8 - June 2013
FROM CITATION DATA TO
JOURNAL IMPACT FACTOR
Impact Factor = mean 2 year
citation rate
Journal x
2001
OAI8 - June 2013
All (2003)
2002
2003
THAT CONCLUDES THIS
LECTURE
Thank you for your undivided attention.
OAI8 - June 2013
HOLD ON
It’s just not that simple
OAI8 - June 2013
A FEW THINGS LEFT
TO DISCUSS…
OAI8 - June 2013
“THE MAP IS NOT THE
TERRITORY”
Impact, influence is a social phenomenon
• It already exists in the scholarly community
• Most scholars already have a notion of which ideas,
publications, journals, and authors matter the most
To measure this social construct of scholarly impact we can
choose many different “operationalizations”/measurements:
• Ask scientists: surveys, questionnaires, awards
• Correlates: funding decisions, publication data, citation data
• “Behavioral” data: readership, ILL, reshelving download data,
Twitter mentions, etc.
OAI8 - June 2013
MANY PERMUTATIONS
1.
Data type and which community it represents
•
Citation data: authors
•
Usage data: authors, readers, public
•
Social media data: everyone
2. Type of metric calculated from (1)
•
Counts, normalized counts
•
Social network metrics
•
Trend metrics
3. Level of granularity:
•
Entities: authors, journals, articles, teams, countries
•
Time: 5-year span, 2 year span, etc.
OAI8 - June 2013
METRICS, CUBED
usage
Social media
OAI8 - June 2013
author
Data type
journal
citation
arricle
Metric type
BACK TO CITATION DATA AND
NETWORKS
Johan Bollen, Herbert Van de Sompel and Marko A. Rodriguez. Towards usage-based impact metrics: first
results from the MESUR project, JCDL 2008, Pittsburgh, PA, June 2008. (arXiv:0804.3791v1)
CITATION-BASED METRICS
-
Author-level metrics:
-
Total citations
H-index:
-
-
Nth publication with at least n citations
(rank order pubs by decr. Cites)
g-index, e-index, a-index
-
- Co-author network indicators
Article level metrics:
-
- Total citations
- Normalized citation counts
Journal level:
-
Impact factor
SNIP, Crown indicator
Social network metrics from citation
network (next slide: PageRank,
Eigenfactor, Y-factor, betweenness, etc)
OAI8 - June 2013
Hirsch (2005) PNAS 102(46) 16569-16572
Radicchi et al . (2008) PNAS 105(45) 17268-17272
INNOVATION I : CITATION-BASED
SOCIAL NETWORK METRICS
Degree
•
In-degree
•
Out-degree
Random walk
•
PageRank
•
Eigenvector
Shortest path
•
Closeness
•
Betweenness
SOCIAL NETWORK
ANALYSIS
OAI8 - June 2013
PAGERANK FOR JOURNALS
2003 JCR, Science Edition
5709 journals, L=0.85
Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: theory, with
application to the literature of physics. Information processing and management, 12(5), 297-312.
Chen, P., Xie, H., Maslov, S., & Redner, S. (2007). Finding scientific gems with Google. Journal of Informetrics,
1(1), arxiv.org/abs/physics/0604130.
PAGERANK FOR JOURNALS
POPULARITY VS. PRESTIGE
Outliers reveal
differences in aspects
of “status”
IF ~ general popularity
PR ~ prestige,
influence
Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel. Journal status. Scientometrics, 69(3), December 2006 (DOI:
10.1007/s11192-006-0176-z)
Philip Ball. Prestige is factored into journal ratings. Nature 439, 770-771, February 2006 (doi:10.1038/439770a)
INNOVATION II: “BEHAVIORAL” DATA
Scholarly community and communication is moving online.
Data pertaining to online activities (implicit, behavioral) vs. citation
data (explicit declaration of influence)
Bibliographic
data
Scholarly community
metrics
Citation
Behavioral data
OAI8 - June 2013
Scholarly
communication
items
BEHAVIORAL DATA
Reading/usage statistics
• Interlibrary loan data
• Reshelving data
• Online catalogue systems
Daily, weekly, monthly access or reading statistics
Usage data:
• Web server logs
• Link resolver data (SFX, etc)
Detailed data on “who”, “what”, “where”, “when”: ability to track
scholarly activity in real-time
OAI8 - June 2013
USAGE STATISTICS
COUNTER: member organization defining an auditable
standard for reporting and aggregating monthly usage
statistics (www.projectcounter.org)
• Journal and article level
• Initiative to define “usage factor”
PLoS Article Level Metrics
• Download numbers
• download trends
OAI8 - June 2013
MESUR
Andrew W. Mellon and NSF funded project at
LANL Digital Library Research and
Prototyping and Indiana University
-
Very large-scale usage data from
publishers, aggregators, and library
consortia
-
Metrics of scholarly impact derived from
aggregated usage data
-
Mapping scientific activity from log
clickstream data
-
Examine“scholarly impact” itself (more
later!)
Notable distinction: use of log data that
contains clickstream enables metrics and
analysis beyond level of usage statistics
Presently concluding planning process
(Andrew W. Mellon funded) to evolve to
community-supported, sustainable entity
OAI8 - June 2013
INNOVATION III: ALT-METRICS
Behavioral AND “attention” data.
• Social media attention, bookmarking, mentions
• Attempt to also capture “social” attention or public impact
of scholarly work (not just articles!), another possible
dimension of impact
OAI8 - June 2013
SOME RELEVANT RESEARCH
Eysenbach G (2011) Can tweets predict citations? Metrics of
social impact based on twitter and correlation with traditional
metrics of scientific impact. Journal of Medical Internet
Research 13: e123.
Shuai X, Pepe A, Bollen J (2012) How the Scientific Community
Reacts to Newly Submitted Preprints: Article Downloads,
Twitter Mentions, and Citations. PLoS ONE 7(11): e47523.
doi:10.1371/journal.pone.0047523
OAI8 - June 2013
time
TWITTER MENTIONS ~
DOWNLOADS, CITATIONS?
2010-10-14
Preprint
1010.3003
submitted
to arXiv.org
2010-10-18
1,530
2010-10-18
47
Weekly downloads
on arXiv.org
Tweets about
1103.0609
2010-10-18
73,000
2010-12-11
1
OAI8 - June 2013
Citation in
Google Scholar
Weekly downloads
on arXiv.org
TWITTER MENTIONS CORRELATE
WITH DOWNLOADS AND
CITATIONS!
OAI8 - June 2013
ALT-METRICS AS PART
OF IMPACT ASSESSMENT
OAI8 - June 2013
CITATION DATA, METRICS,
IMPACT, ALT-METRICS, USAGE
DATA, LET’S STEP BACK FOR A
SECOND
OAI8 - June 2013
BLIND MAP-MAKERS
Odd, nearly tautological situation:
-
We have many different metrics or ways to measure impact.
-
But no formal or consistent definition of scholarly impact.
-
No idea of what exactly impact is, how it manifests itself, what
its structure is, along which dimensions it varies. etc
-
Whether our metrics actually measure or represent impact
-
Our metrics ARE the definition of “impact”
OAI8 - June 2013
SCHOLARLY IMPACT
Metric 1
Metric 2
Validity &
Reliability
Some form
of impact
impact
Metric 4
Not quite
impact
Metric 6
Not impact
Metric 3
OAI8 - June 2013
Metric 5
MAPPING OUT IMPACT,
ONE METRIC AT A TIME
•
Bollen J, Van de Sompel H, Hagberg A,
Chute R (2009) A Principal Component
Analysis of 39 Scientific Impact Measures.
PLoS ONE 4(6): e6022.
doi:10.1371/journal.pone.0006022
•
Priem at al. Altmetrics in the wild.
•
Thelwall M, Haustein S, Larivière V,
Sugimoto CR (2013) Do Altmetrics Work?
Twitter and Ten Other Social Web Services.
PLoS ONE 8(5): e64841.
doi:10.1371/journal.pone.0064841
•
PLoS ONE alt-metrics correlations:
investigated by L Juhl Jensen, Novo
Nordisk Foundation
•
Bornmann, L., Mutz, R., & Daniel, H.-D.
(2008). …A comparison of nine different
variants of the h index using data from
biomedicine. JASIST, 59(5), 830-837.
OAI8 - June 2013
FINALLY… WHY?
Just like social status, scholarly impact (or other) is an
interesting scientific study area. It emerges from the
scholarly communication process.
BUT pure science is clearly not the only motivation:
• Metrics used in assessment
• Decision-making: funding, promotion, …
• Information filtering
Some of these applications are tremendously useful and
potentially enabling of radical changes in scholarly
communication, e.g. information filtering and assessing
broader community impact of scholarly work.
OAI8 - June 2013
HOWEVER…
Assuming that scholarly impact exists, independently of
whether we measure it or not:
Why measure it at all in cases where the scholarly
community truly has decision-making power, autonomy?
Isn’t the latter a more desirable option than administrators,
politicians, and bureaucrats making decisions on the basis of
numbers they don’t understand?
So buy me a beer and ask me about our crazy crowd-sourced
funding idea…
Johan Bollen, David Crandall, Damion Junk, Ying Ding, Katy
Boerner. Collective allocation of science funding: from funding
agencies to scientific agency. http://arxiv.org/abs/1304.1067
OAI8 - June 2013
THANK YOU
OAI8 - June 2013