Transcript Institute for Research Information and Quality Assurance “Open access and research evaluation” Thursday 19th October 14.15-15.45 Session 4: Quality Assessment Open Scholarship 2006: New.

Institute for Research Information and Quality Assurance
“Open access and research evaluation”
Thursday 19th October
14.15-15.45 Session 4: Quality Assessment
Open Scholarship 2006: New Challenges
for Open Access Repositories
The University of Glasgow 18-20 October 2006
Stefan Hornbostel
iFQ Institut für Forschungsinformation und Qualitätssicherung
Institute for Researchinformation and Quality Assurance
Godesberger Allee 90
D-53175 Bonn
www.forschungsinfo.de
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Relations between OA and research evaluation
political / rethorical function to spread OA
(to maximize the impact of their articles authors should archive!)
OA offers a better way to evaluate research performance?
(Compensation of shortcomings of traditional bibliometric Indicators)
OA changes scientific behaviour
(more easily accessibility, sooner availability, quick circulation, high
quality).
OA makes it possible to construct new indicators to measure a
completely different dimension (usage, reading, …)
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Relations between OA and research evaluation (1)
political / rethorical function to spread OA
(to maximize the impact of their articles authors should archive!)
Research evaluation practices are used to convince authors of OA
advantages (higher Citationrates, higher attention)
The analysis covers 1,494 publication venues containing at least 5 online and 5 offline articles. For 90% of
venues, online articles are more highly cited on average. On average there are 336% more citations to online
articles compared to offline articles.
Lawrence, S. "Online or Invisible" Nature 411 (6837): 521, 2001.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
„Mandated online RAE CVs Linked to University Eprint Archives“,
“The Funding Councils should mandate that ……. the full text of every refereed research
paper, publicly self-archived in the university's online Eprint Archive and linked to the
CV for online harvesting, scientometric analysis and assessment.
This will
increase the uptake and impact of UK research output, by increasing its visibility,
accessibility and usage, and
set an example for the rest of the world that will almost certainly be emulated, in both
respects: research assessment and research access.”
Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint
Archives. Ariadne 35.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
“The proportion of articles for which their authors provide OA is likely to
increase dramatically now, in part because of the mounting evidence for the
impact advantage OA confers.”
Brody, T. and Harnad, S. (2004) Comparing the Impact of Open Access (OA) vs. Non-OA Articles in the Same
Journals. D-Lib Magazine 10(6).
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
For Example: Cream of Science
“Researchers are enthusiastic about the initiative. A number of
them already refer to www.creamofscience.org on their own
websites for a complete overview of their work. They regard Cream
of Science as a hallmark of quality.”
Dare.net 17.05.2005
•Target
“demonstrat(ing) that scholars are willing to deposit their materials in a repository,
thereby also increasing the awareness of other scholars”
• Method
“DARE partners selected ten of their prominent scientists and made their
complete publication list, with as much full text available as possible”
• Selection Criteria
“there was no objective criteria to define 'prominent',…but most of the DARE
partner used a formal method; selection by the Executive Board and/or using a
letter from the Dean to invite academics to be part of Cream”
Martin Feijen and Annemiek van der Kuil 2005: A Recipe for Cream of Science: Special Content Recruitment for Dutch
Institutional Repositories', Ariadne Issue 45.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
OA a causal factor ?
Kurtz et al. claim that there are (at least) three possible, non-exclusive effects
that cause the effect of higher citation rates.
Open Access Postulate (OA): Because of free, unrestricted access, papers
are read more easily and therefore get cited more frequently
Early Access Postulate (EA): Papers offered as e-print are available sooner
and therefore gain primacy and additional time in press, and therefore they get
cited more often
Self-selection Bias Postulate (SB) or (Quality Postulate) : The most
important -- and therefore most citable -- papers are posted on the Internet
Kurtz, Michael J., Eichhorn, Guenther, Accomazzi, Alberto, Grant, Carolyn, Demleitner, Markus, Henneken, Edwin,
Murray, Stephen S. (2005). “The Effect of Use and Access on Citations”, Information Processing and Management,
Vol. 41, Issue 6, p. 1395-1402.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Open Access Postulate
• no relationship between fulltext arXiv downloads and the number of article
citations until an article has been downloaded more than 400 times
• consistent with the detailed work of Henk Moed on the journal Tetrahedron
Letters, which suggests that a relatively small group of both highly cited and
frequently downloaded papers are responsible for the weak, though
statistically significant, correlation between downloads and citations
Philip M. Davis / Michael J. Fromerth (2006):
Does the arXiv lead to higher citations and reduced publisher
downloads for mathematics articles?
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Early Access Postulate
Partial regression plot of prepublication days
on citation, controlling for the number of days
since publication and the effect of the journal.
Articles deposited in the arXiv after formal
publication are highlighted red.
We have failed to find support for the Early View postulate and have provided
evidence that many highly-cited articles were deposited into the arXiv after formal
publication.
Philip M. Davis / Michael J. Fromerth (2006):
Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles?
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Self-selection Bias Postulate (SB) or Quality Postulate
• approximately 15% of the articles in a journal collect 50%
of the citations
• If authors are indeed depositing their best papers, Open Access
citation advantage may be an artifact, not the cause of a citation
advantage
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Problems: Open Access (green and gold), Open Archive, Self Archiving may produce
different quality distributions
J. Testa and M.E. McVeigh. 2004.“The Impact of Open Access Journals.” A Citation Study from Thomson ISI, www.isinet.com/oaj .
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
OA offers a better way to evaluate research performance?
(Compensation of shortcomings of traditional bibliometric
Indicators)
“These new possibilities may be interesting seen in combination with the
known inherent bias of research evaluation using citation analysis. …..”
Tove Faber Frandsen (2006): Open access-based resources in research evaluation
The Funding Councils should mandate that … all UK research-active
university staff must maintain: .... (II) the full text of every refereed
research paper, publicly self-archived in the university's online Eprint
Archive …..
This will
give the UK Research Assessment Exercise (RAE) far richer, more
sensitive and more predictive measures of research productivity and
impact, for far less cost and effort (both to the RAE and to the
universities preparing their RAE submissions)…
Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives. Ariadne 35.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
ISI / WOS coverage by main field
EXCELLENT
(> 80%)
Good (60-80%)
Good (40-60%)
MODERATE
(<40 %)
Biochem & Mol Biol
Appl Phys & Chem
Mathematics
Other Soc Sci
Biol Sci ~ Humans
Biol Sci ~ Anim &
Plants
Economics
Humanities & Arts
Chemistry
Psychol & Psychiat
Engineering
Clin Medicine
Geosciences
Physics & Astron
Soc Sci ~ Medicine
& Health
Quelle: Henk v. Moed, Evaluation of Research Performance and Funding Programme in Social Sciences; Tagung: Norface Workshop on Research
Programme Development and Management 7.02.06, Bonn: http://www.norface.org/norface/files/file_20060207014936Henk%20Moed.pdf
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Comparison of Mean Citation Rates Between Freely Available Articles and
Those That Are Not Freely Available
Discipline
Mean
(open)
Mean Standard
Error
(open)
Mean
(not open)
Mean Standard
Error
(not open)
Difference in
Means
Philosophy
1.60
0.491
1.10
0.230
.500
Political Science
2.20
0.477
1.18
0.353
1.016
Electronical and
electronic
engineering
2.35
0.449
1.56
0.275
.798
Mathematics
1.60
0.270
0.84
0.230
.762
Antelman, Kristin (2004) Do Open Access Articles Have a Greater Research Impact?. College & Research
Libraries News 65(5):pp. 372-382.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in
Transformation?
Results of a study on publishing habits and information acquisition with regard
to open access
In electrical engineering, computer science and system engineering, 34 % of the respondents
know of this form of publication; the highest percentage here is among computer scientists,
at 51 %. In mechanical engineering and production engineering, 31 % of the scientists said
they knew about open access journals, compared to just 13 % in the area of heat energy
technology and process engineering. In the areas of civil engineering and architecture this
type of publication is completely unknown.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
In the humanities and social sciences, it is mostly social and behavioural scientists who
use the option of secondary open access publication for their academic studies. While
they make around 9% of the articles published in journals available for free on the
internet, this is done by only 3% of the humanities scholars surveyed.
In the engineering sciences it is the computer scientists who stand out, with two-thirds of
their articles in conference proceedings, 46 % of their journal articles, almost 42 % of their
contributions to edited volumes and nearly 24% of their monographs republished on the
internet.
Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in Transformation?
Results of a study on publishing habits and information acquisition with regard to open access
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in Transformation?
Results of a study on publishing habits and information acquisition with regard to open access
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
AUGMENTING INTEROPERABILITY ACROSS SCHOLARLY REPOSITORIES
A meeting sponsored and supported by Microsoft, the Andrew W. Mellon Foundation, the Coalition for Networked
Information, the Digital Library Federation, and the Joint Information Systems Committee.
April 20 – 21, 2006, New York, NY
• Richer
cross-repository services, that is services that overlay multiple repositories
• Cross-repository scholarly communication workflows
Examples of such value-chains include:
•The transfer of (parts of) digital objects from digital repositories to parties that provide discovery-oriented
services over (parts of) the digital objects.
•The transfer of digital objects from digital repositories to parties that provide value-added services over the
digital objects.
• The introduction of natively machine-readable and machine-actionable bibliographic citations.
• The re-use of datasets (from different repositories) as the basis for building a new dataset or for writing a
publication.
Ithaca, NY and Los Alamos, NM - The Open Archives Initiative (OAI), with the generous support of the Andrew W. Mellon
Foundation, announces a new effort as part of its mission to develop and promote interoperability standards that aim to
facilitate the efficient dissemination of content.
Object Reuse and Exchange (ORE) will develop specifications that allow distributed repositories to exchange information
about their constituent digital objects. These specifications will include approaches for representing digital objects and
repository services that facilitate access and ingest of these representations. The specifications will enable a new
generation of cross-repository services that leverage the intrinsic value of digital objects beyond the borders of hosting
repositories.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
OA changes scientific behaviour
(more easily accessibility, sooner availability , quick circulation, high
quality)
Scholarly communication has changed rapidly over the last decade as
scientists now publish their findings in e.g. open access journals and preprint
servers. Bauer & Bakkalbasi (2005)
The communication system is changing primarily due to the use of new
technology. Friend (2004)
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
But: only c. 20% of the number of papers
published annually are open access
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Poeschl,U. 2006,
Peer Review Revisited,
16.05.2006, Ms.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
OA makes it possible to construct new indicators to measure a
completly different dimension (usage, reading, …)
Web indicators based on usage data are a way to obtain unique information of
the utility and scholarly acceptance of OA research output, but these new user
oriented web indicators are still in an early experimental phase.
The rationale of these approaches is strong but standardization of web
metrics is at the beginning. …. there is still much to be done in evaluation of
OA research. In addition we need further approaches to build robust
measures and metrics especially for OA publications/documents. Following
the thesis “science is turning to e-science” … we conclude e-science will need
new stable e-indicators or a combination with traditional indicators to
appropriately take e-only research output into consideration.
Philipp Mayr (2006): Constructing experimental indicators for Open Access documents.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
The rendered journal impact rankings
were compared to those derived from
the Reader Generated Network, and
the ISI IF. Our results indicate the
following:
First, journal relationships in the RGN network seem to be valid and representative of
the community whose downloads have shaped the network.
Second, structural journal impact metrics derived from the RGN deviate strongly from
the ISI IF.
Third, the applied structural impact metrics correlate strongly with the ISI IF when
calculated over the AGN, indicating they do validly operationalize journal impact, if we
honor the assumption that the ISI IF does.
Fourth, the AGN and RGN networks overlap to some degree, but exhibit striking
differences.
Darmoni, Roussel, Benichou, Thirion, and Pinhas (2002) compare journal usage
frequency to the ISI IF for a medical DL collection. They define a “Reading Factor” (RF)
which consists of the ratio of a particular journal’s download frequency to the total
downloads of all journals as recorded in the DL’s logs. The authors report a low and
statistically insignificant correlation between the observed RF and the ISI IF for the
same set of journals.
Johan Bollen / Herbert Van de Sompel / Joan A. Smith / Rick Luce (Los Alamos National Laboratory) 2005: Toward alternative metrics of
journal impact:A comparison of download and citation data.
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Standards!
OA research evaluation should follow the classical methods of bibliometrics:
Measuring Research activity
P
amount of Publications
P%
Percentage of worldwide Publications (international comparison) or
national Publications (national comparison)
Measuring Impact
C
amount of citations
Pp%
percentage of publications not cited until now
Cmax Maximum of Citations
C/P
Citation Rate
C/P/FCSm (mean field citation score) Relative Citation Rate (related to field)
(C/P)/JCSm (mean journal citation score) Relative Citation Rate (related to
journal)
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Thank you!
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
Difficulties of OA research evaluation
Only fits to publications in traditional databases - especially ISI
Difficult for publications which are not covered by ISI
Missing subfield information
Metadata
No standardised metadata
No gold standard for the evaluation of open access papers
No best practices of hundred percent covering
Units
Main units for OA research evaluation are inexplicit
Different kinds of publications
Different motives for OA publishing -> bias!
Hornbostel
Institut für Forschungsinformation und Qualitätssicherung
19.10.2006
It was found that during the first 3 months after an article is cited, its
number of downloads increased 25% compared to what one would
expect this number to be if the article had not been cited. Moreover,
more downloads of citing documents led to more downloads of the
cited article through the citation. An analysis of 1,190 papers in the
journal during a time interval of 2 years after publication date revealed
that there is about one citation for every 100 downloads. A Spearman
rank correlation coefficient of 0.22 was found between the number of
times an article was downloaded and its citation rate recorded in the
SCI. When initial downloads - defined as downloads made during the
first 3 months after publication - were discarded, the correlation raised
to 0.35. However, both outcomes measure the joint effect of
downloads upon citation and that of citation upon downloads.
Correlating initial downloads to later citation counts, the correlation
coefficient drops to 0.11. Findings suggest that initial downloads and
citations relate to distinct phases in the process of collecting and
processing relevant scientific information that eventually leads to the
publication of a journal article.
Statistical relationships between downloads and citations at the level of
individual documents within a single journal
Henk F. Moed Journal of the American Society for Information Science and
Technology
Volume 56, Issue 10 , Pages 1088 - 1097
Published Online: 31 May 2005
Institut für Forschungsinformation und Qualitätssicherung
Hornbostel
19.20.2006
Preprint submitted to Elsevier Science 17 May 2006
Similarly, Darmoni, Roussel, Benichou,
These
Thirion, and Pinhas (2002) compare
journal usage frequency to the ISI IF for a results furthermore raise questions
regarding the validity of the ISI IF as the
medical DL collection. They define
a “Reading Factor” (RF) which consists of sole
assessment of journal impact, and sugges
the ratio of a particular journal’s
download frequency to the total downloadsthe possibility of devising impact metrics
based on usage information in general.
of all journals as recorded in the
Toward alternative metrics of journal
DL’s logs. The authors report a low and
A comparison of download and citati
statistically insignificant correlation
Johan Bollen
between the observed RF and the ISI IF for
Department of Computer Science, O
the same set of journals. These
University, 4700 Elkhorn Ave.,
results show that journal download
Norfolk VA 23529
frequency within a local DL community
Herbert Van de Sompel
does not correspond to the ISI IF, which
Research Library, Los Alamos Natio
raises questions regarding the ISI
Alamos, NM, 87554
IF’s validity as the sole indicator of Ig
Joan A. Smith
among a specific community of readers.
Department of Computer Science, O
4
University, 4700 Elkhorn Ave.,
Hornbostel
Norfolk VA 23529
Institut für Forschungsinformation und Qualitätssicherung
19.20.2006
Rick Luce