Transcript Kein Folientitel

BIBLIOMETRICS –
USE AND LIMITATIONS
Wolfgang Glänzel
KU Leuven, Belgium
ISPR, HAS, Hungary
STRUCTURE OF THE PRESENTATION
1. Introduction
2. Scope and structure of bibliometrics
3. Data sources of bibliometric research
4. “Perspective shift” in use of bibliometrics
5. Bibliometrics as a tool in research evaluation
6. Bibliometric research and services at the
Steunpunt O&O Statistieken
Wolfgang Glänzel
Bibliometrics. Use and limitations
2/37
1. INTRODUCTION
Wolfgang Glänzel
Bibliometrics. Use and limitations
3/37
1.
Introduction
What is bibliometrics?
The terms bibliometrics and scientometrics were introduced
by Pritchard and Nalimov & Mulchenko in 1969, and are
nowadays used almost synonymously.
Pritchard
“bibliometrics is the application of mathematical and
statistical methods to books and other media of
communication”.
Nalimov & Mulchenko
Scientometrics is “the application of those quantitative
methods which are dealing with the analysis of
science viewed as an information process”.
Wolfgang Glänzel
Bibliometrics. Use and limitations
4/37
1.
Introduction
Gorkova (1988)
Informetrics stands for a more general subfield of
information science dealing with mathematicalstatistical analysis of communication processes in
science.
A quite broad definition of the scope of bibliometrics
given by Glänzel & Schoepflin (1994) includes “all
quantitative aspects and models of science
communication, storage, dissemination and retrieval
of scientific information”.
Wolfgang Glänzel
Bibliometrics. Use and limitations
5/37
1.
Introduction
These notions also integrate many presently existing
orientations such as applications to science policy, library
science, technometrics, Web analyses and information
retrieval. Prominent methodological issues are related to
•
•
•
•
•
•
•
•
•
•
subject delineation
subject evolution and structure
publication activity/scientific productivity
scientific collaboration, co-authorship networks
citation impact and citation networks
bibliographic coupling, co-citation analysis
computational linguistics; text mining, co-word analysis
visualisation techniques
data matching
science-technology linkage
Wolfgang Glänzel
Bibliometrics. Use and limitations
6/37
1.
Introduction
Common misbeliefs on bibliometrics
• Main task of bibliometrics should be the expeditious
issuing of “prompt” and “comprehensible” indicators for
science policy and research management.
• Bibliometrics is only an auxiliary tool, research on
methodology is unnecessary. Instead, bibliometricians
should elaborate guidelines explaining the use of their
indicators.
• Bibliometrics might be reduced to simple counting
activities in order to replace/supplement qualitative
assessment by quantitative indicators and to set
publication output off against funding.
Wolfgang Glänzel
Bibliometrics. Use and limitations
7/37
1.
Introduction
Facts about bibliometrics
Bibliometrics is a powerful, multifaceted endeavour
encompassing sub-areas such as
- Structural scientometrics came up with results like
the re-mapping of the epistemological structure of
science.
- Dynamic scientometrics constructed sophisticated
models of scientific growth, obsolescence, citation
processes, etc.
- Evaluative scientometrics developed arrays of
indicators to be used to characterise research
performance at different levels of aggregation and
come up with benchmarking solutions.
Wolfgang Glänzel
Bibliometrics. Use and limitations
8/37
2. SCOPE AND STRUCTURE OF
BIBLIOMETRICS
Wolfgang Glänzel
Bibliometrics. Use and limitations
9/37
2.
Structure of Bibliometrics
Three “components” of present-day bibliometrics
according to its three main target-groups
Bibliometrics for bibliometricians
The domain of bibliometric “basic research”.
Bibliometrics for scientific disciplines
A large but also the most diverse interest-group. Due to the
scientists’ primary scientific orientation, their interests are
strongly related to their speciality.
Bibliometrics for science policy and research management
The most important field of application. Here the assessment
and comparative analysis of research performance are in the
foreground.
Wolfgang Glänzel
Bibliometrics. Use and limitations
10/37
2.
Structure of Bibliometrics
Links of bibliometrics with related research fields and application services
Science policy
Research management
Scientific information
Librarianship
Services for
Research in
Economics
Sociology of science
History of science
applied
Library and
Information Science
Scientometrics
basic
Life sciences
Informetrics
Mathematics/Physics
Webometrics
Wolfgang Glänzel
Bibliometrics. Use and limitations
11/37
3. DATA SOURCES OF BIBLIOMETRIC
RESEARCH
Wolfgang Glänzel
Bibliometrics. Use and limitations
12/37
3.
Sources of Bibliometrics
Data sources of bibliometrics are bibliographies and
bibliographic databases.
Large scale analyses can only be based on bibliographic
databases.
Despite objections to journal coverage and data processing
policy of the ISI, the databases of the Institute for Scientific
Information (Thomson - ISI), above all, the Science Citation
Index (Expanded) have become the most generally
accepted data source.
Recently several countries have developed their own
national citation indexes which are also used (as
supplementary) data sources.
Wolfgang Glänzel
Bibliometrics. Use and limitations
13/37
3.
Sources of Bibliometrics
Bibliographic databases are generally designed for the
retrieval of information. However, their versatility is often
limited in bibliometric use.
Originally, bibliometrics was developed for studying
communication in basic research. Present-day tasks:
• How to extend bibliometrics towards application to
engineering, social sciences and humanities?
• How to measure and evaluate aspects of technology
linkage?
Since non-periodicals are important conveyors of
information in engineering, social sciences and
humanities, journal based data-sources are accepted by
scientists only with certain reservations.
Wolfgang Glänzel
Bibliometrics. Use and limitations
14/37
4. “PERSPECTIVE SHIFT” IN USE OF
BIBLIOMETRICS
Wolfgang Glänzel
Bibliometrics. Use and limitations
15/37
4.
Perspective Shift
Bibliometrics deals with the quantitative analysis of the
(social) network of scientific communication.
From the historical perspective, information scientists and
bibliometricians proceeded from the same approach.
First citation analyses were designed to aid the decision
which periodicals should best be purchased by small
libraries. Many tools and indicators developed by the ISI
were designed for use in scientific information, information
retrieval and libraries.
 The application to science policy has brought a new
perspective, and resulted in re-interpretation of
bibliometric conceptions.
Wolfgang Glänzel
Bibliometrics. Use and limitations
16/37
4.
Perspective Shift
Example: The notion of citations in information
science, bibliometrics and science policy
Citations became a widely used measure of the impact of
scientific publications, but the notion of what citations
actually express sheds light on different aspects.
These different views might results in conflicts caused by
re-interpretation and “perspective shift”.
Wolfgang Glänzel
Bibliometrics. Use and limitations
17/37
4.
Perspective Shift
Some examples:
According to Merton, citations are intimately connected with
the reward system of science.
Cozzens: “Citation is only secondarily a reward system.
Primarily, it is rhetorical-part of persuasively arguing for the
knowledge claims of the citing document.”
Cronin: Citations are "frozen footprints in the landscape of
scholarly achievement … which bear witness to the
passage of ideas“.
Glänzel and Schoepflin: Citations give “a formalised account
of the information use and can be taken as a strong
indicator of reception at this level.”
Wolfgang Glänzel
Bibliometrics. Use and limitations
18/37
4.
Perspective Shift
The process of re-interpreting the notion of citation and its
consequences
interpretation
citation
Bibliometrics/
Information science
Information use
repercussion
re-interpretation
(possible distortion of
citation behaviour)
Research evaluation/
Science policy
Wolfgang Glänzel
uncitedness: unused information
frequent cite: good reception
self-cite: part of scient. communication
Reward system
(quality measure)
uncitedness: low quality
frequent cite: high quality
self-cite: distortion of impact
Bibliometrics. Use and limitations
19/37
5. BIBLIOMETRICS AS A TOOL IN RESEARCH
EVALUATION
Wolfgang Glänzel
Bibliometrics. Use and limitations
20/37
5.
Research Evaluation
Present-day use of bibliometrics
The evaluation of scientific research performance is the most
important application of present-days bibliometrics.
• Growing, often controversial, policy interest to use bibliometric
techniques as output measures of research productivity and
efficiency.
• Thus bibliometrics evolved to a standard tool in science policy
and research management.
• Bibliometric components are used, among others, in national
science reports, bibliometric indicators appear in the daily
press, benchmarking results are used for research
management and for allocation of research funding.
Wolfgang Glänzel
Bibliometrics. Use and limitations
21/37
5.
Research Evaluation
What can bibliometrics not be responsible for?
• Bibliometrics can be used to develop and provide
tools to be applied to research evaluation, but is not
designed to evaluate research results.
• Bibliometrics does not aim at replacing qualitative
methods by quantitative approaches.
• Consequently, bibliometrics is not designed to correct
or even substitute peer reviews or evaluation by
experts but qualitative and quantitative methods in
science studies should complement each other.
Wolfgang Glänzel
Bibliometrics. Use and limitations
22/37
5.
Research Evaluation
Distorted behaviour based on policy use and misuse of
bibliometric data
One issue concerns the changes in the publication, citation
and collaboration behaviour of scientists (both positive and
negative) that the consistent policy use of bibliometric
indicators might potentially induce.
Uninformed use and misuse are not always beyond the
responsibility of bibliometricians. Unfortunately,
bibliometricians do not always resist the temptation to follow
popular, even populist, trends in order to meet the
expectations of the customers.
Any kind of uninformed use or misuse of bibliometric results
involves the danger of bringing bibliometric research itself into
disrepute.
Wolfgang Glänzel
Bibliometrics. Use and limitations
23/37
5.
Research Evaluation
The problem of inappropriate use ranges from uninformed
use, over selecting and collecting ‘most advantageous’
indicators to the obvious and deliberate misuse of data.
Uninformed use
• incorrect presentation, interpretation of bibliometric
indicators or their use in an inappropriate context caused
by insufficient knowledge of methodology, background and
data sources
Misuse
• intentionally incorrect presentation, interpretation of
bibliometric indicators or their deliberate use in
inappropriate context
• tendentious application of biases or tendentious choice of
(incompatible) indicators
Wolfgang Glänzel
Bibliometrics. Use and limitations
24/37
5.
Research Evaluation
 But even correct use might have undesired consequences
• Author self-citations are shown in an unfavourable light.
Authors might thus feel urged avoiding self-citations – a
clear intervention into the mechanism of scientific
communication.
• Indirect repercussions can be observed when bibliometric
tools are used in decision-making and the scientific
community recognises the feedback in terms of their
funding.
• Butler (2004) showed on the example of Australia what
might happen when funding is linked to publication counts.
She found that this has stimulated an increased publication
activity in the lower-impact journals.
Wolfgang Glänzel
Bibliometrics. Use and limitations
25/37
5.
Research Evaluation
Schematic visualisation of the feedback of policy use of
bibliometrics on the scientific community
Bibliometrics
Scientific community
Publication & citation
behaviour
Science policy &
Research management
Funding &
Promotion
Wolfgang Glänzel
Bibliometrics. Use and limitations
26/37
5.
Research Evaluation
Possible positive effects
• Scientists might recognise that scientific collaboration and
publishing in high-impact or even top journals pays off. Also
their publication activity might be stimulated.
Possible negative effects
• Exaggerated collaboration, even trends towards hyperauthorship, inflating publication output by splitting up
publications to sequences, inflating citation impact by selfcitations and forming citation cliques, etc.
• Trend towards replacing quality and recognition by visibility at
any price or towards preferring journals as publication
channels in social sciences and humanities might be among
these effects.
• A certain “champions league” mentality is spreading among
scientists (“Shanghai ranking”, H-index, “IF Filter”).
Wolfgang Glänzel
Bibliometrics. Use and limitations
27/37
6. BIBLIOMETRIC RESEARCH AND SERVICES
AT THE STEUNPUNT O&O STATISTIEKEN
Wolfgang Glänzel
Bibliometrics. Use and limitations
28/37
6.
The Steunpunt O&O Statistieken
The Steunpunt O&O Statistieken (Research Centre for
R&D Statistics – SOOS) was created in January 2002
after open calls for proposals.
SOOS is a unique and dedicated research center to
support its science and innovation policy in Flanders.
This center has to develop an appropriate system of
quantitative indicators to quantify R&D efforts at
Flemish universities, research institutes and industry
that can be used by policy makers to support the
development of appropriate regional science and
innovation policy for Flanders.
Wolfgang Glänzel
Bibliometrics. Use and limitations
29/37
6.
The Steunpunt O&O Statistieken
ACTIVITY OVERVIEW
The activities of SOOS can be classified into four
categories:
1. Developing an appropriate IT-infrastructure to
support S&T indicator development;
2. Providing the Flemish government with appropriate
S&T indicators on which to base its science and
technology policy resource allocation decisions;
3. Executing specific studies on science and
technology domains or science policy relevant
topics for the Flemish government;
4. Doing original research in the field of S&T studies.
Wolfgang Glänzel
Bibliometrics. Use and limitations
30/37
6.
The Steunpunt O&O Statistieken
LONG- AND MEDIUM-TERM SERVICES FOR THE
FLEMISH GOVERNMENT
I. The BOF-key
One of the major funding mechanisms for basic science
in Flanders is the BOF-mechanism. The Bijzonder
Onderzoeksfonds is a research fund of the Flemish
government that is distributed to the Flemish universities on
the basis of a specific allocation model. This allocation key
comprises both input and output measures on the research
performance of the universities involved. Part of the key is
based on publication and citation data as derived from WoS
by SOOS.
Wolfgang Glänzel
Bibliometrics. Use and limitations
31/37
6.
The Steunpunt O&O Statistieken
BOF-funding for Flanders amounts to 100 million EUR
annually. This amount is distributed in basic science funding
amongst the six Flemish universities (LUC/UHasselt, KUB,
KUL, VUB, UA and UGent).
II. The IOF-key
The IOF-key is a distribution key to determine the annual
allocation of 12 million EUR (to be augmented in the year
2006) amongst the six Flemish universities in order to support
applied research.
The IOF-key is partially based on patent statistics derived from
the USPTO, EPO and PCT databases. SOOS has been
instrumental both in the development of and the data delivery
for those distribution instruments.
Wolfgang Glänzel
Bibliometrics. Use and limitations
32/37
6.
The Steunpunt O&O Statistieken
III. The Flemish Indicatorbook
The Flemish Indicatorbook on
Science, Technology and
Innovation appears biannually,
and does not only support
government questions on R&Dproficiency in Flanders, but also
aims to supply adequate and upto-date information on the
European R&D-potential of the
Flanders region.
Two editions of the Indicatorbook
have been published by SOOS
so far: 2003 and 2005.
Wolfgang Glänzel
Bibliometrics. Use and limitations
33/37
6.
The Steunpunt O&O Statistieken
IV. Domain Studies (here bibliometrics only)
Aim of these studies is to conduct and to provide
analyses of Flemish research and technology activity
in the international context in highly important areas.
Expected outcomes are relevant information about
performance and competitiveness in Flanders.
Three domain studies have been prepared and published
by SooS: Nanoscience and –technology, Biotechnology
and molecular biology (both in 2003) and Stem-cell
research (2004).
The studies are based on both publication and patent
data extracted from the WoS, the USPTO and EPO
databases.
Wolfgang Glänzel
Bibliometrics. Use and limitations
34/37
6.
The Steunpunt O&O Statistieken
BIBLIOMETRIC RESEARCH PROJECTS AT SOOS
Methodological and applied research is closely linked
to the service tasks, for instance, in the context of
methodological validation, of improvement of the
efficiency of indicators or of the subject classification.
‘Basic research’ at the Steunpunt is structured in a
systematic way to support policy-relevant research
and services and to contribute to the advancement of
the field of bibliometrics.
Similarly to the services, research is done at the
national/regional, institutional and research-group
level.
Wolfgang Glänzel
Bibliometrics. Use and limitations
35/37
5.
The Steunpunt O&O Statistieken
1. Development of a hierarchic subject classification
scheme for evaluation purposes
2. The role of author self-citations in scientific
communication
3. The myth of delayed recognition
4. Analysis of possible extension of bibliometric
methods to social science and humanities
5. Studying the network of science communication
6. “Indicator research” (improving journal and subject
impact measures and science indicators)
Wolfgang Glänzel
Bibliometrics. Use and limitations
36/37
6.
The Steunpunt O&O Statistieken
7. Bibliometrics beyond the WoS (e.g., proceedings
literature)
8. Bibliometric approach to social sciences and
humanities
9. Combining text-mining and bibliometrics
10. Mapping academic mobility and scientific
collaboration in Europe
11. Research group dynamics and their impact on
performance and visibility
12. Bisociation analysis of scientific communication
Wolfgang Glänzel
Bibliometrics. Use and limitations
37/37