Transcript Bibliometrics

Bibliometrics:
coming ready or not
CAUL, September 2005
Cathrine Harboe-Ree
Information kit


Introduction
Assessing Quantitative Performance Indicators
Workshop
– Reading list
Citation Analysis in Research Evaluation, Henk F
Moed
 Managing Australian Research Output for
Increased Return on Investment: the Role of
Open Access Institutional Repositories, Cathrine
Harboe-Ree
 Research Indicator Databases available from
Thomson Scientific (ISI)

Introduction
What is bibliometrics?
 Why is bibliometrics important?
 What’s bibliometrics got to do with us?

Quantitative Performance
Indicators Workshop

ARC Linkage project
– Range of performance indicators to assess research
performance
– Assessment of each indicator for validity, etc
– Evaluation of different performance measures on
institutions, groups and individuals
– Applicability of field-specific weights to counter
inequalities



ANU, UQ, DEST
Chief Investigator – Linda Butler
16 May workshop to test various indicators
Quantitative Performance
Indicators












Citations to non-source publications
Journal publications classified by impact quartiles
Identification of highly cited publications
Book publications weighted by publisher prestige
Honours, awards and prizes
Election to learned academies and professional academic
associations
Conferences
Service to journals
Visiting Fellowships
Office Bearers in learned academies and professional
academic associations
Copyright Agency Limited (CAL) data
Membership of grants committees
Quantitative Performance
Indicators findings








Number of publications as the sole indicator is not valid
Some sort of peer review process is desirable
Several indicators should be used in combination
Multiple indicators should not be aggregated to a single
number
Quality measures have to be responsive to their fields
Individual or small unit assessment is expensive and
inappropriate
Journal impact factors of the ISI type are highly
inappropriate
Process/technical errors can develop over time,
invalidating all data
Bibliography
Linda Butler’s ARC reference list
 Henk F. Moed Citation analysis in research

evaluation
Henk Moed: validity elements







Formal
Open
Scholarly founded
Supplemented with expert knowledge
Carried out in clear policy context
Explicit statement of basic notions of scholarly
quality
Enlightening rather than formulaic
Henk Moed: Thomson ISI
Coverage excellent in physics, chemistry,
molecular biology, biochemistry, biological
sciences, clinical medicine
 Good in applied and engineering sciences,
biological sciences, geosciences, mathematics,
medicine and health related social sciences
 Moderate in sociology, political science,
anthropology, education and humanities
 Affected by

– non-source material (books, conference proceedings
– Language and national barriers
Henk Moed: findings










Alternative data sources and methodologies should be
explored
The expertise of specialists is important
There is no single ‘perfect’ indicator of journal performance
Thomson ISI journal impact factors are not a reliable guide
Data accuracy is a key problem
The substantive content of work should be taken into account
The level of aggregation is crucial
Both citation analysis and peer review have strengths and
limits
A study of RAE questions its ability to identify ‘top’ research
departments
Including more sources (ie web) does not necessarily lead to
more valid assessment of scholarly contribution
Open access impact





Strong consistent evidence that open access
availability has a significant impact on citation rates
(50-250%)
Citations themselves are not a measure of quality
Scholars value citedness
RQF could consider it more
University ranking systems using journal impact
factors
– Should use citation factors

Tension between ‘quality’ and ‘accessibility’ and
“impact”
Thomson Scientific databases







Essential Science Indicators
Institutional Citation Reports
University Science Indicators
National Science Indicators
High Impact Papers
Journal Citation Reports
Web Citation Index
Thomson Scientific databases

Linda Butler
– Uses raw data
– Cleans it
– Maps to RFCD codes
– Cautions against use by people who don’t
understand the data
– Is concerned that there are not enough
people who can interrogate the data