Transcript Nessun titolo diapositiva - Issirfa-cnr

Valutazione nelle scienze umane e sociali
********************************
Evaluating Social Sciences and Humanities
CNR,
Aula Marconi
Piazzale Aldo Moro 7, 00185 Roma
4 aprile 2013
Giorgio Sirilli
ISSiRFA - CNR
Giorgio Sirilli
ISSiRFA - CNR
Our subject: Evaluation
Knowledge: the bundle
Knowledge
There are many different types of knowledge and knowledge holders.
Not all knowledge is scientific, much knowledge is practical and
contex specific
Society is interested in its:
Production
Transmission
Use
Research evaluation: the ladder
Knowledge
New public management
Evaluation
Research Assesment Exercise (RAE-REF)
Bibliometrics
CNR Department SSH
The learned society
University
teaching
research
“third mission”
Research agencies
research
problem solving
management
New Public Management
Catchwords
Accountability
Value for money
“Performance based research funding systems”
The rationale
“The rationale of performance funding is that funds should flow to
institutions where performance is manifest: ‘performing’ institutions
should receive more income than lesser performing institutions, which
would provide performers with a competitive edge and would stimulate
less performing institutions to perform. Output should be rewarded,
not input.”
Herbst, 2007
“Performance based research funding systems”
Performance based research funding systems are national
systems of ex-post university research output evaluation used
to inform distribution of funding
Share of university funding dependent on
“Performance-Based Research Funding Systems”
Criteria
Research (teaching excluded)
Evaluation ex-post (ex-ante excluded)
Research output
Research funding must depend on the results of evaluation
National system (internal university evaluations excluded)
Share of university funding dependent on
“Performance-Based Research Funding Systems”
Country
Share (%)
of what
Australia
6
Total revenue
Italy
2
Block grant
New Zeland
10
Block grant
Norway
2
Total funding
Slovak Republic
15
Total funding
UK
25
Research support
Hicks D., Reseach Policy (2012)
Share of university funding dependent on
“Performance-Based Research Funding Systems”
“The distribution of university research funding is something of an
illusion”
“It is the competition for prestige that creates powerful incentives
within university systems”
“Performance-based research funding systems aim at excellence:
they may compromise other important values such as equity and
diversity”
Hicks D., Reseach Policy (2012)
Evaluation of CNR Institutes
Carried out by CNR in 2009-2010
Evaluation of structures, not individuals
Methodology
- General panel
- Panels (26) composed of 156 scientists
- Documentation provieded by Institutes
- Site visit
- Final score
Lessons: “The General Panel recommended that in the future the two
groups of scientific areas (natural sciences and engineering, social
science and humanities) remain separated. Experience showed in
fact that differences in objectives, research methodologies,
evaluation criteria suggest to be deal with them separately”
ANVUR
VQR (Evaluation of the Quality of Research)
Habilitation of university professors
bibliometric areas
non-bibliometric areas
Both exercises largely based on bibliometrics
Knowledge: the bundle
When two types of evaluation do not match
Four types of publications
Internationally oriented journal articles
Books
National journals
Enlightenment literature
Hicks D., Research Policy (2012)
The four literatures
International
journal
literature
WoS/Scopus
Books
High impact
Enlightenment
National literature
Knowledge transfer
Local context
Hicks D., Research Policy (2012)
Other
means
of
transfer
Journal literature
Problem: missing output
Science
Journal articles – Social Science
counted in
WoS/Scopus
indicators
45-61%
81-85%
The 3 other
literatures –
not counted
Sources: Bourke et al,1996; Pestana et al., 1995; Winterhager, 1994
Other: books, edited
books, book chapters,
monographs, reports,
creative works
Norwegian social science – WoS coverage by field
Percentage of journal articles indexed in WoS
Sciences
Economics
Psychology
Political Science
Linguistics
History
Sociology
Business & Finance
Education
Theology & Religion
Philosophy & History of Ideas
Comparative Literature
Media & Communication
Law
Ossenblok et al. 2012
Ossenblok et al. 2012
0
20
40
60
80
100
National literature
Genuine scholarly research in
any area leads to
internationally relevant
results
Social sciences are embedded
in national contexts, national
issues are important, subtle
theory is developed in
national languages
Academics write in English
Social scientists more often
publish in their own language
Academics are international in Social scientists are more
orientation
oriented to their own national
literature than scientists
The problem with national literature:
Lower level of critique and peer
review, questionable quality
SSCI coverage of national
literatures inadequate
Enlightenment literature by field
Sciences
Burnhill & Tubby-Hille, 1994, UK social science
Research evaluation
NSE
SSH
Bibliometrics
Helpful
Not very helpful
Scientific literature
Internationally oriented
Nationally oriented
Peer review
Important
Very important
Habilitation of professors
Bibliometrics + peer
review
Peer review +
bibliometrics
Cost
Expensive
More expensive
Time
Limited
Longer
CNR Department of Cultural Identity and Heritage
Cultural identity
15 Institutes and 336 researchers/technologists
SSH
Cultural heritage
4 Institutes and 110 researchers/technologists
SSH + NSE
CNR Departments - Researchers and technologists by training
(percentage)
Cultural identity
10%
10%
38%
(15 Institutes)
42%
Cultural heritage
18%
(4 Institutes)
46%
33%
3%
Significant differences in specialisation of researchers between former Departments
CNR Departments – Research products (2007-2011)
(percentage)
Significant similarities in types of “products” between former Departments
CNR Departments – Research products (2007-2011)
(percentage)
Structural difference between SSH Departments and a NSE Department
CNR Departments – Journal articles (2008-2010)
36%
46%
11%
23%
53%
31%
Structural difference between former Departments in terms of output
Contributo in rivista
Articolo in rivista
Traduzione in rivista
Nota a sentenza
Scheda bibliografica
Abstract in rivista
Abstract/Comunicazione in rivista
Abstract/Poster in rivista
Rassegna e recensione
Contributo in volume
Contributo in volume (capitolo o saggio)
Traduzione in volume
Prefazione/Postfazione
Breve introduzione
Voce (in dizionario o enciclopedia)
Scheda di catalogo
Contributo in atti di convegno
Contributo in atti di convegno
Abstract/Poster in atti di convegno
Comunicazione a convegno
Abstract/Poster in convegno
Abstract/Comunicazione in atti di convegno
Libro
Monografia o trattato scientifico
Traduzione di libro
Edizione critica
Commento scientifico
Concordanza
Indice
Pubblicazione di fonti inedite
Antologia e manuale scolastico
Dizionario e thesaurus
Bibliografia, repertorio, glossario
Catalogo monografico
Manuale tecnico
z
Curatela
Curatela
Brevetto
Brevetto
Altra tipologia
Composizione
Performance
Banca dati
Software
Cartografia
Commento giuridico
Rapporti progetti di ricerca
Rapporti tecnici, manuali, carte geologiche e tematiche e
prodotti multimediali
Rassegne, recensioni, prefazioni
Altre pubblicazioni
Rapporti finali progetti di ricerca
Materiale didattico
Rapporti di commissioni
Rapporti statistici
Note interne, manuali e guide
Prodotti multimediali
Risultati di valorizzazione applicativa
Progetti
Disegni e design
Rappresentazioni
Mostre ed esposizioni
Manufatti, prototipi d'arte e relativi progetti
Rapporti tecnici/preprint/working paper
Data set
Altro prodotto
25
Other products: 12 out of 25 (2001-2011)
Market oriented
Academically oriented
“Products” for the “market” are marginal: other categories should be added
CNR products in data base
International
journal
literature
Books
Other
means
of
transfer
National literature
Enlightenment
Evaluation of SSHs: some comments
Knowledge is our concern
Do not forget the knowledge bundle and its various facets
International Classification of Occupations (ISCO)
2113 0-11.10 Chemist
2113 0-11.50 Chemist, analytical
2113 0-11.40 Chemist, corrosion
2113 0-11.40 Chemist, crystallography
2113 0-11.20 Chemist, detergents
2113 0-11.20 Chemist, dye
2113 0-11.20 Chemist, food
2113 0-11.30 Chemist, glass
2113 0-11.30 Chemist, inorganic
22 leather
2113 0-11.20 Chemist,
2113 0-11.30 Chemist, metallurgical
2113 0-11.40 Chemist, nuclear
2113 0-11.20 Chemist, organic
2113 0-11.20 Chemist, paint
2113 0-11.20 Chemist, petroleum
2113 0-11.90 Chemist, pharmaceutical
2113 0-11.40 Chemist, physical
2113 0-11.20 Chemist, plastics
2113 0-11.20 Chemist, polymer
2113 0-11.50 Chemist, quality control
2113 0-11.20 Chemist, rubber
2113 0-11.20 Chemist, textile
2111 0-12.10 Physicist
2111 0-12.50 Physicist, acoustics
2111 0-13.50 Physicist, astronomy
2111 0-12.80 Physicist, atomic
2111 0-12.20 Physicist, ballistics
2111 0-12.60 Physicist, electricity and magnetism
2111 0-12.70 Physicist, electronics
2111 0-12.30 Physicist, heat
2111 0-12.20 Physicist, hydrodynamics
19 light
2111 0-12.40 Physicist,
2111 0-12.20 Physicist, mechanics
2111 0-12.80 Physicist, molecular
2111 0-12.80 Physicist, nuclear
2111 0-12.50 Physicist, optics
2111 0-12.20 Physicist, rheology
2111 0-12.90 Physicist, solid-state
2111 0-12.50 Physicist, sound
2111 0-12.90 Physicist, theoretical
2111 0-12.30 Physicist, thermodynamics
3415 4-31.20 Representative, sales/technical
XXXXXXX Researcher: MISSING
1315 5-10.30 Restaurateur
Evaluation of SSHs: some comments
Knowledge is our concern
Do not forget the knowledge bundle and its various facets
Why evaluate?
Eveluation is a complex business, and bibliometrics as a shortcut
Much information on outputs of SSH is not covered by bibliometrics
SSHs are structurally different from NSEs: different types of evaluation
Extending the SSH coverage of literature is OK, but there are inherent
limits
Significant differences amongst CNR SSH Department Institutes
deserve further analysis
More experimentation needs to be done in SSH: in the meantime be
cautiuous
Research evaluation in SSH
Thank you for attention