Five Myths

About Funding Scientific Research (in Austria) ….

and what Evaluation can do to make them more ‘Evidence Based’.

Five Myths

• „No Money for the Humanities and Social Sciences in Austria!“ • „There is a funding gap between basic research and applied research!“ • „Engeneering is treated unfair!“ • Networks, Fretworks • Impacts – now!

Myth 1:

„No money in the humanities / social sciences“

• Humanites get a raw deal… (Der Standard, 31. Mai 2005) • Social Sciences and Humanities are „starved out“ financially (Austrian Green Party, 13. Mai 2005) • …marginalisation of the Humanities…“.

(M. Nießen, DFG)

Mapping the Social Sciences & Humanities in Austria…

• No Evaluation / No Benchmarking Exercise in the field • Lack of data – Contract research of the ministries?

• No vivid programme scene • … but looking at further empirical evidence…

R&D in the higher education sector, 2002

R&D personnel number of R&D units headcount FTE R&D expenditures in 1000 EUR Natural sciences Technical sciences Medicine (incl. clinics) Agriculture, Forestry & Veterinary Medicine Social Sciences & Humanities >> Social Sciences >> Humanities 197 173 144 44 411 208 203 6.469

3.502

7.284

1.060

6.757

3.775

2.982

4.865,2 2.690,6 6.025,6 847,5 4.993,8 2.718,4 2.275,4 387.193

173.493

333.516

70.089

301.813

165.755

136.058

Source: FTB 2005

Contract Research, 2003

Natural sciences Technical sciences Medicine (incl. clinics) total sum 11.099.561

in % 19,6 7.472.237

13.264.064

13,2 23,5 bm:bwk 8.794.489

1.686.947

12.848.845

bm:vit 617.055

5.214.199

89.969

bm:wa 187.097

132.180

Agriculture, Forestry & Veterinary Medicine Social Sciences & Humanities >> Social Sciences >> Humanities 2.997.521

21.698.122

14.735.356

6.962.766

5,3 12,3 464.558

38,4 18.189.699

26,1 11.226.933

6.962.766

1.129.160

1.129.160

10.433

950.817

950.817

Source: FTB 2005

FWF-project funding Acceptance Rates, 1998-2003

• Highest Acceptance Rate – Natural Sciences and Humanities: 58% • Lowest Acceptance Rate – Agriculture and Social Sciences: 35% • Funding Rates – Quite homogeneous – 70 % Human Medicine – 80% Humanities Source: Streicher 2004

A „Benchmarking Exercise“

• Benchmark Project – Natural Sciences, male co-ordinator – Age 40-50, Size 150 – 250 k€ – Approval Rate: 52,4% Variable Technical Sciences Human Medicine % Difference in approval rate - 8,5 -15,1 Agriculture, Forestry, VetMed Social Sciences -18,1 -19,2 Humanities + 4,5 (A cautionary remark: It would be wrong to interpret the co-efficents causally) Source: FWF Evaluation, Streicher, Schibany 2004

A „Benchmarking Exercise“

• Yes, projects of a different scientific flavour face significantly different chances – Against a benchmark project, Social Sciences are rejected far more frequently – Humanities are (slightly but significantly) more successful Source: Streicher 2004

Take into account…

• Classification • Structural Issues – Age?

– Fragmentation of Research Units?

– Perspectives for younger researchers?

– Researchers = working poor?

• Quality – Kind of Indicators • ……

Heterogeneous average working loads

(in % of total working hours) Natural sciences Technical sciences Medicine (incl. clinics) >> without clinics >> clinics Agriculture, Forestry & Veterinary Medicine Social Sciences & Humanities >> Social Sciences >> Humanities teaching & training 29,5 31,3 16,8 24,7 14,0 25,6 45,0 43,8 46,5 R&D 64,4 61,5 36,7 65,8 26,3 57,0 47,4 48,5 46,1 other tasks 6,1 7,2 46,5 9,5 59,7 17,4 7,6 7,7 7,4 Source: FTB 2005

Conclusions • There is never enough money for doing research – No evidence, that Humanities / Social Sciencies are treated unfair • „Not enough money for the humanities / social Sciencies?“ – This is an urban legend

Challanges for the future:

• Evaluators – Evaluators should be • Sceptical, • suspicious of everybody – Triangulation is necessary!

– Quantitative Methods are valuable sources of information • Stakeholders – Ask the big questions (from time to time), too.

– Give the evaluators the degrees of freedom to answer these questions.

Mythos 2:

Funding Gaps between basic and applied sciences

„Funding Gap“ Basic Sciences Applied Sciences

Dream Nightmare Reality

Risk Aversity & FFF

Source: FFF Evaluation, Arnold 2004 Source: FFF Evaluation, Jörg 2004 Source: FFF Evaluation, Arnold 2004 • Overall [FFF] tends to take too little risk.

• FFF funding practice is risk-averse.

• [The linear model] is a misleading oversimplification that encourages us to make poor policy decisions.

FWF Projects: Commercial Output & Usability

41%: results are relevant for industry 30%: important lab results 20%: working prototypes exist 13% research results are suitable for commercialization straight away Source: FWF Evaluation, Streicher, Schibany 2004

101.51

~ 70 Millionen € CDG K-Ind / Knet Kplus 7.3 (2003) K-Ind ?

18.9

Informationstechnologien: FIT-IT Nanotechnologie: Nano-Initiative Luftfahrt: TAKE OFF Weltraum: ASAP & ARTIST Verkehrstechnologien: ISB & A3 BRIDGE: Translational Research & Brückenschlagprogramm 5.9

10.8

3.5

11.6

8 5.11 (2004 Translational FWF) 127.15

Basic Sciences Applied Sciences

Conclusions

• There is no funding gap (anymore) • „Funding gap: No guiding principle for policymakers (anymore) • Funding Gap: Urban Legend II Source: mid term Evaluation FIT-IT, 2005

Challenges for the Future I

• In a NIS, there is the need for Evaluation of Systems

(from time to time)

• In a NIS, there is the need for Evaluation of Portfolios

(from time to time)

Challenges for the future II

• Room for „curiosity driven Evaluation“ • Methodological Development – Evaluation is no pure science, but – It is no consulting business, too.

– Of cause, Evaluation must have a sound scientific basis • Ensure degrees of freedom – Budget!

– TORs • Fight Evaluation Fatigue – Realistic expectations – sufficent time spans

Next Steps

Paper, part of the conference….

„New Frontiers in Evaluation“ www.fteval.at/conference06

24./ 25. April 2006 Vienna, Austria

Team

Michael Dinges, Joanneum Research Michaela Glanz, WWTF Brigitte Tempelmaier, WWTF