Transcript Models of Evaluation For Research Proposals in Turkey
Models of Evaluation For Research Proposals in Turkey
Prof. Dr. Omer CEBECI Vice President – Funding, TUBITAK (The Scientific and Technical Research Council of Turkey) & S. GENC, A. KERC, H. KARATAS, A. FEYZIOGLU, F. COSKUN, O. OZPEYNIRCI, B. DIKMEN, G. KOZANOGLU, H. GULER & O. KARA
SUMMARY
• Research funding in Turkey, TUBITAK & recent boost • Three-dimensional evalution used by TUBITAK for the evaluation/selection of research project proposals • (1) curiosity driven academic research, and • (2) customer driven applied research, • both in universities and research institutions, and • (3) technological and innovation driven research conducted by the private industry, plus • (4) research equipment & infrastructure.
• Details of the “Phrase-anchored rating scale”
Actors of Turkish National Science and Technology System (policy makers) BTYK- Supreme Council of Science and Technology TAEK- The Turkish Atomic Energy Commission TOBB-Union of Chambers and Commodity Exchange of Turkey The Turkish Academy of Sciences (TÜBA) KOSGEB- Small and Medium Sized Industry and Development Organizations DPT- State Planning Organization TUBITAK-The Scientific and Technical Research Council of Turkey MIT-Ministry of Industry and Trade ME-Ministry of Education HEC-Higher Education Council (YÖK) TTGV- Technology Development Foundation of Turkey TÜRKAK- Accreditation Board DTM-Foreign Trade and Treasury
Actors of Turkish National Science and Technology System (science and research performers) Turkish Industry Turkish State Universities TUBITAK National Research Centers and Institutes
Marmara Research Center (MAM)
Institute of Energy
Chemistry and Environment Institute
Food Institute
Materials Institute
Information Technologies Research Institute
Earth and Marine Sciences Research Institute
Information Technologies and Electronics Research Institute (BİLTEN)
National Electronics and Criptology Research Institute (UEKAE)
Defence Industry Research and Development Institute(SAGE)
Basic Science Research Institute
National Academic Network Center
Genetic Engineering and Biotechnology Research Institute National Metrology Institute
Turkish Industry Management Institute
National Observatory Other National Research Centers and Institutes Turkish Private Universities and Research Centers
Turkish National Science, Technology and Innovation System President of Republic Prime Minister
BTYK MEB Universities STB YÖK DIE DTM DPT TÜBA TÜBİTAK KOSGEB TPE TOBB TTGV TSE TÜRKAK R&D Institutions BTYK- Supreme Council of Science and Technology STB- Minister of Industry and Trade MEB- Ministry of National Education TÜBİTAK- The Scientific and Technological Research Council of Turkey YÖK- Higher Education Council (YÖK) TÜBA- The Turkish Academy of Sciences (TÜBA) DPT- State Planning Organization TÜRKAK- Accreditation Board
TPE- Turkish Patent Institute TSE- Turkish Standards Institution DIE- State Institute of Statistics DTM- Undersecretariat of the Prime Minister for Foreign Trade TOBB-Union of Chambers and Commodity Exchange of Turkey KOSGEB- Small and Medium Industry Development Organizations TTGV- Technology Development Foundation of Turkey
National S&T Indicators
Number of triadic patents per million population 2,5 Total R&D personnel per 1000 employment Japan 2 1,5 1 0,5 0 Germany US Number of papers per million population EU-15 GERD as a percentage of GDP GERD per capita population EU-15 Values (2000-2001): Number of triadic patents per million population:36 Total R&D personnel per 1000 employment: 10.4
GERD per capita population:467. 6 (PPP $) GERD as a percentage of GDP:1,89 Number of papers per million population:822
National S&T Indicators
Total R&D personnel per 1000 employment Number of triadic patents per million population 1 EU-15 Hungary 0,5 Spain Number of papers per million population GERD as a percentage of GDP Turkey GERD per capita population
Scientific Publications
1,20 1,10 1,00 0,90 0,80 0,70 0,60 0,50 0,46 0,40 1998 0,52 1999 0,57 2000 0,64 2001 Year 0,86 2002 0,94 2003 1,11 2004
Knowledge as a Tool
Knowledge is a tool that can be used for a variety of social objectives, including: Meeting Basic Human Needs Increasing Safety & Security Improving the Quality of Life Economic Growth and Development
Knowledge - R&D Cost per Kg
Concrete Cement Steel Aluminium Automobile Passenger Aircraft Fiberoptical Cable Combat aircraft Communication Satellite Microprocessor (Si) Chip 1 cent 5 cent 50 cent 1,5 $ 10 – 100 $ 100 – 1.000 $ 3.000 $ 10.000 $ 100.000 $ 4.000.000 $
Share of TUBITAK in TR Fiscal Budget
50 45 40 35 30 25 20 15 10 5 0 1971 1973 1975 1977 1979 1981 1983 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005
Research Proposals
*from universities and research institutions: 1) curiosity driven academic research 2) customer driven applied research *from private industry: 3) technological & innovation driven research
TUBITAK Research Project Proposals
Science and Technology Project Proposals 3000 2500 2000 1500 1000 500 0 1999 2000 2001 2002 Years 2003 2004 Career International Sector Units Infrastructure Research
Running Research Projects
3000 2500 2600 2000 1500 1000 500 264 430 587 694 750 885 889 929 899 833 837 982 1212 1199 1353 0 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
EVALUATION & MEASUREMENT
TARİHTEN BUGÜNE SAĞLIK ve ÖLÇME
• Ateş ve nabız – 1625 (Santario ve Galileo) • Torricelli - 1643 • Tansiyon - 1733 (Hales) • 1896 - Riva-Rocci • Korotkov - 1905 • Ateş - nabız - tansiyon ölçüm ve kayıtları ile hasta izleme - 1920
ÖLÇME ve DEĞERLENDİRME
“ Ölçemediğinizi anlıyamazsınız ! ”
Lord Kelvin (1824 - 1907)
(If you can’t measure it you don’t understand it)
“Her hesaba katılması gerekenin sayılır olması gerekmez ! “
Albert Einstein (1879 - 1955)
(Not everything that can be counted counts & not everything that counts can be counted)
Çıktı mı ? Sonuç mu ? Etki mi?
Çıktıların sonucu olarak gerçekleşen
ETKİ
!
OUTPUT – OUTCOME – IMPACT IMPACTS achieved as consequences of the OUTCOMES resulting from the realisation of the OUTPUTS
OUTPUT – OUTCOME – IMPACT IMPACTS achieved as consequences of the OUTCOMES resulting from the realisation of the OUTPUTS ( My eye jelly project )
Intended / Expected / Desired IMPACT : . . . in full control of my mandate . . .
Unintended / Unexpected / Unwanted BUT ACTUAL REAL IMPACT : . . . under strict control of my wife ! . .
Evaluation Criteria
three-dimensional
evaluation criteria developed in collaboration with
researchers & reviewers
1) Curiosity Driven Academic Research - the three dimensions -
1. intellectual/scientific/professional merit 2. expected impact of the anticipated outcome 3. achievability of the research with the proposed research team, equipment/facilities and methods
2) Customer Driven Applied Research - the three dimensions -
1. research and development merit 2. expected impact of the anticipated outcome 3. achievability of the research with the proposed research team, equipment/facilities and methods
3) Technological & Innovation Driven Research - the three dimensions -
1. technological level of the research 2. innovative level of the product/outcome 3. feasibility of the process
Evaluation Process
all three-criteria are given equal weights
Phrase-Anchored Rating Scale
versus Likert Scale sub-criteria phrases describing
*very competitive *competitive *not-competitive
features of the proposal
Very competitive
*scientifically and professionally outstanding & very well justified project & *points to an opportunity for a major contribution to the advancement of the knowledge and/or to the resolution of a problem of practical importance
Competitive
*scientifically and professionally competent and justified proposal which will make a contribution to the advancement of knowledge &/or the resolution of a problem of practical value & therefore *support is suggested if funds are available
Not competitive
*work routine in character *scientifically and professionally unsatisfactory and poorly organized
Evaluation Process
5-8 Individual Reviewers
evaluate 10-15 proposals by referring to the
Panels a meeting for a final verdict
Curiosity / Merit = V. Comp.
in journals or books listed in international • The originality of the work has been supported by extensive and critical literature • Hypothesis for evaluating the research topic is very well defined • Explanation and analysis of the expected
Curiosity / Merit = Not comp.
• Scientific consistency and the rationale of • A clear scientific / technological question is not put forward. • Project is more like an investigation / data collection / routine work.
literature survey does not point out the basis / importance of the project.
hypothesis.
Curiosity / Impact = V. Comp.
development of the country /very likely to be problems of society • Very likely to be employed in different scientific & technological fields • Very likely to generate new projects • Commercialization potential of the outcomes is very high • The project is supported by international,
Curiosity / Impact = Not comp.
• Potential for adding value to science and technology is low • Subject of the project is not among the priorities of the country • Not likely to result in intellectual property worth-protecting.
Curiosity / Achiev. = V. Comp.
Project Team
• The team is experienced in national / international projects related with same/similar subject • They have experience as advisors /authors / referees / editors / book authors • The can allocate enough time for the project • Competencies / responsibilities / roles of the team members are well defined and adequate • End users of the project outcomes are also members of the team.
Curiosity / Achiev. = Not Comp.
Project Team
• The team is not experienced in conducting projects of this size • The knowledge and awareness of the team is not sufficient • They do not have important publications in the subject of the project • Some of the team members are irrelevant for the project • Essential competencies are lacking
Curiosity / Achieve. = V. Comp.
Infrastructure
• Infrastructure of the institution is very adequate for the project • Additional equipment requested within the scope of the project is very compatible with the existing infrastructure and the project • Existing sources / equipment are used rather than purchasing new sources • Requested equipment can also be used in other / future projects
Curiosity / Achieve. = Not Comp.
Infrastructure
• Infrastructure of the institution is not adequate for the project, unless supported with major equipment • Equipment requested within the scope of the project is not compatible with the existing infrastructure and the project
Curiosity / Achieve. = V. Comp.
Methodology
• Approach / methodology are very well designed to reach the target • Methodology is correct and well-defined; and standard methods and literature are cited • Preliminary experiments have been conducted to rationalize the hypothesis • Alternatives (plan B) have been considered if difficulties are encountered
Curiosity / Achieve. = Not Comp.
Methodology
• Methodology is not adequate to reach the target • Relations between the experiments and hypothesis are not well defined • Methodology is not explained with a common scientific basis considered considered
Curiosity / Achieve. = V. Comp.
Timeline
• Proposed period & time schedule are realistic
Budget
• Proposed budget is realistic and well-justified • Project is also supported by other institutions
Curiosity / Achieve. = Not Comp.
Timeline
• Proposed period and time schedule are not synchronized • Time schedule is not adequate
Budget
• Budget is not well-define and requested amount is too low / high • There is no possibility of support from other institutions
Customer / Merit = V. Comp.
• Aims to develop a national / international novel technology (methodology, system, product, process / technique) • Brings comparable superiority to the existing system • Outcome will be a technology to be protected under the intellectual property rights • Work consists of a scientific and rationale approach • An interdisciplinary project with the collaboration of the Public- Private Industry University-Research Institutions
Customer / Merit = Not comp.
• Literature survey and market survey are not satisfactory, work is based on unreliable data • No scientific / technological rationale and integrity in the project • It is a study that had been done before in the country • Aim, objective and motivation are not clear • Outcome is not qualified as “applicable/usable” • Not related with a real need / problem
Technological Level = V. Comp.
• Technology / product developed aims to fill a existing technology within the following 3 years • Has an interdisciplinary approach to solve more than one problem • Rationale of the R&D is well established (theoretical / analytical / experimental) • Added value of the anticipated outcome of the R&D project is considered • Has contribution to increase R&D staff • A doctorate / masters study is incorporated potentials to be published in national/international journals.
Technological Level = Not comp.
• Literature survey and market survey are not adequate • No scientific/technological advance and integrity in the project • R&D rationale (analytical and /or experimental) is not adequate • Aim, objective and motivation are not clear. Procedure that has been used will not provide a change / improvement in the methodology and technology
Innovation Level = V. comp.
• Outcomes may lead to spin-off company • Outcomes include production standards and technical specifications • Outcomes will affect other sectors as well • Outcomes very likely to increase the export capacity of the country • Very likely to be a basis for generating new projects / products
Innovation Level = Not comp.
• Outcome is not qualified as “applicable/usable” • Period of usefulness is very limited or not effective • Does not provide new areas of work and as a result does not increase employment • Expenditure for the R&D study is far beyond the expected economical benefit of the product • Support provided by the private organization is not sufficient.
Comments
Reviewers (hundreds): • The model reduced the, – burden of the review process on them, & – subjectivity and variability of the opinions • • of the individuals Convenience of selecting from a comprehensive list of phrases as well as the freedom of offering their own judgments Researchers (thousands): • Sub-criteria phrases to be very instrumental in guiding the development of their proposals
Thank you
Hundreds of scientists, researchers and reviewers participated in workshops for the development of the criteria and phrases listed in the tables. Their contributions are gratefully acknowledged.