Transcript Strategic Evaluation

Outcome-based Strategic Evaluation
of Research Units in AIST
National Institute of Advanced
Industrial Science and Technology
(AIST)
Naoto Kobayashi
Vice President
Director, Evaluation Department
@Joanneum Research in Vienna on Feb. 12h, 2007
1
CONTENTS
1.Strategic Research Evaluation
2.Missions and Activities of AIST
3.Strategy and Evaluation
4.Evaluation System from the Outcome
Point of View
5.PDCA cycle in the Research Institute
2
Strategic Research Evaluation
Strategic Research Evaluation；
Research Evaluation Procedure and Implementation
Suitable for Strategic Promotion of R&D
◆Strategic Design and Implementation of Evaluation
◆Strategic Reflection of Evaluation Results to R&D
◆Strategic Planning and Effective Promotion of R&D
3
Research Organization Evaluation System
DIET, CSTP
P
P
P
A
C
A
Evaluation Committees for IAI’S
A
R&D Administration (METI)
C
Evaluation Committee for AIST
A
Research Institute（AIST）
Management
P
C
R&D Administration (MIC)
Action
Check
Plan
Planning Research Units
Do Research
CSTP;Council
for S&T Policy
C
Evaluation
D
MIC;Ministry of
Internal Affairs &
Communications
METI;Ministry
of Economy,
Trade & Industry
IAI;Independent
Administrative
Institution
4
CONTENTS
1.Strategic Research Evaluation
2.Missions and Activities of AIST
3.Strategy and Evaluation
4.Evaluation System from the Outcome
Point of View
5.PDCA cycle in the Research Institute
5
Missions of AIST
Missions of AIST: for all mankind and global consciousness to our future
The mission of AIST is to generate new technologies and new industries and to
contribute to the economic development of our country and, thus, to improve the
lives of people. With this mission in mind, AIST has been performing researches
in the following areas .
Interdisciplinary Researches
to enable to plan long-range
governmental policies
attributing to the current
and future needs of society
Environment
Energy
Resources
Standard of measuring
& weighing
Oceanography
Geology
Life science
Information
Communication
Electronics
Manufacturing
Machine
Nanotechnology
Material
Type-II Basic Research
Full Research
Advanced Researches
to explore broad spectra of
research fields and to integrate
multidisciplinary subjects
Basic Researches
to develop and maintain high standards of scientific and engineering research
6
Organization of AIST
（ as of July 1, 2006 ）
◆Top-down management by president of AIST
◆ Flexible reorganization of research units
◆ Complete research support by administrative units
Planning HQs
General Administration HQs
Evaluation Dept
Safety & Environmental Protection
Dept
Auditor
President
Public Relations Dept
AIST Advisory Board
Legal Office
Senior Vice-President
Vice-President
Information Disclosure & Personal
Information Protection Promotion
Office
AIST Fellow
Research Coordinator
Gender Equality Office
Emeritus Advisor
Audit Office
Emeritus Councilor
(57)
Research Centers
Research Institutes
(29)
(21)
Research Initiatives
(7)
Tsukuba Advanced Computing Center (TACC)
International Patent Organism Depositary
AIST Innovation Center for Start-ups
Geo-information Center
Metrology Management Center
Technology Information Dept
Collaboration Dept
administrative units
Research Units
Intellectual Property Dept
International Affairs Dept
General Administration Dept
Human Resources Dept
Financial Affairs Dept
Research Facilities Dept
7
産総研
Research Bases of AIST
old AIST
AIST Hokkaido
15 Research Institutes
8 Institutes in Tsukuba
7 Institutes in other regions
In 2001
new AIST
AIST Kansai
Merged into 1 Institute
9 Research Basis
AIST Chugoku
AIST Tohoku
AIST Tsukuba
AIST Tokyo Waterfront
Tokyo Headquarters
AIST Kyushu AIST Shikoku
AIST Chubu
8
AIST: Staff
<Number of Employees>
(as of April 1, 2006）
Total number of employees 3,209
Researchers
2,505
Administrative staffs
704
Executives etc
11
Number of Visiting Researchers at AIST
(in 2005 FY)
・Postdoctoral researchers
650
・From private companies
800
・From universities 2,000
・From corporation etc
950
・From overseas
850
Geological Survey &
Applied Geoscience
Metrology & 11%
Measurement
Technology
Life Science &
Technology
18%
Information
Technology,
Electronics
15%
Environment &
Energy
16%
23%
Nanotechnology,
Materials &
Manufacturing
Statistics
17%
9
Full Research
AIST defines Type-II Basic Research, which integrates multiple disciplines and
creates methods for the use of integrated knowledge; traditional basic research
is defined as Type-I Basic Research, which is the pursuit and discovery of
novel laws, principles and theories that govern natural phenomena. AIST
places its highest priority on the pursuit of complete research, “Full Research”,
ranging from Type-I Basic Research to the development of “products” by
conducting intensive Type-II Basic Research.
10
Characterization of Research Units
1. Research Center (29)
-conducts pioneering & strategic projects within
a certain period (<7 years)
-priority in investing research resources is given
-operated by top-down management
2. Research Institute (21)
-conducts mid & long term research
-research themes are proposed in a bottom-up manner
3. Research Initiative (7)
-promotes formation of new research fields and/or
research center
-contributes to sudden administrative needs
（ as of July 1, 2006 ）
11
CONTENTS
1.Strategic Research Evaluation
2.Missions and Activities of AIST
3.Strategy and Evaluation
4.Evaluation System from the Outcome
Point of View
5.PDCA cycle in the Research Institute
12
Future Visions and Research Field
AIST Goals and Related Research Fields
13
Research Strategy
AIST Research Strategy
- Directional Vision toward Innovation • Overall Strategy
–
–
–
–
Basic Principle: To realize sustainable society
Four missions
Overall Goals (5 outcomes and 2 practices)
Policies to be specified
• Strategy in each research field
– Strategic goal
– Strategic research subject
– Focal research subject
• Strategy with integrating different
research fields
• Innovation Hub Strategy
→ Definite description as our long-term guideline
of research
Concrete research goals and plans
・・・”2nd Mid-term Plan”
（”Contract” between AIST and Minister in Charge)
14
Purpose and Viewpoint
Purpose and Viewpoint of Research Unit Evaluation in AIST
Purpose:
Activation & increasing in efficiency for research unit
Reference for top-management of AIST
Accountability to people for activities of AIST
Point of View:
Promotion of
1) evaluation from the qualitative viewpoint of research
2) evaluation from the viewpoint of “full research” and
outcomes related to AIST strategy
15
Evaluation of Units
METI
Five-year target
President
(AIST)
Five-year Plan
Evaluation
Committee
for IAI in METI
Resource allocation,
Report
Annual Report
Evaluation of Activity
Evaluation
Dept
Research Unit
Evaluation
Reorganization
Output Evaluation
（Every other year）
Evaluation
Start-up Evaluation
（The Ist year）
Report
Research Units
Administrative Unit
Evaluation
（Every other year）
Evaluation
Report
Administrative Units
16
Evaluation Interval
Research Center
Start-up eval.
Output eval.
Output eval.
Output eval.
-top down
-time limit
Intermediate Review
Advise for
Roadmap
Research Institute
-bottom up
-no time limit
2nd term’s
start-up eval.
Output eval.
Output eval.
Advise for
Roadmap
-to be Research Center
Intermediate Review
Output eval.
Research Initiative
-time limit
Final Review
Advise for
Roadmap
Intermediate Review
A new Research Center
Review & reorganization of research unit
17
CONTENTS
1.Strategic Research Evaluation
2.Missions and Activities of AIST
3.Strategy and Evaluation
4.Evaluation System from the Outcome
Point of View
5.PDCA cycle in the Research Institute
18
Definition of Output, Outcome in AIST
Output:
Direct results in R&D (papers, patents, etc)
Outcome:
Social and economical effects caused by output
(products, standards, novel research fields, etc)
Input
・Budget
・Human
resource
・Equipment
Output
・Academic paper
・Patent
・Geologic Map
・Standard
・Software
・Process Develop.
・Database
Outcome
Impact
・Popularization
・GDP growth
of products
・New employment
・New research
・International
field
competition ability
・Service
・High QoL
・Saving energy
・Environmental
・Risk management protection
・Disaster
・Security & safety
prevention
society
19
Evaluation from the Outcome Point of View
（by Research Evaluation Committee in AIST）
① Evaluation of Realized Outcomes
② Evaluation of Scenario/Roadmap toward Realization of Outcomes
③ Evaluation of Outputs to contribute to Outcomes
④ Evaluation of Management to contribute to Outcomes
Past
Outputs in the past
Present Research
Future
① Realized Outcomes
② Roadmaps
③ Outputs
④ Management
Expected Outcomes
in the Future
Performance Evaluation: ②Roadmap, ③Output, and ④Management
Startup Evaluation: ② Roadmap and ④Management
20
Example of Roadmap for Outcomes:
2002
Subjects
(Targets)
2004
2006
2008
4 inch Wafer
> Few 10μm/h
2010
High-reliable SiO2
Self-Alignment
R&D in AIST
Low-Temp Process
Wire-less
（ＦＥＴ）
（ＩＧＢＴ）
600V10A
600V100A
Low-EMI
Heatproof
Lamp Ballasts
Applications
2020
6 inch Wafer
Bipolar-Tech.
T j >250℃
Packaging
FET Devices
Modules
2012
Multi-Wafer-Epi.
low-Ron
Processing
（Power Electronics Research Center）
The 2nd Term
The 1st Term
Substrate / Wafer 3 inch SiC Wafer
Epitaxy
Ultra-Low-Loss power devices
Air-Con.
2.5kV100A
Bipolar Devices
Technology transfer
Capacitor-less
Dispersed
Power supply
General
Inverter
Factory
Motor drive
Power Transmission
System
Instruments
Electric Power Distribution System
21
Output Evaluation
External Reviewers:
Internal Reviewers:
Roadmap, Output in each
main research theme
Management, Roadmap of
main research themes
Research Unit
AIST Strategy
＊Evaluation of Roadmaps
＊Evaluation of Outputs
＊Evaluation of Management
OUTPUT
EVALUATION
Report , Presentation
＊Roadmap
＊Output
＊Management
Evaluation : Rating
with Comments
Evaluation Dept
Report
People
Open
Comments
President of AIST
Reflection on
Resource Allocation,
Reorganization
22
Evaluation of Roadmap
Evaluation : validity of research plan
from the qualitative viewpoint in research target
Object：
Roadmap of whole research theme and each main theme＊
Item and its point：
outcome, milestone toward outcome, core technique of
research unit, technical elements, benchmark
Assignment：
External reviewers：Roadmap of each main theme（Rating with
comments）
Internal reviewers：Roadmap of main themes（Rating with
comments）
＊Main Theme ：Research theme in 5-years plan
＊Challenging Theme ： Research theme that is not in 5-years plan but is
recognized to be important by research unit
23
Evaluation of Output
Evaluation : whether outputs in the evaluation period
are contributed to the outcome or not
Object：
Academic paper, patent, start-up,
prototypal product, intellectual right, prize, commendation
Criterion：
Target recorded in milestone, World top level,
Government preparation plan
Assignment：
External reviewers：
Rating with comments to each main research theme
Rating with comments to research themes
Internal reviewers：
Comments to each research theme
Rating with comments to main research themes
24
Evaluation of Management
Evaluation : “full research” and its management from
the viewpoint of realizing outcome
Object:
Policy statement and actual actions for bridging toward innovation
Items：
1. “Full research”
2. Innovation：Technology transfer of intellectual right, Standard, R&D consortium,
3. Training of
Research collaboration, Government project, etc.
human resources: Training staffs,
Training of young researchers,etc
4. Research unit management
・Total management：Leadership of the unit leader, Common mission, etc
・Budget management：Budget management, Budget management from the
viewpoint of cost vs effect, Taking of external funds, etc
・Risk management: Risk management (Risk about disaster, accident、ethics)
Assignment：
External reviewers： Comments
Internal reviewers： Rating with comments
25
CONTENTS
1.Strategic Research Evaluation
2.Missions and Activities of AIST
3.Strategy and Evaluation
4.Evaluation System from the Outcome
Point of View
5.PDCA cycle in the Research Institute
26
PDCA Cycle
Peopl
e
Plan
President
Strategy of AIST
Action
Resources Allocation
Reorganization
Do
Research Units
＊Roadmap for outcomes
＊Output indexes
＊Management
Evaluation
of Each Unit’s
Evaluation Committee
for IAI in METI
Results
Principal
reviewers’ report
Report
Check
Evaluation
Department
1) Roadmap
Evaluation
2) Output Evaluation
3) Management
Evaluation
Results
of Evaluation
Evaluation:
Rating with
comments
Evaluation Committees
external reviewers
internal reviewers
27
Strategic Research Evaluation
Significance of Strategic Research Evaluation
1. Importance of strategy
2. Significance of ex-ante evaluation (Appraisal)
3. Evaluation from the outcome point of view
4. Reflection of evaluation and the design of new
strategy
5. Strategic evaluation linkage
Novel System of Strategic Research Evaluation
28
Strategic Research Evaluation
Appendix
29
Budget of AIST
Facilities
management cost
5,800
Miscellaneous
3,851
Facilities
management
grants 5,800
Employment cost
35,002
Direct research cost
44,626
Expenditure
98,574
Revenue
98,574
Subsidy
66,437
Commissioned
research fund
22,486
( unit: million yen )
Statistics
Indirect research cost
13,146
(as of April 1, 2006）
30
Pattern of Outcome Creation
(Product Type)
Solar Panel
Startups
Companies、New Startups
Production・Marketing
Collaboration/ Technology Transfer
(Prototype・Improvement）
New Industry
New Market
Research in AIST
Liquid Crystal Display
Novel Idea
Output
Sample
Transfer
Market
Market Share
Outcome
Acceptance in Market & Society
31
Rating
表１　評点基準
評点
評価項目
6
1
3
重
点
課
題
評
価
（
課
題
ご
と
・
外
部
委
員
）
評価内容
D
C
B
A
AA
適
切
卓
越
（
特
記
的
に
優
れ
て
い
る
）
目標とするアウトカムは明確に示され
たか。
Roadmap eval.
ロードマップ評価
課題の目標達成のためのマイルストー
ン、ユニットのコア技術、克服すべき
技術要素及びベンチマークが明確かつ
適切に示されたか。
また、それらはアウトカム創出に寄与
すると思われるか。
極
め
て
不
適
切
要
改
善
同上
同上
同上
同上
同上
同上
同上
同上
同上
同上
0.1
同上
同上
同上
同上
同上
0.3
概
ね
適
切
For each theme
アウトカム実現に寄与すると思われる
アウトプットが得られているか。
・内容
・世界最高水準との比較
・副次的成果
Output eval.
アウトプット評価
総合評点
算出時の
重み
0.6
(課 題 の 重
み考慮、挑
戦課題を除
く)
マ
ネ
ジ
メ
ン
ト
評
価
（
内
部
委
員
）
本格研究の考え方
アウトカム実現に向けてのマネジメン
ト
・全体運営の工夫・努力
・研究運営の工夫・努力
・予算運営の工夫・努力
・人材配置の工夫・努力
・成果活用の工夫・努力
・その他の工夫・努力
Management eval.
ポリシーステートメ
ントに盛り込まれた
ユニット運営の状況
Performance
Rating
AA：Extremely Effective
Ａ： Highly Effective
Ｂ： Effective
Ｃ： Needs Improvement
Ｄ：Unsatisfactory
5
4
3
2
1
2.Calculation method
ユニット全体のロードマップ及びそれ
に 基theme
づく課題全般のロードマップ・ア
For whole
ウトプットの評価
課題全般評価（内部委員）
1.Performance rating
・Arithmetic mean of roadmap &
output of each main research
theme.
・The score is calculated by applying
weight of each main research
theme.
3.Weighting for Final Score
注：A/B 等、中間の評点も認める。
Main theme:
Whole theme:
Management:
6
1
3
32
History of Research Units of AIST
研究センター(RC)
深部地質環境RC
活断層RC
化学物質リスク管理RC
フッ素系等温暖化物質対策テクノロジーRC
ライフサイクルアセスメン トRC
パワーエレクトロニクスRC
生命情報科学RC
生物情報解析RC
ティッシュエンジニアリングRC
ジーンディスカバリーRC
ヒューマンストレスシグナルRC
強相関電子技術RC
次世代半導体RC
サイバーアシストRC
マイクロ・ナノ機能広域発現RC
ものづくり先端技術RC
高分子基盤技術RC
光反応制御RC
新炭素系材料開発RC
シナジーマテリアルRC
超臨界流体RC
スマートストラクチャーRC
界面ナノアーキテクトニクスRC
グリッドRC
爆発安全RC
糖鎖工学RC
年齢軸生命工学RC
技術と社会RC
デジタルヒューマンRC
近接場光応用工学RC
ダイヤモンドRC
バイオニクスRC
ジーンファンクション RC
太陽光発電RC
システム検証RC
ナノカーボンRC
健康工学RC
情報セキュリティRC
固体高分子形燃料電池先端基盤RC
コンパクト化学プロセスRC
バイオマスRC
2001
平 成13年 度
2002
平 成14年 度
2003
平 成15年 度
2004
平 成16年 度
2005
平 成17年 度
04.03.31 廃止(再編)
04.03.31 廃止(再編)
02.06.30 廃止(改組)
04.07.14 廃止(再編)
04.03.31 廃止(再編)
04.03.31 廃止(再編)
04.03.31 廃止(再編)
04.04.01 名称変更 (→ナノカ ーボン研究セン ター)
04.03.31 廃止
05.03.31 廃止(再編)
04.03.31 廃止(再編)
※1
04.04.30 廃止
(新炭素系材料開発研究セン ター→)名称変更
Research Centers
(as of 31 Mar., 2006）
Research Institutes and Initiatives
研究部門/系
計測標準研究部門
地球科学情報研究部門
地圏資源環境研究部門
海洋資源環境研究部門
エネルギー利用研究部門
電力エネルギー研究部門
環境管理研究部門
環境調和技術研究部門
情報処理研究部門
知能システム研究部門
エレクトロニクス研究部門
光技術研究部門
生物遺伝子資源研究部門
分子細胞工学研究部門
人間福祉医工学研究部門
脳神経情報研究部門
物質プロセス研究部門
セラミックス研究部門
基礎素材研究部門
機械システム研究部門
ナノテクノロジー研究部門
計算科学研究部門
生物機能工学研究部門
計測フロンティア研究部門
ユビキタスエネルギー研究部門
セルエンジニアリング研究部門
ゲノムファクトリー研究部門
先進製造プロセス研究部門
サステナブルマテリアル研究部門
地質情報研究部門
環境管理技術研究部門
環境化学技術研究部門
エネルギー技術研究部門
情報技術研究部門
人間系特別研究体
生活環境系特別研究体
研究ラボ
グリーンプロセス研究ラボ
薄膜シリコン系太陽電池開発研究ラボ
デジタルヒューマン研究ラボ
ライフエレクトロニクス研究ラボ
次世代光工学研究ラボ
微小重力環境利用材料研究ラボ
純度制御材料開発研究ラボ
メンブレン化学研究ラボ
マイクロ空間化学研究ラボ
先端バイオエレクトロニクス研究ラボ
極微プロファイル計測研究ラボ
ジーンファンクション 研究ラボ
単一分子生体ナノ計測研究ラボ
循環バイオマス研究ラボ
システム検証研究ラボ
実環境計測・診断研究ラボ
メタンハイドレート研究ラボ
シグナル分子研究ラボ
超高速光信号処理デバイス研究ラボ
2001
平成13年度
2002
平成14年度
2003 平成16年度2004平成17年度 2005
平成15年度
04.04.30 廃止(再編)
04.04.30 廃止(再編)
04.06.30 廃止(再編)
04.06.30 廃止(再編)
04.04.30 廃止(再編)
04.04.30 廃止(再編)
04.07.14 廃止(再編)
02.08.31 廃止(再編)
02.08.31 廃止(再編)
04.03.31 廃止(再編)
04.03.31 廃止(再編)
04.03.31 廃止(再編)
04.03.31 廃止(再編)
04.03.31 廃止(再編)
04.03.31 廃止(再編)
平成13年度
平成14年度
平成15年度
平成16年度
平成17年度
02.3.31 廃止
04.03.31 廃止(セン ター設立)
03.03.31 廃止(セン ター設立)
04.03.31 廃止
03.03.31 廃止(セン ター設立)
03.03.31 廃止
03.03.31 廃止
05.03.31 廃止(セン ター設立)
05.03.31 廃止
03.07.31 廃止(セン ター設立)
04.03.31 廃止(再編・部門設立)
03.08.31 廃止(セン ター設立)
05.03.31 廃止(再編・セン ター設立)
05.09.30 廃止(セン ター設立)
04.03.31 廃止(セン ター設立)
33
R&D Evaluation in Japan
Structure of R&D Evaluation in Japan
National
Level
(CSTP)
Ministry
Level
Funding
agency
Level
R&D Institute
National Guideline for R&D Evaluation
Cabinet decision
Decided by the Prime Minister
METI Guideline
S&T Basic Plan
MEXT Guideline
Other Ministries
JST Guideline
NEDO Guideline
AIST Guideline
RIKEN Guideline
Level
・CSTP has been promoting R&D evaluation as an integral part of S&T policy.
・Ministries and R&D Organizations carry out the major part of evaluation activities.
METI :Ministry of Economy, Trade and Industry
MEXT：Ministry of Education, Culture, Sports, Science and Technology
34
New Science and Technology Administration System in Japan
The Council for Science and Technology Policy (CSTP) was established
in January 2002, as a headquarter for national S&T policy.
Council for Science and Technology Policy
(Chaired by Prime Minister)
15 Regular members of the council
Chief Cabinet Secretary
Minister of State for
Chairman of
8 members from
S&T policy
the Science Council academia & business
＜Academic S&T＞
Minister of Education, Culture,
Sports, Science & Technology
Universities
Research
Institutes
＜Industrial S&T＞
Minister of Economy,
Trade & Industry
Temporary members
＜Finance＞ ＜Policy Evaluation＞
Minister of
Finance
Minister of
Public Management,
Home Affairs, Post &
Telecommunication
Minister
Minister
Industries
35
R&D Budget of Japanese Ministries
(including expenses for labors and facilities, 2005)
MIC
MLIT
2%
2%
MAFF
3%
MHLW
4%
others
3%
JDA
4%
36 Billions
US$
METI
17%
MEXT
65%
36