Transcript Slide 1

PRODUCTIVE EVALUATION
OF
SOCIALLY ROBUST RESEARCH
re-thinking evaluation
TD-conference Bern, 16-09-2011
Jack Spaapen
2
FOUR SECTIONS
1.
What kind of research are we talking about?
2.
What are the problems when evaluating this kind of research?
3.
Answers offered by SIAMPI (and ERiC) [productive interactions]
4.
What is needed for a ‘productive’ evaluation of transdisciplinary , socially
robust research?
3
1. WHAT KIND OF RESEARCH ARE WE TALKING ABOUT?
Mode 2 (not purely academic, context of application, interaction with
stakeholders, iterative process, interdisciplinary input technical expertise,
content, use/behavior etc.)
“Knowledge” that emerges in TD networks is scientifically reliable, but also
‘socially robust’. Researchers are aware of social context.
Socially robust = relative to the social context, and liable to testing and
validation by a variety of stakeholders.
Robustness is produced when research has been infiltrated and improved
by social knowledge.
4
EXAMPLES OF TD/SR RESEARCH
Power plants by the Great Lakes - USA
Primary Health care in the Netherlands [SIAMPI]
‘Top sectors’ in the Netherlands [knowledge – skill – cash!]
Mining industry in Argentina [SIAMPI]
17th century theatre studies in Spain [SIAMPI]
Green cars [EU Framework Programs]
Creative industry, new media, (serious) games [top sector in NL]
Green meat production in 2020 in NL
Healthy aging [everywhere]
Water management (New Orleans)
5
TOP SECTORS, NEW DUTCH INNOVATION POLICY
10 topsectors: health, chemistry, high tech, agro-food, horticulture, water,
energy, creative industry, logistics and head offices
10 topteams, dominated by industry, but also sme’s, government, research
Innovation contracts before end of 2011
Reorientation of 500 M€, no ‘new’ money
Philosophy: strong in global competition, local/regional strength
(Wageningen food valley), knowledge  skills  cash
Main policy measures: fiscal advantages, export missions
6
MAIN CHARACTERISTICS of TD/SR RESEARCH
Triple helix, golden triangle: research, industry, society (government, NGOs,
general public), new collaborative arrangements: PPP
Input from different disciplines (science/technical fields, social sciences,
humanities) and from a variety of stakeholders (technical knowledge,
content, use/behavior)
Consensus about long term goals (“ clean energy”), but in the meantime
shifting coalitions, different partners, different intermediate goals, different
interests
Co-creation of ‘ knowledge’ or results (=something reliable that you need to
gain insight and solve problems), research by design, iterative process (nonlinear), using open source, open innovation
7
DIFFICULTIES WITH TD RESEARCH
Attuning different goals, expectations, interests of partners (big industry,
SMEs, local/national/supranational politics, public debate)
Different academic disciplines having different "cultural" habits in publishing
and organization etc., not independent of the surrounding society (publish or
perish, ‘top sector’ policy, ethics, expectations)
Partners not used to work together in these transepistemic arenas
(academics – industry, not knowing what to formulate as research
questions), institutional problems, political and cultural problems….
“In conflict-intense policy contexts, research results are hard to establish”
(Cozzens and Snoek 2010)
8
2. PROBLEMS WITH EVALUATION
Traditional ways don’t work (peer review, bibliometrics, patent analysis) –
based on linear thinking and measure the wrong thing (journal and rankingoriented)
New ways underdeveloped or to elaborative
- case studies (labor intensive, comparability)
- (social) impact analysis (attribution, temporality)
- gap between theory and practice
What to ‘measure’? (complex process, moving target)
Gap between theory and practice of evaluation
9
RIFT BETWEEN THEORY AND PRACTICE
The current state of the art in evaluation practice for measuring policy
impacts does not match the concepts that are most common in the
research literature to describe the connections between knowledge and
policy.
The dominant concept in evaluation practice is linear, framed by logic
models and the terminology of inputs, activities, outputs, and outcomes,
sometimes with a loop back to planning. The policy process itself is a
black box.
In contrast, the dominant concept in the research literature is the network
or system, which is made up of many small conversations, interactions,
and adjustments among a diverse set of actors; and complex concepts of
the ebbs and flows of the policy process itself are incorporated. [Cozzens
and Snoek, 2010]
10
INTERACTIVE PROCESS IN NANO RESEARCH (EE)
SIAMPI ©TILO PROPP
11
KNOWLEDGE CIRCULATION IN ARCHITECTURE
E RIC PROJECT © PETER VD BESSELAAR
12
Indicators to capture societal quality of Architecture and building
research, proposed by mixed panel of researchers and stakeholders
1. Dissemination of knowledge into society: Professional publications, nonacademic publications, exhibitions etc.
2. Spread of technology, artefacts, standards
3. Advisory and consultancy activities
4. Popularisation, education and contribution to public debate
5. Professional training, mobility of graduates
6. Master’s dissertations and graduation projects that address questions from
practitioners
7. Interest of stakeholders
8. Number of researchers with relevant practical experience in the sector(s) that
the research programme targets
9. Public funding related to societal issues
10. Funds from contract research commissioned by potential users
11. Collaboration with societal stakeholders on research, tests and evaluations
12. Consortiums with non-academic organisations
13. Impact and use of results Income from use of results
14. Visibility in public debate/public media rankings
13
MANY ATTEMPTS TO BRIDGE THE GAP…..
REVIEW Cozzens and Snoek, 2010
Dutch Academy initiated studies (2005, 2008, 2010, 2011); national
developments in the Netherlands: Standard Evaluation Protocol (SEP),
Evaluating Research in Context (www.eric-project.nl)
Other European countries, Denmark (Hanne Foss Hansen), UK research
councils, also in USA (broader impact criteria)
EU SIAMPI project on social impact measurement (2009-2011)
14
....BUT NOT VERY SUCCESSFUL YET
Lack of insight in the issues of interaction and collaboration in complex
constellations
Hesitating policy makers and research boards who love hard metrics, but at
the same time pressure for more ‘socially relevant’ research (valorisation).
Pressure from increasing global competition, politicians want excellent
research and profitable results (top sector policy)
15
METHODOLOGICAL PROBLEMS WITH ASSESSING TD/SR
-
Attribution (who is responsible for which result in such TD clusters)
-
Temporality (processes tend to continue over time, players trade places)
-
What is good result for the one is not necessarily good for the other
-
Different accountability and reward systems
-
Lack of robust data
16
3. ERiC and SIAMPI projects
ERiC = Evaluating Research in Context, Dutch initiative to develop methods for
social impact evaluation
SIAMPI, FP7 project to develop methods for social impact assessment
http://www.siampi.eu/
• Focus on what goes on between research and context, what kind if
interaction, role of stakeholders, indicators?
• Can we find ways to evaluate TD research and social impact of research
• Feasibility
17
PRODUCTIVE INTERACTIONS
Productive interactions are exchanges of knowledge and expertise
between science and society that result in behavioral change 
socially robust research
Change can be broad: from a different way of thinking, following a
different method, extending or intensifying collaboration, to actual
use by stakeholders (including other researchers)
‘productive’ = producing change in social relations
not in an economic sense but in a social sense
18
THREE INDICATOR CATEGORIES OF PI
• Direct (personal) interactions : joint projects, advisory, consultancy,
double functions, mobility
• Indirect interactions :
Texts : articles, books, catalogues, protocols
Artifacts : instruments, exhibitions, models, designs
• Money : contracts, subsidies, patenting, licensing
19
STEPS IN SIAMPI CASE STUDIES
1. Identify productive interactions and see what role they play in
collaboration and achieving social impact
2. Construct different research ‘profiles’, based on productive interactions,
recognition of different missions, policy contexts, etc.
3. Experimental development of indicators that represent the different
profiles, do it together with researchers and stakeholder (focus groups),
try and find some common indicators and some specific
4. Broad discussion of mission and strategy, learning type of evaluation
20
RESULTS PER CASE STUDY
We paid attention to differences in:
•
•
•
•
•
National research cultures and policies
Research mode
Research type
Productive interactions
Social impact
21
Research domain
nano
ict
Health care
ssh
Country
NL, France
UK, NL, EU
NL
Esp, UK
Research type
Frontier, basic,
strategic
Basic, applied, TD
strategic, applied,
policy
Basic, strategic,
applied
Research mode
Academic, in
collaboration with
industry
Open to partnership of
knowledge producers
and users
Academic, open to
collaboration with
industry, government, patient
groups, professionals
Academic, open to
collaboration with
policy, institutions,
wider public,
industry
Productive
Interactions
Public
understanding,
ethical debates,
policy making,
products
Transport use,
security, interaction
between citizens and
government
Consultation, collaboration, regulations, proto-cols,
commercial
exchanges, PPPs,
post academic
training, patient
organizations
Informal links and
advice, formal
research contract
and collaborative
projects,
consultancy, cultural
events
Social Impact
Health, safety,
public
acceptance of
nano tech
Transport use,
security, interaction
between citizens and
government
Diagnostics,
treatments, safety,
general health,
policy advice
Policy tools and
techniques,
management
methods, cultural
goods and services
22
HEALTH CASE: HIGHLY ORGANISED FORM OF SOCIALLY
ROBUST RESEARCH
NIVEL mission includes applied research for policy makers
Stakeholder contacts actively organised by the board in a strategic
consultancy process (top down)
Discussion of research results with stakeholders required
to safeguard financial support
to enhance chance of implementation of results
23
SOCIAL SCIENCES CASE: SERENDIPITUOUS PATTERN, BUT GOAL
ORIENTED
BRASS, an ESRC funded research centre
General goal to achieve social impact , but researcher were also
motivated on a personal level
Multiple roles, part researcher, part political activist
Chance pattern, one project leads to another, almost by chance encounter
Political implications, not typical for SSH
24
NANO CASE: INTERMEDIAIRY CHECK POINTS
Promises and expectations galore in this new field
No direct links with end-users
Long networks of circulation
Intermediairy check points (proxy indicators) towards social relevance
25
ICT CASE: CONTRIBUTIONS INSTEAD OF ATTRIBUTION
Insight in who is contributing what is more effective than trying to reconst
attribution over a long time
It took a decade from start of spin-off (1997) to develop semantic technology
that was bought by market and used succesfully
Research system not patient enough, what can you say to the researcher who
after five years and no concrete ‘results’ wants more money?
26
SOME CONCLUSIONS FROM THE CASES
Productive Interactions help to identify steps in producing socially robust research
Indicators can be developed, there is a data problem, but options are growing
thanks to more awareness and internet and new databases
Great variety of interaction patterns (very incidental, personal and informal
relations to highly organized and professionalized networks)
Policy conflicts, power differences important in evaluation
Research organisations have to make serious efforts to gather more robust data on
social impact and on research output and outcome to wider audiences.
27
4. What is necessary for the evaluation of SRR?
o
A different perspective, non-linear, learning, new ‘peers’,
administrators who dare to care
o
Involvement of all stakeholders in development of criteria and
indicators, acknowledging TD partners, raising commitment
o
New type of self evaluation, focus on project as a whole, all
participate and debate SWOT
o
Serious investing in robust data collection
o
Use / acceptance of these methods at all levels of research
(institute, national, EU)
28
SIAMPI APPROACH TO EVALUATION OF TD/SR RESEARCH
1.
Mission oriented, establish research profile and (intermediate)
research goals
2.
Contextual, analyse stakeholder context, and engage
stakeholders, also in (self) evaluation process
3.
Focus on productive interactions for (proxy) indicators
4.
Use indicators not to judge, but to inform
5.
TD group to write self evaluation incl. SWOT analysis
6.
Build in feed back to all involved
7.
The goal of the evaluation is learning and improving in stead of
accounting
29
EXAMPLE OF SOCIAL IMPACT OF THREE MEDICAL
GUIDELINES FOR GPS [CONTEXTUAL RESPONSE ANALYSIS]
© AD PRINS
Medical Journals
30
25
Chat, Blog's
20
Professional information
sites
15
10
5
University & Research
organizations
0
Dementia
Public or Patient
information sites
Epicondylitis
Thyroid Function
Libraries
Health Care Providers
Professional Organizations
Google searches for three guidelines of Leiden Dep. of Public Health and General
Practice