Transcript Achieving and Demonstrating Research Impact

Achieving and
Demonstrating Research
Impact
John Scott
Workshop Outline
What is ‘impact’ and how can it be achieved?
This part of the workshop will consider the REF definition of impact, the various types
of impact, and the criteria for scoring impact. It will consider the broad range of types
of impact and their varying advantages and disadvantages. It will also consider how
the achievement of impact might be built into research projects and the support
mechanisms that Universities can establish to promote the achievement of impact.
Coffee break
How can impact be demonstrated?
In this part of the workshop I will consider the issue of producing evidence on impact
and of constructing a plausible account of the impact achieved. We will consider the
issues involved in constructing and presenting appropriate case studies.
The Future of the REF.
This final session will be an open discussion of wider issues in the REF: panel
structure, assessment mechanisms, bibliometrics, etc. It is an opportunity to explore
issues and potentialities in the approach to REF 2020.
Final Questions
What is ‘impact’ and how can it be
achieved?
The Development of Research Assessment
•
1986 UGC Review
Subject areas rated as ’outstanding’, ‘above average’,
‘average’ or ‘below average’.
•
1989 UFC Research Selectivity Exercise
•
1992 HEFCE (S&W) Research Assessment Exercise
Subject areas rated on scale from 1 to 5.
Grade 4 means ‘some evidence of international excellence’.
•
1996 RAE
Grade 3 split into 3a and 3b. 3a ‘ possibly showing
evidence of international excellence.’
•
2001 RAE
•
2008 RAE
Each subject area given a ‘profile’ with % of activity at
levels from 1 to 4. Grade 2 means ‘Quality that is
recognised internationally in terms of originality,
significance and rigour
•
2014 Research Excellence Framework (REF)
All exercises involved, to a greater or lesser degree, an assessment
of publications and the research environment. Later exercises
added assessment of ‘esteem’.
20014 REF introduced the assessment of ‘impact’, replacing ‘esteem’.
Subject Panels
•
Assessments made by ‘expert’ panels, generally
discipline-based, but have become broader in composition.
•
Subject panels coordinated by ‘Main Panels’ covering large
areas.
•
Universities can choose how to group their staff in
subject areas and which panels to submit them to.
A
B
C
D
1 Clinical Medicine
7 Earth Systems and
Environmental
Sciences
16 Architecture,
Built Environment
and Planning
27 Area Studies
8 Chemistry
17 Geography,
Environmental
Studies and
Archaeology
2 Public Health,
Health Services and
Primary Care
3 Allied Health
Professions,
Dentistry, Nursing
and Pharmacy
9 Physics
4 Psychology,
Psychiatry and
Neuroscience
11 Computer
Science and
Informatics
5 Biological Sciences
12 Aeronautical,
Mechanical,
Chemical and
Manufacturing
Engineering
6 Agriculture,
Veterinary and Food
Science
10 Mathematical
Sciences
13 Electrical and
Electronic
Engineering,
Metallurgy and
Materials
18 Economics and
Econometrics
19 Business and
Management
Studies
28 Modern
Languages and
Linguistics
29 English Language
and Literature
30 History
31 Classics
32 Philosophy
20 Law
33 Theology and
Religious Studies
21 Politics and
International
Studies
34 Art and Design:
History, Practice and
Theory
22 Social Work and
Social Policy
35 Music, Drama,
Dance and
Performing Arts
23 Sociology
14 Civil and
Construction
Engineering
24 Anthropology
and Development
Studies
15 General
Engineering
25 Education
26 Sport and
Exercise Sciences,
Leisure and Tourism
36 Communication,
Cultural and Media
Studies, Library and
Information
Management
The ‘Profile’ system
Each element of the assessment (outputs, environment, and
esteem/impact) assessed separately. Panels judge the % of
Each element that is rated at the quality levels.
Profiles for each element combined into an overall profile using
a ‘weighting’.
Weightings used in REF:
Outputs
65%
Environment
15%
Impact
20%
Structure of the Impact Assessment
Universities must submit one case study for about every eight staff submitted,
with a minimum of two case studies.
Must also submit an Impact Statement – strategy and approach to impact.
Case Studies and Statement are equal constituents of the assessment.
Where did ‘impact’ come from?
•
Government demand for ‘value for money’.
•
Public reaction to ‘ivory towers’
HEFCE pre-empted Government action by introducing ‘impact
assessment’.
Consultation and debate led to broadening of meaning of ‘impact’.
Definition of impact for the REF
(From Assessment Framework and Guidance on Submissions)
140. For the purposes of the REF, impact is defined as an effect on, change or benefit
to the economy, society, culture, public policy or services, health, the environment
or quality of life, beyond academia (as set out in paragraph 143).
141. Impact includes, but is not limited to, an effect on, change or benefit to:
• the activity, attitude, awareness, behaviour, capacity, opportunity, performance,
policy, practice, process or understanding
• of an audience, beneficiary, community, constituency, organisation or individuals
• in any geographic location whether locally, regionally, nationally or internationally.
142. Impact includes the reduction or prevention of harm, risk, cost or other
negative effects.
Paragraph 143 excludes ‘academic impact’ and impact on students.
•
This definition is specified by each main panel.
•
There are slight differences, so it is important to know which
panel is to be submitted to.
•
I will illustrate with the criteria for Main Panel C:
‘The main panel acknowledges that impact within its remit may take many forms
and occur in a wide range of spheres. These may include (but are not restricted to):
creativity, culture and society; the economy, commerce or organisations;
the environment; health and welfare; practitioners and professional services;
public policy, law and services.
‘The categories used to define spheres of impact, for the purpose of this document,
inevitably overlap and should not be taken as restrictive. Case studies may describe
impacts which have affected more than one sphere.’
Impacts on creativity, culture and
society:
Impacts where the beneficiaries are
individuals, groups of individuals,
organisations or communities whose
knowledge, behaviours, practices,
rights or duties have been influenced








Economic, commercial,
organisational impacts:
Impacts where the beneficiaries may
include new or established
businesses, or other types of
organisation undertaking activities
which create wealth










Impacts on the environment:
Impacts where the key beneficiaries
are the natural, historic and/or built
environment, together with societies,
individuals or groups of individuals
who benefit as a result









Enhancements to heritage preservation, conservation and
presentation; the latter including museum and gallery exhibitions.
Production of cultural artefacts, including for example, films, novels
and TV programmes.
Public or political debate has been shaped or informed; this may
include activity that has challenged established norms, modes of
thought or practices.
Improved social welfare, equality, social inclusion; improved access
to justice and other opportunities (including employment and
education).
Improvements to legal and other frameworks for securing
intellectual property rights.
Enhancements to policy and practice for securing poverty
alleviation.
Influential contributions to campaigns for social, economic political
and/or legal change.
Enhanced cultural understanding of issues and phenomena;
shaping or informing public attitudes and values.
Changed approach to management of resources has resulted in
improved service delivery.
Development of new or improved materials, products or processes.
Improved support for the development of ‘small scale’ technologies.
Improved effectiveness of workplace practices.
Improvements in legal frameworks, regulatory environment or
governance of business entities.
Better access to finance opportunities.
Contribution to improved social, cultural and environmental
sustainability.
Enhanced corporate social responsibility policies.
More effective dispute resolution.
Understanding, developing and adopting alternative economic
models (such as fair trade).
Specific changes in public awareness or behaviours relevant to the
environment.
Improved management or conservation of natural resources or
environmental risk.
Improved management of an environmental risk or hazard.
Operations or practice of a business or public service have been
changed to achieve environmental objectives.
Improved design or implementation of environmental policy or
regulation.
Changed conservation policy/practice or resource management
practices.
Changes in environmental or architectural design standards or
general practice.
Influence on professional practice or codes.
Changes in practices or policies affecting biodiversity.
Health and welfare impacts:
Impacts where the beneficiaries are
individuals and groups (human or
animal) whose quality of life has been
enhanced (or harm mitigated) or
whose rights or interests have been
protected or advocated
Impacts on practitioners and
professional services:
Impacts where the beneficiaries may
include organisations or individuals
involved in the development and/or
delivery of professional services and
ethics

















Development or adoption of new indicators of health and well-being.
Development of policy and practice with regard to medical ethics,
health services or social care provision.
Influence on CPD.
Influence or shaping of relevant legislation.
Influencing policy or practice leading to improved take-up or use of
services.
Improved provision or access to services.
Development of ethical standards.
Improved standards in training.
Improved health and welfare outcomes.
Changed practice for specific groups (which may include cessation
of certain practices shown to be ineffective by research).
Influence on professional standards, guidelines or training.
Development of resources to enhance professional practice.
Use of research findings in the conduct of professional work or
practice.
Influence on planning or management of services.
Use of research findings by professional bodies to define best
practice, formulate policy, or to lobby government or other
stakeholders.
Practitioner debate has been informed or stimulated by research
findings.
Research has challenged conventional wisdom, stimulating debate
among stakeholders.
The criteria for assessing impacts are ‘reach and significance’.:
‘Reach will be understood in terms of the extent and diversity of the communities, environments, individuals,
organisations or any other beneficiaries that have benefited or been affected.
‘Significance will be understood in terms of the degree to which the impact has enriched, influenced, informed or
changed policies, opportunities, perspectives or practices of communities, individuals or organisations.’ (Panel C
Criteria)
These are assessed on the scale of starred levels:
Four star Outstanding impacts in terms of their reach and significance.
Three star Very considerable impacts in terms of their reach and significance.
Two star Considerable impacts in terms of their reach and significance.
One star Recognised but modest impacts in terms of their reach and significance.
Unclassified The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not
underpinned by excellent research produced by the submitted unit.’
Key considerations:
•
Eligibility of impact
Research must have been carried out at the institution between 1st Jan
1993 and 31st December 2013. i.e., 20 year period up to submission
date. For REF 2020, probably research from 1st Jan 2000.
Impact must be apparent between 1st January 2008 and 31st July 2013.
For REF 2020, probably impact after 1st January 2014.
•
Presentation of evidence
Must put in place mechanisms for collecting evidence on impact. This
needs to be an integral part of the research and of Department/Faculty
practice.
•
Support at institution/unit level
Must have mechanisms and strategy in place that support project-level
activities.
How can impact be demonstrated?
What is the panel looking for?
Clear Evidence. Assessment must be based on what is on the
paper.
This involves three key aspects:
•
Quality of underpinning research: the 2* threshold
•
Significance and reach of impact: the evidence
•
Compelling narrative linking research to impact
Key questions considered by panel in assessing case studies:
•
What was the research activity: a person, a project, or a programme?
•
Was research actually carried out at the university?
•
Was the research carried out within the time period allowed?
•
Did the impact occur within the time period allowed?
•
Did the research actually contribute to the impact?
•
Is there supporting evidence for the impact?
Issues in considering Impact Statements
•
Is an impact strategy demonstrated?
•
Are institutional support mechanisms actually taken up?
•
Relation of case studies to Impact Statement. Are they
examples of a successful application of strategy,
examples that have informed the development of a strategy, or
serendipitous successes from which something can be learned.
Problems apparent in the assessment
•
Impact assessment was based on a small number of elements, so results
were ‘lumpy’ compared with outputs.
•
This affected the overall assessment profile
What may change for 2020
•
Impact weighting unlikely to go down
•
More metrics to help avoid the ‘lumpy’ distribution
•
Panel configuration – key issues about which panel to submit to.