Transcript Impact workshop

Impact workshop
Phil Hannaford
VP Research and Knowledge Exchange
Agenda
• Introduction
• Overview
• HEFC Case Study Template
• Impact case studies- examples of best
practice
• Next steps
www.abdn.ac.uk
Overview
Purpose of the REF
The REF replaces the RAE as the UK-wide framework for
assessing research in all disciplines. Its purpose is:
•
To inform research funding allocations by the four UK HE
funding bodies (approximately £2 billion per year)
•
Provide accountability for public funding of research and
demonstrate its benefits
•
To provide benchmarks and reputational yardsticks
Key changes since the 2008 RAE
•
•
•
•
•
•
•
•
Inclusion of assessment of impact
•
Publication of overall quality profiles in 1% steps
Fewer UOAs/panels, operating more consistently
Strengthened equality and diversity measures
Revised eligibility criteria for staff
Addition of (limited) use of citation data in some UOAs
Removal of ‘esteem’ as a distinct element
Revised approach to ‘environment’ and data collection
Increased ‘user’ input on panels; and an integrated role for
additional assessors
The assessment framework: Overview
Overall quality
Outputs
Impact
Environment
Maximum of 4 outputs
per researcher
Impact template and
case studies
Environment data and
template
65%
20%
15%
Timetable
2011
2012
2013
2014
• Panels appointed
(Feb)
• Final panel criteria and
methods (Jan)
• Panels assess
submissions
• Guidance on
submissions (Jul)
• HEIs submit codes of
practice (final deadline
Jul)
• Launch REF
submissions system
(Jan)
• Draft panel criteria
for consultation (Jul)
• Close of
consultation (5 Oct)
• Requests for multiple
submissions (final
deadline Dec)
• Survey of submission
intentions complete
(Dec)
• Recruit additional
assessors
• Staff census date
(31 Oct)
• Submissions
deadline (29 Nov)
• Publish outcomes
(Dec)
Impact- the big unknown
www.abdn.ac.uk
Impact- the big unknown
www.abdn.ac.uk
Impact- the big unknown
www.abdn.ac.uk
Impact- the big unknown
www.abdn.ac.uk
Impact: Submissions
Impact template (REF3a)
• Sets out the submitted unit’s
general approach to enabling
impact from its research
• One template per submission – with
a page limit depending on the
number of staff submitted
• Covers the period 1 Jan 2008 to 31
Jul 2013
• Contributes 20% to the impact subprofile
Case studies (REF3b)
• Specific examples of impacts
already achieved, that were
underpinned by the submitted
unit’s excellent research conducted
between 1 Jan 1993 to 31 Dec 2013
• Number required depends on the
number of staff submitted (2 up to
14.99 FTE, plus 1 for every extra 10
FTEs (or part thereof))
• Impacts during 1 Jan 2008 to 31 Jul
2013; underpinned by research
since 1 Jan 1993
• Contributes 80% to the impact subprofile
Impact: Definition for the REF (1)
•
An effect on, change or benefit to the economy,
society, culture, public policy or services, health, the
environment or quality of life, beyond academia
•
Impact includes an effect, change or benefit to:
- The activity, attitude, awareness, behaviour,
capacity, opportunity, performance, policy,
practice, process or understanding
- Of an audience, beneficiary, community,
constituency, organisation or individuals
- In any geographic location whether locally,
regionally, nationally or internationally
Impact: Definition for the REF (2)
•
Impact includes reduction or prevention of harm, risk,
cost or other negative effects
•
It excludes impacts on research or the advancement
of academic knowledge within HE; and impacts on
teaching or other activities within the submitting HEI
•
Other impacts within the HE sector, including teaching
or students, are included where they extend
significantly beyond the submitting HEI
Impacts: Definition for the REF
Economic
Social
Quality of life
Types of impact
Public policy &
services
Environment
Cultural
Health
www.abdn.ac.uk
Impact: Criteria
The criteria for assessing impacts are reach and significance*
Four star
Outstanding impacts in terms of their reach and significance
Three star
Very considerable impacts in terms of their reach and significance
Two star
Considerable impacts in terms of their reach and significance
One star
Recognised but modest impacts in terms of their reach and
significance
Unclassified
The impact is of little or no reach and significance; or the impact was
not eligible; or the impact was not underpinned by excellent research
produced by the submitted unit
* Each main panel provides a descriptive account of the criteria
Reach – How widely felt it was
Significance – How much difference it made to
beneficiaries
www.abdn.ac.uk
Impact: Template (REF3a)
•
The unit’s approach to enabling impact from its research:
- Context for the approach
- The unit’s approach during 2008-2013
- Strategy and plans for supporting impact
- Relationship to the submitted case studies
•
Provides additional information and context for the case
studies, and can take account of particular circumstances
that may have constrained a unit’s selection of case
studies
•
To be assessed in terms of the extent to which the unit’s
approach is conducive to achieving impact of ‘reach and
significance’
Impact: Case studies (REF3b)
•
•
In each case study, the impact described must:
-
Meet the REF definition of impact
-
Be underpinned by excellent research (of at least
2* quality) produced by the submitting unit
between 1 January 1993 to 31 December 2013
Have occurred between 1 Jan 2008 and 31 July
2013 (can be at any stage of maturity)
Submitted case studies need not be representative of
activity across the unit: pick the strongest examples
NB. Its not about the esteem or influence of an
individual or unit
Impact: Case studies (REF3b)
Each case study is limited to 4 pages and must:
- Describe the underpinning research produced by
the submitting unit
- Reference one or more key outputs and provide
evidence of the quality of the research
- Explain how the research made a ‘material and
distinct’ contribution to the impact (there are
many ways in which this may have taken place)
- Explain and provide evidence of the nature and
extent of the impact: Who/what was affected?
How were they affected? When?
- Provide independent sources that could be used to
verify claims about the impact (on a sample audit
basis)
Impact: other key points
•
Impacts may be at any stage of development or maturity
•
Impacts stay with the institution (unlike publications)- so cannot
‘buy-in’ impacts
•
Can be shared with other insitutions (each has to show their
disticntice contribution to the imapct)
•
Impacts must have taken place during the assessment period (not
future or potential impacts)
•
Impacts or benefits arising from engaging the public with the
submitted unit’s research will be eligible (but not dissemination
activity unless there is evidence of its benefits)
•
Impacts arising from public engagement must show that that the
engagement activity was at least in part based on the submitted
unit’s research AND drew materially and distinctly upon it
www.abdn.ac.uk
Glasgow’s pilot experience
•Clinical Medicine (17 case studies) + Earth Systems (4 case studies)
•Writing team
•Steering Group (SG) of senior academics chaired by VP R&E
•Process was:
− Trawl for stories
− Interviewed researchers
www.abdn.ac.uk
Glasgow’s pilot experience
Labour intensive for the staff
involved
− collection and collation of
the material we need to
submit
− the iterative nature of the
drafting process
www.abdn.ac.uk
Glasgow’s pilot experience
External supportive evidence
• Challenging to engage external
contacts:
- who?
- ‘Sorry they left a few months ago...’
-S–t–r–e–t–c–h–i–n–g
Goodwill!
www.abdn.ac.uk
Glasgow’s pilot experience
External supportive evidence
• Challenging to engage external
contacts:
- who?
- ‘Sorry they left a few months ago...’
-S–t–r–e–t–c–h–i–n–g
Goodwill!
NB. We need to think when to engage with external supporters
www.abdn.ac.uk
Glasgow’s pilot experience
Possible types of evidence
•
Testimonials from named individuals
•
Press coverage
•
Guidelines/Documents/Reports
•
Training materials
•
Details of conference/invitations to speak
•
Links to relevant background information
•
Public engagement – speak to the ‘organiser’ of the event
•
Details of grants
•
Publications – highlighted where peer-reviewe
www.abdn.ac.uk
Glasgow’s pilot experience
Avoid subjectivity
• Striking the balance
• Once upon a time...
www.abdn.ac.uk
Glasgow’s pilot experience
Reading group
• External reading team from
user community
•
Clinical medicine UoA
•
Users and academics
•
Extremely helpful
• Internal reading group
www.abdn.ac.uk
Glasgow’s pilot experience
Selecting our case studies
•
Pipeline – extra stories
•
Look into the past
•
Consider ‘reach’ and ‘significance’
•
Pilot panel reports
•
Refer to panel criteria
•
Furthest along the pathway
•
Believable
•
Choose your strongest
www.abdn.ac.uk
Glasgow’s pilot experience
Our results
• Very pleased with the overall result – there were no
surprises
Managed to avoid:
• Generalised, vague claims
• Excessive publication lists or web references
• Lack of coherence
• Claiming potential impact
• Lack of necessary information
www.abdn.ac.uk
Glasgow’s pilot experience
Top four tips...
• Watch the template limits
• Make it easy for the reader
• So what?
• Start with the impact
www.abdn.ac.uk
Glasgow’s preparations
• Mini-REF - best examples of impact
• UoA “Pipelines” created (Colleges/R&E)
• Prioritising pipelines (UoA Champions/R&E)
• Developing case studies with academics
• Horizon scanning
www.abdn.ac.uk
Useful resources
•
HEFCE REF2014 site
www.hefce.ac.uk/research/ref/
•
Pilot panel reports and best practice examples
www.hefce.ac.uk/research/ref/impact/
•
HEFCE REF FAQs
www.hefce.ac.uk/research/ref/faq/
•
Public Engagement – NCCPE Materials
www.publicengagement.ac.uk/how-we-help/event-reports/ref-workshop
•
Your colleagues (in Aberdeen and outside)
www.abdn.ac.uk
Read the guidance !!!!!
• Assessment framework
and guidance on
submissions
• Panel criteria and
working methods
www.abdn.ac.uk
Case study template (REF 3b)
Examples of good practice
Preparing an impact case study
Preparing an impact case study
Suggested questions to help clarify ‘impact’:
•What user groups outwith academia did you work with?
•What was the purpose of the ‘interaction’?
•What has been the effect on the users/audience?
•Did it change something for them?
•How did they benefit from the ‘interaction’?
•How did your contribution effect the impact/benefit?
www.abdn.ac.uk