Transcript Academic-Business-Clinical Research Innovation Facility

HEFCE
RAE & REF
Presented at
Council of Deans,
Seminar, Manchester,
Feb 2009
By
Hugh McKenna
Karen Cox
BRIEF BACKGROUND
 Conducted jointly by the Higher Education Funding
Councils for England (HEFCE), Scotland (SHEFC),
Wales (HEFCW) and Northern Ireland (DEL)
 Purpose: to produce ratings of research quality for the
HE funding bodies to determine research funding
allocations to HEIs
RAE 2008
 Principles of Equity, Diversity, Equality, Expert Review,
Clarity, Consistency, Continuity, Credibility, Efficiency,
Neutrality, Transparency.
 All types of research, including practice-based research,
applied research, basic/strategic research, interdisciplinary
research.
 Assessment based on peer review.
 15 Panels and 67 Sub Panels
Sub Panel 11
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Chair
Prof Hugh McKenna, University of Ulster
Panel Secretaries
Mrs Lilian Caras, Brenda Purkiss
Members
Ms Sarah Buckland, INVOLVE
Prof Charlotte Clarke, University of Northumbria
Prof Karen Cox University of Nottingham
Prof Dame Pauline Fielding,
Prof Martin Johnson, University of Salford
Prof William Lauder, University of Dundee
Prof Karen Luker, University of Manchester
Prof Dame Jill Macleod-Clarke, University of Southampton
Dr Maggs Maguire, Department of Nursing, Scottish Executive
Professor Dave Thompson, University of Leicester
Prof Ruth Northway University of Glamorgan.
Prof Anne Marie Rafferty, Kings College London
Dr Kate Seers, University of Warwick
Prof Roger Watson, University of Sheffield
Prof Anne Williams, University of Wales, Cardiff
Sub Panel 12
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Chair
Prof Julius Sim, Keele University
Panel Secretaries
Mrs Lilian Caras, Brenda Purkiss
Members
David Billington, Liverpool John Moores
Michael Campbell, University of Sheffield
Aedin Cassidy, University of East Anglia
Stephen Downes, University of Ulster
Pamela Enderby, University of Sheffield
Bernard Gilmartin, Aston University
Martin Griffin, Aston University
Jenny Hewison, University of Leeds
Tracey Howe, Glasgow Caledonian University
Ian Kitchen, University of Surrey
Di Newham, King's College London
Margaret Nicol, Queen Margaret University Edinburgh
David Rogers, University of Portsmouth
Diane Scutt, University of Liverpool
Debbie Sell, Great Ormond Street Hospital
Christina Victor, University of Reading
Keith Wafford, Eli Lilly and Company
Main Panel C
•
•
•
•
•
•
•
•
•
•
Professor Maggie Pearson (Chair)
Professor David Williams (10)
Professor Hugh McKenna (11)
Professor Julius Sim (12)
Professor Robert Hider (13)
Professor Mi Ja Kim
Professor John Stamm
John Stageman
MRC/AMRC/ESRC/DoH/Scot Gov Observers
Secretariat
RAE 2008
 January 2005 Guidance to panels issued;
 Summer 2005 Guidance on submissions issued;
 Autumn 2005 Consultation on criteria and working methods;
 February 2006 Final criteria and working methods issued;
 31 July 2007 End of assessment period for research income and
research student data
 31 December 2007 End of publication period (cut-off point for
publication of research outputs)
 31 October 2007 Census date
 30 November 2007 Closing date for submissions
 December 2008 Results published
DEFINITION OF RESEARCH
• Research - original investigation undertaken in order to gain
knowledge and understanding. It includes work of direct relevance
to the needs of commerce and industry, and to the public and
voluntary sectors; scholarship*; the invention and generation of
ideas... etc, where these lead to new or substantially improved
materials, devices, products and processes...
[* The creation, development and maintenance of the intellectual infrastructure of subjects and
disciplines, in forms such as dictionaries, scholarly editions, catalogues and contributions to
major research databases.]
 It excludes routine testing and routine analysis of materials,
components and processes (such as for the maintenance of
national standards…).
 It also excludes the development of teaching materials that do not
embody original research”.
 It excludes Audit
CRITERIA & WORKING METHODS
Sub Panel 11
 “Research activity relevant to the disciplines of nursing
and midwifery, specialist community public health nursing
and all the contexts within which they operate, including
policy, practice, education and management”.
 Proportional weighting given to quality of Research
Outputs, Research Environment and Research Esteem;
 Opportunity to refer to other Sub Panels and specialist
advisors;
CRITERIA & WORKING METHODS
Sub Panel 12
• “The UOA includes (but is not limited to): biomedical
sciences; nutrition and dietetics; optometry and orthoptics;
radiography; podiatry; occupational therapy;
physiotherapy; speech and language therapy; arts
therapies; health promotion; psychosocial and ethical
aspects of health and healthcare; associated health services
research (to include methodological work on quantitative
or qualitative procedures)”;
 Proportional weighting given to quality of Research
Outputs, Research Environment and Research Esteem;
 Opportunity to refer to other Sub Panels and specialist
advisors.
Category
Staff
information
Description
•Research-active staff details (RA1)
Category A: Academic staff in post on census date.
Category B: Academic staff who held contract with
institution after 1 January 2001 and left the institution
(or, exceptionally, being returned under a different UOA)
Category C: Other individuals active in research in the
department as independent investigators at the census
date.
Category D: Other individuals active in research in the
department as independent investigators after 1 January
2001 but not at the census date
* Early Career Staff = entered academic employment as
Category A staff on or after 1st Aug 2003*
Research
Profile
•Research output (RA2)
period 1 January 2001 to 31 December 2007. up to
four items of research output for each researcher
(subject to panel).
•Research students (RA3a)
No’s full-time & part-time research students and
degrees awarded during the period 1 January 2001
to 31 July 2007.
•Research studentships (RA3b)
No’s research studentships and source of funding in
the period 1 January 2001 to 31 July 2007.
•External research income (RA4)
Amounts & sources of external funding in the period
1 January 2001 to 31 July 2007.
Textual
descriptions
•Research environment and esteem (RA5a)
Research Environment includes info on research
structure, staffing policy, research strategy.
Indicators of Research Esteem and significance; and
any reasonable additional information requested by the
sub-panel for period 1 January 2001 to 31 July 2007.
•Individual staff circumstances (RA5b)
Including details of any individual staff circumstances
which have significantly affected their contribution to
the submissions, such as engagement on long-term
projects, secondment, and any matters covered by
equal opportunities legislation.
Star Ratings?
Four
star
Research judged to be world-leading in terms of
originality, significance and rigour regarded as a primary
point of reference in its field, which has made or is likely to
make an outstanding contribution to knowledge, theory,
practice or policy.
Three star
Research judged to be internationally excellent in
terms of originality, significance and rigour, which has made
or is likely to make a highly significant contribution to
knowledge, theory, practice or policy
Two
star
Research judged to be at an international standard in
terms of originality, significance and rigour, which has made
or is likely to make a significant contribution to
knowledge, theory, practice or policy
One
star
Research judged to be original, significant and rigorous,
which has made or is likely to make a contribution to
knowledge, theory, practice or policy
Unclassified
Work that does not meet the definition of one star, or
which does not meet the published definition of research for
the purposes of the RAE.
How did this work in reality?
Overall
quality profile
4*
3*
2*
1*
u/c
% of research
Research 20
activity
Activity
25
30
15
10
Quality level
Level
Research outputs
The overall quality profile
comprises the aggregate
of the weighted profiles
produced
producedfor
foroutputs,
research outputs,
research environment and
esteem indicators.
Research environment
Esteem
Research
indicators
Esteem
4*
3*
2*
1*
u/c
4*
3*
2*
1*
u/c
4*
3*
2*
1*
u/c
15
25
35
15
10
25
25
15
20
15
30
20
10
20
20
eg 70%
(Minimum 50%)
eg 25%
20%
(Minimum 5%)
eg 10%
5%
(Minimum 5%)
Examples - Quality Profile
UoA 11
FTE Staff
Numbers
4*
3*
2*
1*
Un
Class
University A
50
15%
25%
40%
15%
5%
University B
20
0%
15%
40%
45%
0%
Overview - Sub Panel 11
• 761 individuals were returned, submitting 2,851
•
•
•
•
•
•
•
outputs, mostly from peer reviewed journals;
Selectivity in number of staff returned;
Selectivity in type of staff returned;
Selectivity of panel/sub panel;
2001 – 43 submissions, 2008 - 35 submissions;
All thirty-five submissions were reviewed in detail by
every sub-panel member;
All quality profiles represent the consensus of the
sub-panel as a whole;
Strict cognisance was taken of equal opportunity
issues and individual staff circumstances.
RAE 2008: Submissions to
Sub Panel 11- THE STRENGTHS
•
•
•
•
•
•
•
•
•
•
•
44th out of the 67 subject areas in the RAE;
Internationally recognised research in all submissions;
Strengths in translational/applied research;
Good service user involvement;
Positive impact on patient/client care;
Good interdisciplinary work – various disciplines;
Good PhD completions in review period (446);
Good international collaborations;
Good methodological and theoretical sophistication;
Strategic research appointments;
Research linked to national and international priorities;
RAE 2008: Submissions to
Sub Panel 11- THE STRENGTHS
•
•
•
•
•
•
•
•
•
Impact on national and international health care policy;
Substantial competitive research grant capture (> £103 M);
Modern and high quality infrastructure;
Robust staffing policies;
Emphasis on capacity building and sustainability;
Realistic and sustainable research strategies;
Highly focused, strategically sound, financially viable;
Good links between NHS, Voluntary bodies and HEIs;
Esteem indicators - numerous awards, involvement in
prestigious national and international committees, funding
bodies & editorial boards.
RAE 2008: Submissions to
Sub Panel 11- THE WEAKNESSES
• In some institutions service user/public engagement
•
•
•
•
•
was very limited;
Few departments were actively engaged in technology
and knowledge transfer activities;
Very few links with industry;
Some researchers working in isolation in poorly
developed research environments;
Lone researchers and small disparate groups working
on the same topic area;
Little evidence of activity around the economic
evaluation of what nurses and midwives do or on the
evaluation and impact of new roles;
RAE 2008: Submissions to
Sub Panel 11- THE WEAKNESSES
• Fewer outputs on educational and managerial issues;
• Research into aspects of care fundamental to nursing and
•
•
•
•
midwifery were not sufficiently evident in submissions;
Over one third of the 1,283 research studentships were
supported by the submitting HEIs themselves;
The doctoral degrees awarded per research student FTE
were lower than expected;
Only fifty-nine research-active staff were returned as
designated ‘early career researchers’;
Insufficient critical mass, inadequate onward investment
and inferior succession planning.
Overview- Sub Panel 12
•
•
•
•
•
•
•
•
68 submissions - 18 to this sub-panel for the first time;
1,622 individuals returned, submitting 6,216 outputs;
External research income totalled £355.7 million;
Varying levels of selectivity exercised by HEIs, in
submissions and outputs;
Heterogeneous in substantive and methodological terms;
Very wide distribution of quality across the submissions;
All quality profiles represent the consensus of the subpanel as a whole;
Strict cognisance was taken of equal opportunity issues
and individual staff circumstances;
RAE 2008: Submissions to
Sub Panel 12- THE STRENGTHS
• Clearly-articulated and focused research strategy;
• Coherent and synergistic research groupings, with an
•
•
•
•
•
•
appropriate ratio of researchers to research foci;
Strong evidence of sustainability;
Well-balanced portfolio of research funding, from
competitive peer-review funding sources;
Appropriate and productive links with partners in the NHS
and industry, and with service users;
Good evidence of translational research.;
Outputs showed rigorous and innovative methodology;
Research outputs had clear evidence of actual or potential
impact on practice or policy.
RAE 2008: Submissions to
Sub Panel 12- THE STRENGTHS
• Evidence of internationally excellent or world-leading
•
•
•
•
•
•
research in virtually all the disciplines – especially
biomedical science, nutrition and optometry;
Evidence of investigations into the efficacy or effectiveness
of therapeutic interventions;
Large body of important work on the underpinning basic
sciences;
Studies providing valuable insights in the social and
psychological aspects of health, illness, and the delivery of
professional care;
Appropriate and well developed collaborative research links
Some Departments were leading important international
research collaborations
50% increase over review period in PhDs awarded (2,096).
RAE 2008: Submissions to
Sub Panel 12- THE WEAKNESSES
• Submissions were heterogeneous, consisting of a number of
•
•
•
•
•
•
small and seemingly unrelated areas of work, raising questions
as to cohesiveness and future proofing;
Lack of clear evidence of sustainability;
Lack of strategic focus, often with disparate research groupings;
Underdeveloped research infrastructure;
Inappropriately large number of research foci for the number of
staff submitted;
Weak funding profile;
Discrepancy between the profiles for outputs and environment;
RAE 2008: Submissions to
Sub Panel 12- THE WEAKNESSES
• A number of outputs did not meet the RAE research definition;
• Need for an increased focus on the many areas within AHPs
•
•
•
•
where the evidence base is as yet underdeveloped
3,426 new research studentships - institutional self-funding
(36%); Research Councils (11%);
High quality doctoral work may not have attracted, or been
unable to access, Research Council funding;
Research outputs that were of poor methodological quality
and/or of little potential impact;
Poorly developed interdisciplinary and/or institutional
collaboration.
Purpose of QR
(points from HEFCE)
• Creating a sustainable and flexible national baseline capacity
•
•
•
•
which enables the sector to respond strategically to a
changing external environment
To maintain a research base of world leading quality across
the full range of disciplines
To encourage institutions to take investment decisions which
lead to excellence in research, funded selectively by reference
to robust indicators of research quality.
Invest in innovative research, including in new fields and
opening new lines of enquiry, supporting early-career staff
who aspire to grant funding
create potential to make connections across subjects
REF –current state of play
• HEFCE suggests reduce the burden on institutions
• Will be designed to take better account of the impact research
makes on the economy and society.
• A pilot exercise for bibliometric indicators of excellence is now
underway
• HEFCE will set out its plans for all aspects of assessment,
including user-focused research and subjects where
bibliometrics are not appropriate, by summer 2009.
Framework for assessment
RAE assesses a body of research activity in terms of:
•
•
•
Quality of outputs
Research environment
Esteem
HEFCE suggests indicators in REF may also capture
•
How a research environment is supportive of User
valued research
•
Evidence that specific work is esteemed for its quality by
particular audiences
Tools and approaches for assessment
HEFCE suggests assessment is likely to include:
•
Bibliometric analysis
•
•
•
Citations?
Expert review of outputs
Other available statistical indicators
•
•
•
Students (PG) and completions
Income
Studentships
Cunning Plans
• Succession Planning;
• Increase and support Early Career Researchers (ECR)s;
• Ruthless selectivity in all things;
– Staff appointments and support for ECRs -Home grow Vs recruit;
– Publish for high citations;
– Research Grants from prestigious bodies – NIH guidelines;
– Network and collaborate for best esteem nationally and Internationally;
– NHS-HEI collaborations;
– Impact – does the research make a difference?
•
•
•
•
•
•
Service user involvement from design to dissemination;
Interdisciplinarity in the widest sense; esteem is important – who you work with
Translational & policy Research;
Internal Vs External Studentships;
Address policy issues;
Increase the amount and quality of pedagogic research.
Conclusion
• The sub-panel were impressed with most of the submissions
and with the pattern of 3 and 4 star research;
• There are many models of good practice from which
developing research groupings can learn in terms of research
activities, outputs, environment and esteem;
• Investment by Governments, funding bodies and universities
has increased research capacity and developed research
leaders;
• These funding streams need to be sustained and enhanced if
the upward trajectory and momentum are to continue and if the
quality differentials between the strongest and weakest
departments are to be addressed.
• Additional slides added from notes taken
by Alan Simpson at meeting of Academy
of Nursing, Midwifery and Health Visiting
Research (UK) 26/02/2009.
Alison Tierney, Scottish Exec
• Need a continuous and stable funding stream to
•
•
•
•
•
maintain and build on progress
In Scotland, over 40% submitted to other panels, e.g.
social policy, psychology – some may be best
researchers
What do we do with low quality and sub-Rae
researchers?
Approx 10% N&M HEI staff submitted to Rae
How do we build structures and processes that best
support HEI/Health Services partnerships in research
and how best to measure outcomes?
Few joint academic-services authors of papers (7-10%?)
Developing HEI/NHS partnerships
for research
•
•
•
•
•
•
•
Sustainable partnerships
Joint targets for research
Capacity building jointly agreed
Jointly managed research programmes
More shared infrastructure (as in medicine)
Joint and Honorary appointments at senior level
Clinical-academic careers (see Post Registration
Career Framework for Nurses, Welsh Assembly
consultation document, November 2008)
Professor Maggie Pearson
Chair, RAe 2008 Main Panel C
• 70 HEIs in nurse education
• 1996 36 HEIs and 396.5 staff submitted
• 2001 43 HEIs and 578.4 staff
• 2008 36 HEIs and 641.7 staff
• So, over half did not submit – does this
matter?
• 1996 11.1 staff per submission
• 2001 13.5
• 2008 17.8
•
•
•
•
•
•
7 HEIs submitted only in 1996
5 HEIs submitted only in 2001
14 ‘dropped out’
9 ‘new entrants’ in 2008
12 HEIs submitted to all 3 exercises
7 high quality
2008 C Report
• Stronger groups
• Strategic, strong
•
•
•
•
leadership, clear foci
Peer reviewed,
competitive funding
Critical mass (no lone
researchers)
Institutional investment
National/international
collaborations
•
•
•
•
•
Weaker groups
Lack of strategy
Weak leadership
Small, disparate group
Lone researchers