The Stern Review of the Research Excellence Framework has been published today.
Broadly speaking the review recommends keeping the REF more or less the same as before – i.e. still a periodic exercise based predominantly on peer review, rather than metrics, and with recommendations for weightings of outputs, environment and impact more or less the same. This is not a proposal to radically overhaul the system.
However, the review does contain some fairly significant tweaks which will have implications for both academics and research managers, if they are adopted. The review makes the following recommendations (in bold below – followed by my own comment and reflections):
- All research active staff should be returned in the REF: This was a contentious issue going by the responses to the consultation on the REF, with some concerned that it could lead to a greater distinction between staff on teaching only contracts and those whose contracts include research responsibilities. On the positive side, it should lead to less burdensome selection processes for HEIs and reduce the negative stigma of not being “REF-able”.
- Outputs should be submitted at Unit of Assessment level with a set average number per FTE but with flexibility for some faculty members to submit more and others less than the average: The suggestion in the review is that the minimum should be 0 and the maximum 6 per FTE submitted, but further work will be required to model this so that it doesn’t lead to a large increase in work for panels.
- Outputs should not be portable: This has caused the largest outcry on Twitter following the publication of the review, particularly among early career researchers, many of whom argue that it will make it much more difficult to get new jobs as their previous publications will not count towards the next REF. On the other hand, the review makes the case that this will discourage the so-called “transfer market” of REF staff before the deadline.
- Panels should continue to assess on the basis of peer review. However, metrics should be provided to support panel members in their assessment, and panels should be transparent about their use: The report recognises that bibliometric data is not appropriate for use in all units of assessment (following the Metric Tide review), but that it can be used “judiciously” to help panels in their peer review assessment.
- Institutions should be given more flexibility to showcase their interdisciplinary and collaborative impacts by submitting ‘institutional’ level impact case studies, part of a new institutional level assessment: This is a genuinely new part of the assessment and perhaps reflects the slightly amorphous nature of research impact assessment, which in many cases is difficult to tie down to a particular body of work which fits neatly within the boundaries of a single UoA.
- Impact should be based on research of demonstrable quality. However, case studies could be linked to a research activity and a body of work as well as to a broad range of research outputs: Again this appears to be about increasing the flexibility of what counts as impact and reducing the instrumental approach of linking research outputs directly to impacts. How this will play out in reality will, I imagine, depend heavily on precisely how this is interpreted in the guidance, assuming the recommendation is adopted.
- Guidance on the REF should make it clear that impact case studies should not be narrowly interpreted, need not solely focus on socioeconomic impacts but should also include impact on government policy, on public engagement and understanding, on cultural life, on academic impacts outside the field, and impacts on teaching: The fact that the review contains three recommendations wholly focusing on impact shows that this element of the assessment is still critical. Several people have pointed out on ARMA mailing lists that the guidance for REF2014 anyway allowed these kinds of impacts (except perhaps for academic impacts), so this might partly be about emphasising to institutions and panels that these are eligible impacts and should be taken seriously.
- A new, institutional level Environment assessment should include an account of the institution’s future research environment strategy, a statement of how it supports high quality research and research-related activities, including its support for interdisciplinary and cross-institutional initiatives and impact. It should form part of the institutional assessment and should be assessed by a specialist, cross-disciplinary panel: As widely predicted, the REF3a “impact statement” is part of a wider statement about research environment, though at the institutional level rather than the UoA level.
- That individual Unit of Assessment environment statements are condensed, made complementary to the institutional level environment statement and include those key metrics on research intensity specific to the Unit of Assessment: Recommendations 8 and 9 are listed together in the review and they do appear to complement each other. The focus at UoA-level (9) appears to be on shorter, punchier metrics-based evidence, while at institutional level (8) it is on more narrative-based plans and strategies.
- Where possible, REF data and metrics should be open, standardised and combinable with other research funders’ data collection processes in order to streamline data collection requirements and reduce the cost of compiling and submitting information: The focus here is on reducing the burden of the REF, and the review here acknowledges current events in the form of the TEF and the uncertainty about the future relationship between the UK and the EU. With the emphasis on standardisation and streamlined data collection, there is surely a role here for organisations like Jisc and CASRAI.
- That Government, and UKRI, could make more strategic and imaginative use of REF, to better understand the health of the UK research base, our research resources and areas of high potential for future development, and to build the case for strong investment in research in the UK: Ant Bagshaw suggests in his WonkHE post that this appears to be a “bit of a cheeky request” for cash.
- Government should ensure that there is no increased administrative burden to Higher Education Institutions from interactions between the TEF and REF, and that they together strengthen the vital relationship between teaching and research in HEIs: This again returns to the theme of the review which is to reshape the REF to reduce the burden on HEIs.
The timetable suggested by the review on p32 is also instructive and suggests a lot of work lies in store for the Government and Funding Councils: a consultation on concrete proposals for the next REF by the end of 2016, with decisions made in the summer of 2017. This will also need to be “checked for consistency” with the TEF, as the two exercises will evolve in parallel. The review suggests that this timetable could see a deadline for submissions by the end of 2020, with the assessment itself taking place in 2021.