Stern Review of REF Published: Some Initial Thoughts

Measuring Tape by Evan Amos - Public Domain
Measuring Tape by Evan Amos – Public Domain

The Stern Review of the Research Excellence Framework has been published today.

Broadly speaking the review recommends keeping the REF more or less the same as before – i.e. still a periodic exercise based predominantly on peer review, rather than metrics, and with recommendations for weightings of outputs, environment and impact more or less the same. This is not a proposal to radically overhaul the system.

However, the review does contain some fairly significant tweaks which will have implications for both academics and research managers, if they are adopted. The review makes the following recommendations (in bold below – followed by my own comment and reflections):

  1. All research active staff should be returned in the REF: This was a contentious issue going by the responses to the consultation on the REF, with some concerned that it could lead to a greater distinction between staff on teaching only contracts and those whose contracts include research responsibilities. On the positive side, it should lead to less burdensome selection processes for HEIs and reduce the negative stigma of not being “REF-able”.
  2. Outputs should be submitted at Unit of Assessment level with a set average number per FTE but with flexibility for some faculty members to submit more and others less than the average: The suggestion in the review is that the minimum should be 0 and the maximum 6 per FTE submitted, but further work will be required to model this so that it doesn’t lead to a large increase in work for panels.
  3. Outputs should not be portable: This has caused the largest outcry on Twitter following the publication of the review, particularly among early career researchers, many of whom argue that it will make it much more difficult to get new jobs as their previous publications will not count towards the next REF. On the other hand, the review makes the case that this will discourage the so-called “transfer market” of REF staff before the deadline.
  4. Panels should continue to assess on the basis of peer review. However, metrics should be provided to support panel members in their assessment, and panels should be transparent about their use: The report recognises that bibliometric data is not appropriate for use in all units of assessment (following the Metric Tide review), but that it can be used “judiciously” to help panels in their peer review assessment.
  5. Institutions should be given more flexibility to showcase their interdisciplinary and collaborative impacts by submitting ‘institutional’ level impact case studies, part of a new institutional level assessment: This is a genuinely new part of the assessment and perhaps reflects the slightly amorphous nature of research impact assessment, which in many cases is difficult to tie down to a particular body of work which fits neatly within the boundaries of a single UoA.
  6. Impact should be based on research of demonstrable quality. However, case studies could be linked to a research activity and a body of work as well as to a broad range of research outputs: Again this appears to be about increasing the flexibility of what counts as impact and reducing the instrumental approach of linking research outputs directly to impacts. How this will play out in reality will, I imagine, depend heavily on precisely how this is interpreted in the guidance, assuming the recommendation is adopted.
  7. Guidance on the REF should make it clear that impact case studies should not be narrowly interpreted, need not solely focus on socioeconomic impacts but should also include impact on government policy, on public engagement and understanding, on cultural life, on academic impacts outside the field, and impacts on teaching: The fact that the review contains three recommendations wholly focusing on impact shows that this element of the assessment is still critical. Several people have pointed out on ARMA mailing lists that the guidance for REF2014 anyway allowed these kinds of impacts (except perhaps for academic impacts), so this might partly be about emphasising to institutions and panels that these are eligible impacts and should be taken seriously.
  8. A new, institutional level Environment assessment should include an account of the institution’s future research environment strategy, a statement of how it supports high quality research and research-related activities, including its support for interdisciplinary and cross-institutional initiatives and impact. It should form part of the institutional assessment and should be assessed by a specialist, cross-disciplinary panel: As widely predicted, the REF3a “impact statement” is part of a wider statement about research environment, though at the institutional level rather than the UoA level.
  9. That individual Unit of Assessment environment statements are condensed, made complementary to the institutional level environment statement and include those key metrics on research intensity specific to the Unit of Assessment: Recommendations 8 and 9 are listed together in the review and they do appear to complement each other. The focus at UoA-level (9) appears to be on shorter, punchier metrics-based evidence, while at institutional level (8) it is on more narrative-based plans and strategies.
  10. Where possible, REF data and metrics should be open, standardised and combinable with other research funders’ data collection processes in order to streamline data collection requirements and reduce the cost of compiling and submitting information: The focus here is on reducing the burden of the REF, and the review here acknowledges current events in the form of the TEF and the uncertainty about the future relationship between the UK and the EU. With the emphasis on standardisation and streamlined data collection, there is surely a role here for organisations like Jisc and CASRAI.
  11. That Government, and UKRI, could make more strategic and imaginative use of REF, to better understand the health of the UK research base, our research resources and areas of high potential for future development, and to build the case for strong investment in research in the UK: Ant Bagshaw suggests in his WonkHE post that this appears to be a “bit of a cheeky request” for cash.
  12. Government should ensure that there is no increased administrative burden to Higher Education Institutions from interactions between the TEF and REF, and that they together strengthen the vital relationship between teaching and research in HEIs: This again returns to the theme of the review which is to reshape the REF to reduce the burden on HEIs.

The timetable suggested by the review on p32 is also instructive and suggests a lot of work lies in store for the Government and Funding Councils: a consultation on concrete proposals for the next REF by the end of 2016, with decisions made in the summer of 2017. This will also need to be “checked for consistency” with the TEF, as the two exercises will evolve in parallel. The review suggests that this timetable could see a deadline for submissions by the end of 2020, with the assessment itself taking place in 2021.

 

Share

RMAS Launch: What it is and how it can support research management

I recently attended the RMAS (Research Management and Administration System) launch event in London where I learned about the outcomes of the recent HEFCE-funded project and how RMAS can help people like me (research managers and administrators) better understand, track and integrate the vast and diverse array of research-related data in Higher Education.

Universities need this data for a number of reasons, including:

  • to analyse and interpret success rates of research bids so we can see where more support is needed;
  • to keep track of and audit research expenditure to ensure externally-funded projects are spending to budget;
  • to feed into important external research-related returns, such as HESA, HEBCIS and REF;
  • to ensure research projects are accurately costed;
  • to make sure the research bidding and management process flows as smoothly as possible, minimizing the amount of data re-entry and hand-offs where possible;
  • as the open access agenda becomes more important, to ensure all outputs from funded projects are published in open access journals or are available as full text on the institutional repository

What is RMAS?

RMAS started out as a HEFCE-funded project, led by the University of Exeter and also involving Kent and Sunderland, to scope and produce a business case for a pre-award to post-award (“cradle to grave”) research management and administration system for HEIs. Phase one of the project confirmed there was a need for this among HEIs and that no such system was currently available on the open market. Phase two, which has just been completed, sought to develop a solution and estimate potential savings across the sector if this were adopted.

So what’s the outcome? In short RMAS consists of three related elements:

  1. RMAS includes a procurement framework containing “best of breed” systems which provide solutions for the full research management process: pre-award to post-award. This will substantially reduce the costs of procuring research management and administration software by effectively allowing HEIs to pick from a “catalogue” of options for each stage in the process. It also ensures RMAS is “modular”, allowing HEIs to start from different places and pick what they need. Suppliers in the framework include: UNIT4 (Agresso), Atira (Pure), SmartSimple (RMS360).
  2. RMAS is a set of integration tools and methodologies based on a centrally hosted Enterprise Service Bus which allows existing and newly procured corporate and research management systems to be integrated and communicate with each other.
  3. The above elements are built around a standard format for research data, CERIF (Common European Research Information Format), which is an open, internationally recognised standard. This means that research data can be integrated and different systems can communicate more easily. In essence, it ensures that all the different elements of the RMAS system “speak the same language”.

What’s the problem?

Research carried out by the RMAS pathfinders across the sector confirmed that different HEIs used a variety of different products for different stages of the process (e.g. project costing, academic expertise, post award management, outcomes and outputs) and that there was often very little, if any, integration between the systems used. This means that data relating to research awards, for example, frequently needs to be re-keyed at various stages to ensure it is correctly imported into the relevant system.

This situation is compounded by the fact that the various HR and finance systems which also feed information into (and out of) the research management process are similarly diverse and lack integration with the various research management software solutions on the market. Moreover, different institutions have different processes and procedures – for example, approvals and submission.

In short, HEIs are all starting from a different place and no two institutions use the same set of systems and processes for research management and administration. In addition, institutions have in many cases invested heavily in their existing systems and tools and would be reluctant to throw this away. The RMAS team therefore determined that what was needed was a methodology of integrating these existing tools, rather than developing and introducing an entirely new system from scratch.

What are the benefits?

Potentially, these are huge. The RMAS pathfinder institutions (Exeter, Kent and Sunderland) have made the following estimates in terms of savings as well as associated data quality and planning improvements:

You could expect the following chain benefits from implementing RMAS modules:

  • Procurement savings of around £35,000 for each OJEU tender
  • Procurement savings of around £14,000 for each sub-OJEU tender
  • Savings for a medium sized university of £75,000 per annum for each ofthe RMAS modules that they use.

The operational savings equate to:

  • £60 per application for proposal management systems
  • £140 per academic or £70 per project for post-award management systems
  • £70 per academic for outputs management systems.

Ripple benefits that the pathfinder institutions have experienced include:

  • Improvements in data quality in corporate systems such as HR, Finance and Project systems
  • Improved accuracy of Business Intelligence and planning
  • Value added through redirection of valuable resource
  • A flexible platform that can be readily adapted to any future requirements
  • A positive user experience facilitating future developments and new system deployments.
These benefits should be understood in terms of the wider policy drive in UK HE to share services and become more efficient. HEFCE are clear that RMAS can contribute towards this goal.

What next?

There was a strong steer from HEFCE representatives on the day that the RMAS approach should be adopted across the sector. To this end, HEFCE have provided further transitional funding to retain the project coordinator for a year and to develop resources on the RMAS website, which includes an RMAS connector demo so you can see some of the principles in action.

However, it became clear on the day that for institutions to actually adopt an end-to-end RMAS-compliant system would require not only financial investment, in terms of procuring relevant software to plug gaps in the research management process, but also in terms of IT development time to build a “connector” to join up the various elements of research admin systems and HR, finance, etc. There will be resources available on the RMAS website to provide guidance and support for this, or alternatively JISC Nexus has been set up as a subscription service to do the same job.

In the question time following the event, one delegate made the very sensible point that they wouldn’t be doing anything further on this until after the REF, for fear of introducing new processes and jeopardising data quality. Whatever the timescales, it seems clear that RMAS is going to play a role in the future of research management in UK HE.

Share

New National Centre for University-Business Partnerships

HEFCE and the Council for Industry and Higher Eduction announced this week a new National Centre for Universities and Business. The centre, funded by HEFCE initially, aims to increase university-business collaborations in order to improve economic growth. According to the HEFCE announcement:

The centre will publish an annual ‘State of the relationship report’ which is intended to become the premier influence on policy development in the area of HE-business links. It will also conduct enquiries into major matters of HE and business interest. This will include examining the impact of the new student funding and fees regime on graduate recruitment and the longer-term business workforce. The Centre will be on hand to offer services to HE and business, such as good practice developments and support in establishing international links.

It’s unclear at this stage whether there will be any funding available for research-industry collaborations. We need to wait for a business plan to be developed by the steering group, which is due by autumn 2012. However, is it is likely to be linked closely the existing HEIF strand of funding already provided by HEFCE.

Share