Presentations from the HEFCE REFlections Event are now available



HEFCE held a conference on 25 March 2015 called ‘REFlections: Evaluation of the Research Excellence Framework (REF) 2014 and a look to the future’ You can now read the presentations from the event including analysis of impact case studies and access to a database of the case studies, feedback on the REF from institutional and REF panel perspectives and a look at the potential for metrics in future research assessments.


Bringing Research Impact to Life – staff session

Why not sign up to attend this informative one hour session run by Lucy Jowett, Research Impact Manager (RBS) on Tuesday 3rd March 2015, 12 – 1pm, City Campus?

The session will encourage researchers from all disciplines and at various career stages, to actively think about how they will achieve excellence with impact and detail the support available.

For more information visit the People Development pages on the Northumbria intranet.



The use of metrics in research assessment.

Ruth Hattam (Assistant Director for Research) recently attended a session about the prospects and pitfalls around the use of metrics in research assessment. The event was hosted by SPRU (Science and Policy Research Unit) based at the University of Sussex, which is undertaking the HEFCE review of metrics, the report for which is due in June 2015.  There was no indication of the likely outcomes, and Steven Hill (Head of Research Policy, HEFCE) was keen to stress that no decision had been made about metrics and the next REF.

The event was well-balanced with a variety of views on the issue from a number of speakers.   There seemed to be a broad consensus that metrics alone should not be used to assess research, with general support for a mix of qualitative and quantitative approaches, although which should come first, or have prominence, was not resolved.

As an observation, those speakers with an interest in promoting metrics were careful to stress that metrics are only one indicator, whilst some speakers on the other side of the debate were more forceful in their criticism of the use of metrics, arguing that they were an unreliable means of assessment.  One speaker used his own citations to illustrate this point, asserting that his most frequently cited articles did not correlate with his best research. Some of the other general discussion points included: metrics could only potentially be useful as an indicator of significance in the three REF criteria for outputs (originality, significance and rigour); issues around impact metrics; peer review is a far from perfect system, potentially subject to individual bias; the public interest should dominate; use of Altmetrics (e.g. social media, blog posts – anything that isn’t citation-based).

The event featured a ‘metrics bazaar’ which allowed participants to explore metric tools and platforms with a range of developers and providers.   Of interest was an overview of ‘The Conversation’ which is an independent source of news and views sourced from the academic and research community and delivered direct to the public.

The afternoon session explored the ‘darker side of metrics’, although the speakers did not perhaps delve into some of the gaming practices which have been unearthed (e.g. self-citation malpractice uncovered at the Journal for Business Ethics.  Some of the discussion points included: the number of retractions is on the rise including  in ‘prestigious’ journals;  the sector had to be realistic and accept the principle of measurement as had other public-funded sectors (e.g. health); that use of metrics would potentially change behaviour; the term ‘metrics’ should be replaced by the term ‘indicators’;  arts and humanities academics needed to engage in the debate.


British Academy highlights the need for variety of Open Access

British Academy highlights the need for variety of Open Access: “This includes consideration of the types of open access licences that should be permitted, the submission says. It believes the Attribution-NonCommercial-NoDerivations (CC-BY-NC-ND) licence will usually be more appropriate for humanities and social science publications because this offers more safeguards against the misuse of work.”

More information here (via Research Professional).


Analysis of REF Impact Case Studies

The Policy Institute at King’s College London and Digital Science will be analysing the case studies submitted to the REF to illustrate the impact of research beyond academia. The analysis has been commissioned by the Higher Education Funding Council for England and its partners in the REF exercise. The work aims to maximise the value of the 6,975 case studies as a separate resource, analysing them and identifying what they can show about the wider impact of the research conducted by UK universities. The case studies will be made freely available for analysis in a database to be hosted on the HEFCE website. The outputs will be a well-curated, readily searchable database of the case studies, an overview report describing the strength of UK science, and a view on how the case study approach works in assessing and auditing impact.

The exercise will not affect the quality-related block funding awarded to institutions by the government, as the case studies have already been incorporated into that process.

See more at:


HEFCE announce review of metrics in research assessment

Remember when HEFCE suggested replacing peer review with metrics after RAE2008? They backed off fairly quickly following strong opposition, and the REF ended up, as the RAE, largely based on peer review. Well, now that REF2014 is over, talk of metrics-based research assessment is back and HEFCE have just announced an independent review, chaired by Prof James Wilsdon. It will report by spring 2015. More from THE [£] and Research Fortnight [£].


HEFCE Open Access Policy for post-2014 REF

Open Access promomaterial by biblioteekje CC BY-NC-SA 2.0HEFCE has today released its open access policy for the post-2014 Research Excellence Framework.

I was intending to write a short summary this morning, but I’ve already been beaten to it by the ever-excellent Martin Eve, who’s got a “Really Short Version” in four bullets, and an extended abridged version if you want to dig a little deeper.

Martin’s full summary is posted below thanks to the magic of open access (Creative Commons Attribution license to be specific). This guide is aimed at academic staff, so it doesn’t cover some of the stuff around “discoverability” of outputs since that will largely be handled by institutional repository staff. (Incidentally, you shouldn’t worry about the veracity of this account – Martin’s version has already been praised by Ben Johnson, the HEFCE policy manager who co-authored the original policy, on Twitter):

The Really Short Version:

  1. Submit journal article.
  2. Check journal policy at (the bits on “post-print” are the thing to pay attention to).
  3. On acceptance go to your institution’s repository and create a record. Upload your author’s accepted version setting the embargo as per SHERPA/RoMEO [at Northumbria repository staff are available to help with this and check it’s set it correctly].
  4. That’s it.

The Geekier Longer Version
Source: “Policy for Open Access in the Post-2014 Research Excellence Framework“, March 31, 2014


  • Applies to all: journal articles and published conference papers (with ISSN) accepted after 1 April 2016. (Paras 11, 13)
  • Exempted: monographs (“and other-long form publications”), edited collections (without ISSN), non-text outputs, data. (Para 14)
  • Outputs to which this applies are subject to open access deposit, discovery and access requirements. (Para 16) It is anticipated that the “discovery” requirement will be met at the institutional and technological level.
  • “Credit will be given” to institutions exceeding the letter of this policy in a future “research environment” component. (Para 15)
  • “Non-compliant outputs will be given an unclassified score and will not be assessed in the REF.” (Para 42)

Deposit Requirements

Authors MUST:

  • upload the “accepted and final peer-reviewed text” to “an institutional repository, a repository service shared between multiple institutions, or a subject repository such as arXiv”. (Paras 17, 19)
  • do so “as soon after the point of [firm (Para 19)] acceptance as possible, and no later than three months after this date”. (Para 18)

Allowable exceptions:

  • the author did not work at an HEI at time of acceptance.
  • “it would be unlawful to deposit, or request the deposit of, the output”.
  • “depositing the output would present a security risk”. (Para 36)

Authors MAY:

  • upload a subsequent version as a supplement or replacement if the publisher allows it. (Para 19) If a replacement, it must also fulfil the access requirement (Para 33).

Access Requirements

Authors MUST:

  • allow others “search electronically within the text, read it and download it without charge”. (Para 25)
  • respect any ‘embargo period’ (an open access delay) specified by the publisher. (Para 25)
  • but an embargo MUST NOT be longer than 24 months for panels C and D and 12 months for panels A and B. (Para 30)
  • regardless of whether publisher specifies delay, you MUST deposit at time of acceptance (deposit requirement)
  • no specific license is required. It is suggested that CC BY-NC-ND could meet the above provisions. (Para 25)
  • if provisions are made to allow text-mining, which could include more liberal licensing, then credit will be given in the “environment” component. (Para 34)

The ONLY allowable exceptions:

  • third-party rights couldn’t be obtained for material within to be made OA
  • the embargo period was above the maximum allowed, or the journal disallows deposit, but the venue was “the most appropriate” (Para 37)

HEFCE Consultation on Open Access post-REF

Open Access promomaterial by biblioteekje CC BY-NC-SA 2.0Both myself and Ruth Hattam recently attended events organised by HEFCE focussing on the recent consultation paper around Open Access (OA) in the post-2014 REF.  The principle behind the proposals is that outputs submitted for the next REF submission (likely to be around 2020) should be open access.

Definitions first

For a publication to be classed as OA according to the criteria proposed by HEFCE, it must:

  • be accessible through an institutional repository immediately upon either acceptance or publication (to be decided as part of the consultation);
  • be the final peer-reviewed text (though not necessarily identical to the publisher’s edited and formatted version); and
  • be presented in a form allowing the reader to search for and re-use content subject to proper attribution.

HEFCE acknowledges the challenges around these proposals, and the system is designed to move towards the principle of full OA. As such, the proposal is that only journal articles and conference proceedings will be expected to comply with OA for the next REF, and that the OA criteria will only apply two years after the date of the policy announcement (i.e. from 2016).

The proposals as they stand mean that outputs which are retrospectively made to comply with the above definition of OA would not be eligible for submission to the REF. There were various issues raised at the consultation events about how this principle might work in cases where staff move institutions, bringing outputs with them. Delegates noted that this would effectively demand an extra administrative step on recruitment to check that outputs published while at the previous institution complied with REF OA guidelines.

Targets and exceptions

HEFCE propose that exceptions to the policy will be permitted, and one of the consultation questions is whether this should be on a case-by-case basis, a set percentage (70% is the suggestion) across an institutional submission, or whether targets are varied on a panel basis to take account of subject differences.

The audience from a variety of HEIs was broadly in favour of the open access proposal, in fact, some felt the proposal did not go far enough in progressing an open access approach.  One view was that tolerating up to 30% non-compliance could be seen as a way for institutions/academics to circumvent the OA principle.

The case-by-case approach was seen as potentially bureaucratic, however, and there was some difficulty in establishing the sort of circumstances that would qualify as an ‘exception’.  At this stage, HEFCE felt unable to expect 100% compliance without raising serious questions about academic freedom.

Publish or perish?

Publishers (and their ever-changing policies on OA) came in for some criticism, but the HEFCE panel was keen to stress that they see the publishing industry as part of the solution rather than the problem. One suggestion was that HEFCE and other funders invest in the SHERPA RoMEO/FACT service to ensure a robust and accurate information service which has capacity to keep track of the shifting publisher OA policies. It was further noted in this context that the issues around monographs were currently too complex to try and tackle in a meaningful way for the post-2014 REF.

Concerns were raised about the additional costs of supporting OA. The HEFCE panel view was that costs did not have to be a major issue if institutions developed policies that allowed for both green and gold OA. The suggestion was that the HEFCE REF guidelines should be viewed as a minimum standard.

Resource requests

The use of institutional repositories would clearly be key in the proposed changes, but the additional resources required to ensure that repositories were used most effectively was seen as an issue.  For instance, the implications of embargoes on outputs was seen as something that would be difficult to monitor, particularly as there was no standard approach among publishers and the parameters were constantly changing.  If publishers extended their embargoes beyond the RCUK recommendations (which were likely to be adopted by HEFCE) then it would be up to an institution whether to choose to pay gold OA fees in order to have outputs available to the REF submission.

Other discussion covered the potential for OA to form part of the REF5 narrative, and it was noted that further consideration needed to be given to the penalties if institutions missed OA targets. HEFCE commented that it was unlikely that OA constraints would be placed on research that supported impact.  There was a strong argument for consistency as far as possible between and across panels.

Comments please!

Anyone who wishes to read the full consultation document from HEFCE can access it at the following link:

If any member of Northumbria staff wishes to comment on any of the seven questions (in Appendix A of the document) please send views to me (email: by 18th October 2013.