Evaluating the Submission Process for the Research Excellence Framework's Impact Element

Background

The Research Excellence Framework (REF) 2014 was a nationwide initiative to assess the quality of research in UK higher education institutions (HEIs). For the first time, REF 2014 introduced the wider impact of research, alongside the quality of research and the vitality of the research environment, into the assessment of research excellence in UK HEIs. The impact of research was weighted at 20% of the total assessment for 2014.

Understanding how the impact element of the REF 2014 submission process worked for HEIs and research users will be important for future rounds of assessment.

Goals

The higher education funding councils for England, Scotland and Wales asked RAND Europe to work with a sample of HEIs to evaluate the submission process for the impact element of the REF 2014. This formative evaluation aimed to:

• Describe the perceived benefits and burdens to HEIs and research users in preparing their impact submissions
• Identify intended and unintended consequences of assessing research impact for different institutions and disciplines
• Formulate evidence-based conclusions and recommendations for improving the processes of preparing submissions for impact assessment
• Highlight innovative and good practices for institutions, research users, the funding councils and other stakeholders.

For this study we worked with 21 HEIs to understand their experience with the impact submission process and consulted with three broad stakeholder groups over the course of the evaluation:

• HEI research leadership teams
• HEI impact case study subjects and authors
• Non-academic research users

A mixed methods approach was used including site visits, face-to-face and telephone interviews, online surveys and a ‘benefit:burden’ analysis to produce robust conclusions and recommendations.

Key Findings

• Academics and institutions reported a number of benefits of engagement with the impact component of REF 2014, such as developing the ability to identify and understand impact, and the stimulation of broader strategic thinking about impact.
• Complying with the impact component of the REF imposed a burden on resources and time for individuals and for the sector as whole. Academics perceived the process to be overly burdensome to research users who were involved to provide evidence, but this was not the experience of research users.
• A cultural change appears to be taking place whereby institutions and individual academics are adopting a new focus on the current and potential impacts of their research. Alongside a wider impact agenda, REF2014 has contributed to this shift.

Publications

• 

  Preparing impact submissions for REF 2014: An evaluation: Findings and observations

  Catriona Manville, Molly Morgan Jones, Michael Frearson, Sophie Castle-Clarke, Marie-Louise Henham, Salil Gunashekar, Jonathan Grant

  As part of a review of the preparation process for the impact element of the REF 2014 within UK higher education institutions to assess the process and identify future improvements, this report summarises the approach and describes the key findings and observations.

• 

  Preparing impact submissions for REF 2014: An evaluation: Approach and Evidence

  Catriona Manville, Molly Morgan Jones, Marie-Louise Henham, Sophie Castle-Clarke, Michael Frearson, Salil Gunashekar, Jonathan Grant

  As part of a review of the preparation process for the impact element of the REF 2014 within UK higher education institutions to assess the process and identify future improvements, this report provides an in-depth analysis of the gathered data.