Standardising quantitative indicators of impact within REF case studies

Background

The Research Excellence Framework (REF), first conducted in 2014, assesses the quality of research in UK higher education institutions (HEIs). REF 2014 demonstrated that research conducted by universities can lead to impact beyond academia – for example, on society, the economy, public policy, and health – within the UK and beyond. In REF 2014, HEIs were invited to submit examples of impact as case studies. Combining a compelling narrative, with the underpinning research and citations of evidence to corroborate the impact, was seen as a 'workable approach'. Accordingly, in REF 2021, impact will continue to be assessed through the use of case studies.

Goals

RAND Europe was commissioned by HEFCE (now Research England) on behalf of the four UK higher education funding bodies, to:

• Identify quantitative indicators of impact in the case studies submitted to REF 2014; and
• Develop guidance for how these indicators could be standardised for potential use in REF 2021.

The study is intended to inform the development of the guidance and criteria for the preparation of impact case studies in REF 2021.

Methodology

We adopted a mixed-methods approach to address the study objectives. This involved a blend of text mining and qualitative techniques:

1. Rapid review of the literature to understand the kinds of quantitative indicators that might be used in impact case studies (to inform our search of the REF 2014 case studies)
2. Construction and analysis of the data set of sentences containing quantitative indicators of impact from the REF 2014 impact case studies (using text mining and qualitative techniques)
3. Initial workshop to identify areas for potential standardisation and develop the corresponding standards themselves
4. Testing of proposed guidance (involving consultation with the Forum for Responsible Research Metrics and interviews with individuals working within the higher education sector)
5. Synthesis of evidence and reporting

Findings

We propose guidance for two broad categories of standards. First, we outline a ‘style guide’ that contains general stylistic items that could be standardised in order to make quantitative indicators of impact, and specific formulations of them, more discoverable in the case studies in general. Second, we detail ‘specific guidance’ that addresses more specific and commonly occurring quantitative indicators that have been used as evidence of impact in the REF 2014 case studies.