Measuring the impact of science: An overview of the 2014 Research Excellence Framework report

How much of an impact does research have outside of science? Is scientific impact quantifiable? These quintessential questions have been pondered over by academicians since the beginning of scientific research. However, the globalization of science has necessitated a growing need to assess what kind and how much of an impact the investment in science is generating. As a result, funding bodies in many countries have taken upon the task of evaluating the quality of research and its potential impact on society. An example of such an analysis is the latest university research audit exercise, the Research Excellence Framework (REF), undertaken by UK’s higher education funding councils.    

The REF measures the impact of research conducted by British higher education institutions, and helps funding bodies make decisions about allocating funds to universities for a period of six years. Based on the samples of research that universities submit, the REF panelists (that includes academics and industry experts) analyze the research undertaken and the impact it generated. They rate the impact of research in various subject areas as well as the credibility of the universities conducting research. The recently published REF report is a collection of accounts of the societal impact claimed by research, which has produced the REF impact case study database. This database is a searchable tool enabling analysis and automated text mining of the case studies, and can be accessed by interested parties in the UK and internationally for further analysis. Some of the research outputs submitted include research projects that improved water quality and the performance of the water industry, quality of care delivery in health-care centers, etc., which have had a global impact.

Facts about the latest report

The REF project was a massive undertaking, and took the panel a year to review the submitted research outputs. These are some facts related to the 2014 REF report:

• 154 UK higher education institutions took part in the exercise.
• They made 1,911 submissions that included 191,150 research outputs and 6,975 impact case studies.
• 4 main panels and 36 sub-panels, comprising 898 academic members 259 research users, reviewed the submissions.
• The panels assessed the quality of outputs against the criteria of ‘originality, significance and rigour.’

Key findings included in the report

The REF report notes that the quality of submitted research outputs has improved exponentially in comparison to the earlier analysis that was called Research Assessment Exercise. It also reports the following:

• The average proportion of the outputs sub-profile judged to be worldleading has risen from 14 per cent in RAE 2008 to 22 per cent in REF 2014. The average proportion judged to be internationally excellent has risen from 37 per cent to 50 per cent.
• Research income flowed in from a range of sources including 38 per cent from the UK Research Councils, 19 per cent from UK government bodies, 19 per cent from UK charities, 6 per cent from UK industry, and 9 per cent from EU government bodies.
• Over 80 per cent of the REF impact case studies included underpinning research from multiple disciplines;
• The impact case studies were diverse and wide-ranging, with over 60 unique ‘impact topics’ identified. 
• Excellence was found across all types and forms of research including applied, basic, practice-based and strategic research; and in all forms of research endeavour including collaborative, interdisciplinary and multidisciplinary research.
• The reported research impacts stemmed from research in a wide-range of subject areas, with over 3,700 unique pathways from research to impact identified; and
• Research undertaken in UK higher education institutions (HEIs) has made a contribution to every country in the world.

The report reflects how the quality of research affects funding policies and throws light on the commercialization of research. UK’s research has benefitted the society, economy, and the quality of life of people locally and globally, it concludes.    

Related Infographic

5 Ways to identify a conflict of interest

A conflict of interest arises whenever there is any…

6k

Expert views on the REF report

Although the REF report is a detailed account of the quality and value of research work being carried out in the UK, some experts have pointed out flaws in the project. The preparation and submission of research outputs and the year-long analysis of these submissions have cost a significant amount of time and money. As Derek Sayer, a professor of history at Lancaster University, puts it, “The official bill for this six-yearly academic Battle Royale is around £47m spent within universities and a further £12m in HEFCE’s administrative costs – most of it taxpayers’ money.”

Some criticism has also been directed towards the method of assessment of research output. Sayer notes that the REF does not draw on international peer reviewing standards. The panelists were majorly experts affiliated with British universities. Furthermore, on some panels, only one assessor would evaluate submitted outputs, which can likely limit holistic assessment. Possibly, this is the reason behind the lack of standardization in the way impact statements were presented, as mentioned in an editorial in Nature.

A more consensual approach is required for identifying the qualitative and quantitative outcomes of research. Another concerning issue some experts have raised is that the REF’s sketch might discourage innovation. Since the submissions would have a direct impact on the funding universities receive, they would try to tailor their submissions according to what they think would get a nod from the panelists. In this bargain, research that is redefining or marks the start of a new field might not be displayed.        

Related Video

An overview of CrossRef and its products

In this interview snippet, Rachael Lammey, Product…

12.1k

Nonetheless, the REF report is evidence that the global impact of research on society and the quality of life of people can be documented. It is an encouraging step towards understanding the big picture of how science is progressing and would provide inroads to guiding the pace and direction of scientific advancement.