Using prediction markets to forecast research evaluations

Abstract

The 2014 Research Excellence Framework (REF2014) was conducted to assess the quality of research carried out at higher education institutions in the UK over a 6 year period. However, the process was criticized for being expensive and bureaucratic, and it was argued that similar information could be obtained more simply from various existing metrics. We were interested in whether a prediction market on the outcome of REF2014 for 33 chemistry departments in the UK would provide information similar to that obtained during the REF2014 process. Prediction markets have become increasingly popular as a means of capturing what is colloquially known as the 'wisdom of crowds', and enable individuals to trade 'bets' on whether a specific outcome will occur or not. These have been shown to be successful at predicting various outcomes in a number of domains (e.g. sport, entertainment and politics), but have rarely been tested against outcomes based on expert judgements such as those that formed the basis of REF2014.

Keywords: prediction market; research evaluation; research excellence framework.

Figure 1.

Comparison of performance of three methods of REF2014 prediction. The relationship between the predicted score (y-axis) and observed score (x-axis) is shown for overall REF2014 score, based on three prediction methods: survey (filled circle), weighted survey (open circle) and prediction market (square). Despite a higher correlation between survey-based predictions and observed outcomes, forecasting errors were lower for the market-based prediction, where the range of predictions matches the range of observed outcomes much better.