On the reliability of predictions on Covid-19 dynamics: A systematic and critical review of modelling techniques

Abstract

Since the emergence of the novel 2019 coronavirus pandemic in December 2019 (COVID-19), numerous modellers have used diverse techniques to assess the dynamics of transmission of the disease, predict its future course and determine the impact of different control measures. In this study, we conducted a global systematic literature review to summarize trends in the modelling techniques used for Covid-19 from January 1st^, 2020 to November 30th^, 2020. We further examined the accuracy and precision of predictions by comparing predicted and observed values for cumulative cases and deaths as well as uncertainties of these predictions. From an initial 4311 peer-reviewed articles and preprints found with our defined keywords, 242 were fully analysed. Most studies were done on Asian (78.93%) and European (59.09%) countries. Most of them used compartmental models (namely SIR and SEIR) (46.1%) and statistical models (growth models and time series) (31.8%) while few used artificial intelligence (6.7%), Bayesian approach (4.7%), Network models (2.3%) and Agent-based models (1.3%). For the number of cumulative cases, the ratio of the predicted over the observed values and the ratio of the amplitude of confidence interval (CI) or credibility interval (CrI) of predictions and the central value were on average larger than 1 indicating cases of inaccurate and imprecise predictions, and large variation across predictions. There was no clear difference among models used for these two ratios. In 75% of predictions that provided CI or CrI, observed values fall within the 95% CI or CrI of the cumulative cases predicted. Only 3.7% of the studies predicted the cumulative number of deaths. For 70% of the predictions, the ratio of predicted over observed cumulative deaths was less or close to 1. Also, the Bayesian model made predictions closer to reality than classical statistical models, although these differences are only suggestive due to the small number of predictions within our dataset (9 in total). In addition, we found a significant negative correlation (rho = - 0.56, p = 0.021) between this ratio and the length (in days) of the period covered by the modelling, suggesting that the longer the period covered by the model the likely more accurate the estimates tend to be. Our findings suggest that while predictions made by the different models are useful to understand the pandemic course and guide policy-making, some were relatively accurate and precise while other not.

Keywords: Accuracy; Pandemic; Precision; Predictions; Ratio; SARS-CoV-2.

Fig. 1

PRISMA flow diagram of the selection process of the 242 studies included in the systematic review.

Fig. 2

Distribution of studies across continents (a) and countries coverage across continents (b).

Fig. 3

Distribution of studied articles according to (a) whether the models used were data-driven or not, (b) whether the models included asymptomatic and/or pre-symptomatic individuals, (c) the topic addressed, (d) the control measures assessed, and (e) the key epidemiological parameters estimated.

Fig. 4

Diversity of modelling techniques used for Covid-19 (a), and topics addressed with the modelling techniques (b).

Fig. 5

Accuracy of the models’ predictions: ratio of the number of cumulative cases predicted over the actual number of cumulative cases observed in 33 studies (a), across types of models (b), according to whether models included asymptomatic or pre-symptomatics (c), according to whether models were parameterized based on real data (d), and in relationships to the number of days since the first case was reported in concerned countries (e). Others (see appendix C). Values in parentheses in (b) and (d) represent the number of predictions found for each case.

Fig. 6

Precision of the models’ predictions: ratio of the amplitude of the 95%CI or 95%CrI of the predicted cumulative number of cases over the predicted cumulative number of cases in 14 studies (a), across types of models (b), according to whether models included asymptomatic or pre-symptomatics (c), according to whether models were parameterized based on real data (d), and in relationships to the number of days since the first case was reported in concerned countries (e). Values in parentheses in (b) and (d) represent the number of predictions found for each case.

Fig. 7

Distributions of predictions (20) of the cumulative number of cases according to whether the values actually observed for the predictions fall within the 95%CI or 95%CrI of the prediction.

Fig. 8

Accuracy of the models’ predictions: ratio of the number of cumulative deaths predicted over the actual number of cumulative deaths observed in nine studies (a), across types of models (b), according to whether models included asymptomatic or pre-symptomatics (c), and in relationships to the number of days since the first case was reported in concerned countries (d). Values in parentheses in (b) and (c) represent the number of predictions found for each case.

Fig. 9

Precision of the models’ predictions: ratio of the amplitude of the 95%CI or 95%CrI of the predicted cumulative number of deaths over the predicted cumulative number of deaths in nine studies (a), across types of models (b), according to whether models included asymptomatic or pre-symptomatics (c), and in relationships to the number of days since the first case was reported in concerned countries (d). Values in parentheses in (b) and (c) represent the number of predictions found for each case.

Fig. 10

Distributions of predictions (10) of the cumulative number of deaths according to whether the values actually observed for the predictions fall within the 95%CI or 95%CrI of the prediction.