Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2018 Mar 27;13(3):e0194889.
doi: 10.1371/journal.pone.0194889. eCollection 2018.

Statistical and Machine Learning forecasting methods: Concerns and ways forward

Spyros Makridakis  1 , Evangelos Spiliotis  2 , Vassilios Assimakopoulos  2
Affiliations

Statistical and Machine Learning forecasting methods: Concerns and ways forward

Spyros Makridakis et al. PLoS One. .

Abstract

Machine Learning (ML) methods have been proposed in the academic literature as alternatives to statistical ones for time series forecasting. Yet, scant evidence is available about their relative performance in terms of accuracy and computational requirements. The purpose of this paper is to evaluate such performance across multiple forecasting horizons using a large subset of 1045 monthly time series used in the M3 Competition. After comparing the post-sample accuracy of popular ML methods with that of eight traditional statistical ones, we found that the former are dominated across both accuracy measures used and for all forecasting horizons examined. Moreover, we observed that their computational requirements are considerably greater than those of statistical methods. The paper discusses the results, explains why the accuracy of ML models is below that of statistical ones and proposes some possible ways forward. The empirical results found in our research stress the need for objective and unbiased ways to test the performance of forecasting methods that can be achieved through sizable and open competitions allowing meaningful comparisons and definite conclusions.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Fig 1
Fig 1. The three possible multi-step-ahead forecasting approaches used by NNs.
(a) The iterative, (b) the direct and (c) the multi-neural network forecasting approach.
Fig 2
Fig 2. Forecasting performance (sMAPE) of the ML and statistical methods included in the study.
The results are reported for one-step-ahead forecasts having applied the most appropriate preprocessing alternative.
Fig 3
Fig 3. Forecasting performance (sMAPE) versus model fitting.
The results are reported for one-step-ahead forecasts having applied the most appropriate preprocessing alternative.
Fig 4
Fig 4. Forecasting performance (sMAPE) versus computational complexity.
The results are reported for one-step-ahead forecasts having applied the most appropriate preprocessing alternative.
See this image and copyright information in PMC

References

    1. Makridakis S. The forthcoming Artificial Intelligence (AI) revolution: Its impact on society and firms. Futures. 2017;90:46–60. 10.1016/j.futures.2017.03.006 - DOI
    1. Zhang G, Eddy Patuwo B, Hu Y M. Forecasting with artificial neural networks:: The state of the art. International Journal of Forecasting. 1998;14(1):35–62. 10.1016/S0169-2070(97)00044-7 - DOI
    1. Hamzaçebi C, Akay D, Kutay F. Comparison of direct and iterative artificial neural network forecast approaches in multi-periodic time series forecasting. Expert Systems with Applications. 2009;36(2, Part 2):3839–3844. 10.1016/j.eswa.2008.02.042. - DOI
    1. Deng L. A tutorial survey of architectures, algorithms, and applications for deep learning—ERRATUM. APSIPA Transactions on Signal and Information Processing. 2014;3 10.1017/atsip.2013.9 - DOI
    1. Zhang L, Suganthan PN. A survey of randomized algorithms for training neural networks. Information Sciences. 2016;364–365(Supplement C):146–155. 10.1016/j.ins.2016.01.039. - DOI