Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks

Abstract

We introduce a data-driven forecasting method for high-dimensional chaotic systems using long short-term memory (LSTM) recurrent neural networks. The proposed LSTM neural networks perform inference of high-dimensional dynamical systems in their reduced order space and are shown to be an effective set of nonlinear approximators of their attractor. We demonstrate the forecasting performance of the LSTM and compare it with Gaussian processes (GPs) in time series obtained from the Lorenz 96 system, the Kuramoto-Sivashinsky equation and a prototype climate model. The LSTM networks outperform the GPs in short-term forecasting accuracy in all applications considered. A hybrid architecture, extending the LSTM with a mean stochastic model (MSM-LSTM), is proposed to ensure convergence to the invariant measure. This novel hybrid method is fully data-driven and extends the forecasting capabilities of LSTM networks.

Keywords: Gaussian processes; Lorenz 96; T21 barotropic climate model; data-driven forecasting; long short-term memory.

Figure 1.

(a) A recurrent neural network cell, where D denotes a delay. The hidden cell state h_t depends on the input i_t and its previous value h_t−1. The output o_t depends on the hidden state. The weight matrices are parameters of the cell. (b) A recurrent neural network unfolded in time (unfolding the delay). The same weights are used at each time step to compute the output o_t that depends on the current input i_t and short-term history (recursively) encoded in h_t−1.

Figure 2.

Iterative prediction using the trained LSTM model. A short-term history of the system, i.e. $z_1^{\mathrm{true}},\dots ,z_d^{\mathrm{true}}$, is assumed to be known. Initial LSTM states are h₀,C₀. The trained LSTM is used predict the derivative ${\dot{z}}_d^{\mathrm{pred}}={\mathcal{F}}^w(z_{d:1}^{\mathrm{true}},{\mathbf{\text{h}}}_0,C_0)$. The state prediction $z_{d+1}^{\mathrm{pred}}$ is obtained by integrating this derivative. This value is used for the next prediction in an iterative fashion. After d time-steps only predicted values are fed in the input. In stateless LSTM, h and C are initialized to zero before every prediction. (Online version in colour.)

Figure 3.

Lorenz 96 contour plots for different forcing regimes F. Chaoticity rises with bigger values of F.

Figure 4.

Energy spectrum E_k and cumulative energy with respect to the number of most energetic modes used for different forcing regimes of Lorenz 96 system. As the forcing increases, more chaoticity is introduced to the system. (Online version in colour.)

Figure 5.

(a–c) Short-term RMSE evolution of the most energetic mode for forcing regimes F=4,8,16, respectively, of the Lorenz 96 system. (d–f) Long-term RMSE evolution. (g–i) Evolution of the ACC coefficient (in all plots average over 1000 initial conditions is reported). (Online version in colour.)

Figure 6.

RMSE prediction error evolution of four energetic modes for the Lorenz 96 system with forcing F=8. (a) Most energetic mode k=8, (b) low-energy mode k=9, (c) low-energy mode k=10 and (d) low-energy mode k=11 (in all plots average over 1000 initial conditions reported). (Online version in colour.)

Figure 7.

(a) Ratio of the ensemble members evaluated using the LSTM model over time for different Lorenz 96 forcing regimes in the hybrid LSTM–MSM method and (b) the same for GPR in the hybrid GPR–MSM method (average over 500 initial conditions). (Online version in colour.)

Figure 8.

(a) Contour plots of the solution u(x,t) of the Kuramoto–Sivashinsky system for different values of ν in steady state. Chaoticity rises with smaller values of ν. (b) Cumulative energy as a function of the number of the PCA modes for different values of ν. (Online version in colour.)

Figure 9.

(a,b) RMSE evolution of the most energetic mode of the K-S equation with 1/ν=10 and 1/ν=16. (c), (d) ACC evolution of the most energetic mode of the K-S equation with 1/ν=10 and 1/ν=16 (in all plots, average value over 1000 initial conditions is reported). (Online version in colour.)

Figure 10.

(a) Mean of the Barotropic model at statistical steady state. (b) Variance of the Barotropic model at statistical steady state. (c) Percentage of energy explained with respect to the modelled modes. (Online version in colour.)

Figure 11.

RMSE evolution of the four most energetic EOFs for the Barotropic climate model, average over 500 initial conditions reported. (a) Most energetic EOF, (b) second most energetic EOF, (c) third most energetic EOF and (d) fourth most energetic EOF. (Online version in colour.)