Blind testing of shoreline evolution models

Abstract

Beaches around the world continuously adjust to daily and seasonal changes in wave and tide conditions, which are themselves changing over longer time-scales. Different approaches to predict multi-year shoreline evolution have been implemented; however, robust and reliable predictions of shoreline evolution are still problematic even in short-term scenarios (shorter than decadal). Here we show results of a modelling competition, where 19 numerical models (a mix of established shoreline models and machine learning techniques) were tested using data collected for Tairua beach, New Zealand with 18 years of daily averaged alongshore shoreline position and beach rotation (orientation) data obtained from a camera system. In general, traditional shoreline models and machine learning techniques were able to reproduce shoreline changes during the calibration period (1999-2014) for normal conditions but some of the model struggled to predict extreme and fast oscillations. During the forecast period (unseen data, 2014-2017), both approaches showed a decrease in models' capability to predict the shoreline position. This was more evident for some of the machine learning algorithms. A model ensemble performed better than individual models and enables assessment of uncertainties in model architecture. Research-coordinated approaches (e.g., modelling competitions) can fuel advances in predictive capabilities and provide a forum for the discussion about the advantages/disadvantages of available models.

Figure 1

Study site and input conditions. (a) Location of Tairua, New Zealand North Island (b) Detail of Tairua Beach. Pressure sensor (S4_N) location used for SWAN model validation (c) Alongshore shoreline position at Tairua beach. Red represents shoreline advance and blue shoreline retreat over time. (d) Daily alongshore-averaged position (e) Shoreline rotation (orientation) with positive values representing southward accretion (anti-clockwise rotation) and negative values representing northward accretion (clockwise rotation). (f) Significant wave height g) Peak period h) Wave direction. Grey shading show the data that was hidden from modelers (Shorecast period, 2014–2017).

Figure 2

Three years of the entire calibration period (1999–2014). Examples of model outputs (see legend) compared to three years (2001–2004) of calibration data (black): (a) Hybrid models; (b) Machine Learning models; (c) Shoreline rotation models. See Methods section and Supporting Information for model details.

Figure 3

Shorecast predictions (2014–2017, blind test). Model outputs (see legends) compared to observations (black) (a) Hybrid models (b) Machine Learning models (c) HM and ML ensemble (d) Multi-model ensemble (e) Rotation models (f) Hybrid models ensemble for beach rotation. Dark shadows in the ensembles figures represent one standard deviation of the models prediction. Light shadows represent maxima/minima envelope of the models predictions. See Methods section and Supporting Information for model details.

Figure 4

Models performance. Quantile-quantile plots of model behavior. Top 3 panels: Calibration period; middle 3 panels: Shorecast. Model prediction vs measured shoreline position for (a) and (d) HM; (b) and (e) ML; (c) and (f) model prediction vs measured shoreline rotation. Dashed grey line represent the average shoreline position during the calibration and the Shorecast period, respectively. R², RMSE, skill and λ for shoreline prediction for; (g–j) averaged shoreline position; (k–n) shoreline rotation. See supporting material for more information about the metrics and individual models.