The importance of making testable predictions: A cautionary tale

Abstract

We found a startling correlation (Pearson ρ > 0.97) between a single event in daily sea surface temperatures each spring, and peak fish egg abundance measurements the following summer, in 7 years of approximately weekly fish egg abundance data collected at Scripps Pier in La Jolla California. Even more surprising was that this event-based result persisted despite the large and variable number of fish species involved (up to 46), and the large and variable time interval between trigger and response (up to ~3 months). To mitigate potential over-fitting, we made an out-of-sample prediction beyond the publication process for the peak summer egg abundance observed at Scripps Pier in 2020 (available on bioRxiv). During peer-review, the prediction failed, and while it would be tempting to explain this away as a result of the record-breaking toxic algal bloom that occurred during the spring (9x higher concentration of dinoflagellates than ever previously recorded), a re-examination of our methodology revealed a potential source of over-fitting that had not been evaluated for robustness. This cautionary tale highlights the importance of testable true out-of-sample predictions of future values that cannot (even accidentally) be used in model fitting, and that can therefore catch model assumptions that may otherwise escape notice. We believe that this example can benefit the current push towards ecology as a predictive science and support the notion that predictions should live and die in the public domain, along with the models that made them.

Fig 1. Is there a fine-time-scale relationship between temperature and eggs?

A) The total egg abundance in each collection (see Methods) shows substantial variability from year to year in both mean and peak levels. Each fish egg collection was made from the Scripps Institution of Oceanography (SIO) Pier. B) The daily averaged sea surface temperature (SST) in °C at the SIO Pier from data taken every 5–10 minutes from the SCCOOS monitoring station. C) Seasonal averaging reveals the strong negative correlation between the average winter (December–February) SST and the average spring and summer (March–August) egg abundance, identified by [15], with additional points for 2018 [16], and now 2019 and 2020. D) The seasonal correlation breaks down at the daily level; there is no similarly strong correlation between daily winter temperatures and daily egg abundances with time delays ranging from 0 to 180 days. E) The S-Map [21] test for nonlinearity shows that forecasts of egg abundance improve (correlation between predictions and observations) as the nonlinearity parameter (θ) is increased, indicating that egg abundance shows nonlinear behavior. F) Convergent cross-mapping (CCM, [22]), shows that when using the egg abundance time series to map onto the temperature time series, predictions improve as library size increases, indicating there is a dynamic causal effect of daily-averaged temperature on egg abundance.

Fig 2. An apparently robust temperature trigger for peak summer egg abundance that ultimately failed in true out-of-sample prediction.

A) We defined the spring temperature trigger (STT) as the largest temperature increase (denoted in red) detected within a monthly sliding window (gray area) as it moves in daily increments over the spring season (dashed lines; see Methods). B) The relationship between STT and peak summer eggs was robust to the width of the sliding window (widths that produce a ρ > 0.95 for the data up to and including 2019 are indicated in green). C) The peak correlation between STT and peak summer eggs (June–August) for 2013–2019 (black dots) and predicted value of 801 eggs for 2020 based on the linear regression (red dot), which differs dramatically from the eventually observed peak summer eggs (blue dot).

Fig 3. Time averaging of the 2013–2019 data obscured the original spring temperature trigger, suggesting the predictive information lay in the daily-resolution temperature data, not in the trend.

A) The relationship between spring temperature trigger (STT defined over 27 days) and maximum summer fish egg abundance declined as SST was increasingly smoothed across the x-axis (from daily to monthly averages). B) The daily (blue), weekly averaged (purple), and monthly averaged (orange) sea-surface temperature in 2017. Note how the magnitude of the STT (red bars) declines with averaging. Note that this figure does not include 2020 data.

Fig 4. The failure of our 2020 prediction and the structural sensitivity of the model.

A) 2020 fish eggs and temperature time series showing the timing of the red tide. The timing of the red tide was estimated based on the duration of anomalous dissolved oxygen levels measured by the Scripps Ocean Acidification Real-time (SOAR) Monitoring Program. B) Correlation between a triggering event that occurs any time before the annual peak eggs, and the annual peak eggs from 2013 to 2020 suggests possible overfitting (see Methods). C) Sensitivity of the original 2013–2019 STT to peak summer eggs relationship to variation in the spring end date and summer start date further suggests overfitting.