Ecosystem Model Skill Assessment. Yes We Can!

Abstract

Need to assess the skill of ecosystem models: Accelerated changes to global ecosystems call for holistic and integrated analyses of past, present and future states under various pressures to adequately understand current and projected future system states. Ecosystem models can inform management of human activities in a complex and changing environment, but are these models reliable? Ensuring that models are reliable for addressing management questions requires evaluating their skill in representing real-world processes and dynamics. Skill has been evaluated for just a limited set of some biophysical models. A range of skill assessment methods have been reviewed but skill assessment of full marine ecosystem models has not yet been attempted.

Northeast us atlantis marine ecosystem model: We assessed the skill of the Northeast U.S. (NEUS) Atlantis marine ecosystem model by comparing 10-year model forecasts with observed data. Model forecast performance was compared to that obtained from a 40-year hindcast. Multiple metrics (average absolute error, root mean squared error, modeling efficiency, and Spearman rank correlation), and a suite of time-series (species biomass, fisheries landings, and ecosystem indicators) were used to adequately measure model skill. Overall, the NEUS model performed above average and thus better than expected for the key species that had been the focus of the model tuning. Model forecast skill was comparable to the hindcast skill, showing that model performance does not degenerate in a 10-year forecast mode, an important characteristic for an end-to-end ecosystem model to be useful for strategic management purposes.

Skill assessment is both possible and advisable: We identify best-practice approaches for end-to-end ecosystem model skill assessment that would improve both operational use of other ecosystem models and future model development. We show that it is possible to not only assess the skill of a complicated marine ecosystem model, but that it is necessary do so to instill confidence in model results and encourage their use for strategic management. Our methods are applicable to any type of predictive model, and should be considered for use in fields outside ecology (e.g. economics, climate change, and risk assessment).

Fig 1. Conceptual Fig comparing the skill metrics (RMSE, AAE, AE, MEF and correlation) performance using simulated observed (open circles) and modeled data (lines).

Fig 2. Map showing the Northeast (NE) USA and the Atlantis ecosystem model regions (blue) and boundary boxes (yellow).

Fig 3. Biomass (normalized) time series.

Survey data time series (dark blue) with 95% confidence interval bands (dotted lines) versus Atlantis model biomass predictions (light blue). Gray area indicates the forecast time-period from 2004–2013. We label (and color code) the data-categories Biomass (blue), Landings (orange), and Indicators (green), consistently throughout the results

Fig 4. Landings (normalized) time series.

Recorded (actual) landings (dark orange) versus Atlantis model catch predictions (light orange). Gray area indicates the forecast time-period from 2004–2013.

Fig 5. Time series of ecosystem indicators calculated using both Atlantis NEUS model data (dark green) and observed data (light green).

Gray area indicates the forecast time-period from 2004–2013.

Fig 6. Forecast skill metrics for biomass data.

Pairwise comparison of forecast and hindcast skill metric performance for 5 skill metrics: MEF, AE, AAE, RMSE and S(Spearman rank) Correlation for 22 species in the NEUS Atlantis ecosystem model. Y-axis limited to show values between -2 and 2.

Fig 7. Principal component (PCA) biplot of loadings and score of the 1^st and 2^nd principal component from a PCA analysis of the different skill metrics: MEF, AE, AAE, RMSE, Pearson Correlation (P), Kendall Rank Correlation (K) and Spearman Rank Correlation(S), calculated for the hindcast (_h) and forecast (_f) for the time-series for biomass, landings and ecological indicators.

Loadings for the skill metrics are shown in red while scores for the species, catches and indicators are in black.

Fig 8. Forecast skill metrics for fisheries landings data.

Pairwise comparison of forecast (_f) and hindcast (_h) skill metric performance for 5 skill metrics: MEF, AE, AAE, RMSE and S(pearman rank) Correlation for 21 species of commercial fish and shellfish in the NEUS Atlantis ecosystem model. Y-axis limited to show values between -2 and 2.

Fig 9. Forecast skill metrics for ecosystem indicators emulating emergent ecosystem properties.

Pairwise comparison of forecast (_f) and hindcast (_h) skill metric performance for 5 skill metrics: MEF, AE, AAE, RMSE and S.(pearman rank) Correlation for 16 ecological indicators based on biomass and landings data. Y-axis limited to show values between -2 and 2.

Fig 10. Comparison of model skill performance for hindcast (1974–2004) and the model forecast (2004–2013) for species biomass or landings and ecosystem indicators.

Pink: <40%, Yellow: 40–60%, Green: >60%