Deterministic skill of ENSO predictions from the North American Multimodel Ensemble

Abstract

Hindcasts and real-time predictions of the east-central tropical Pacific sea surface temperature (SST) from the North American Multimodel Ensemble (NMME) system are verified for 1982-2015. Skill is examined using two deterministic verification measures: mean squared error skill score (MSESS) and anomaly correlation. Verification of eight individual models shows somewhat differing skills among them, with some models consistently producing more successful predictions than others. The skill levels of MME predictions are approximately the same as the two best performing individual models, and sometimes exceed both of them. A decomposition of the MSESS indicates the presence of calibration errors in some of the models. In particular, the amplitudes of some model predictions are too high when predictability is limited by the northern spring ENSO predictability barrier and/or when the interannual variability of the SST is near its seasonal minimum. The skill of the NMME system is compared to that of the MME from the IRI/CPC ENSO prediction plume, both for a comparable hindcast period and also for a set of real-time predictions spanning 2002-2011. Comparisons are made both between the MME predictions of each model group, and between the average of the skills of the respective individual models in each group. Acknowledging a hindcast versus real-time inconcsistency in the 2002-2012 skill comparison, the skill of the NMME is slightly higher than that of the prediction plume models in all cases. This result reflects well on the NMME system, with its large total ensemble size and opportunity for possible complementary contributions to skill.

Fig. 1

Root-mean squared error (RMSE) of the first-lead (0.5 month) forecasts, 1982–2015

Fig. 2

Scatter plots of the first-lead errors (ensemble mean forecast anomaly minus observation anomaly) of a CFSv2 versus CCSM4, b CFSv2-c2 versus CCSM4-2c, and c CFSv2-c2 versus NASA, 1982–2015. Blue circles shows forecasts for 1982–1998, and red circles for 1999–2016

Fig. 3

Forecasts (blue) of the CMC1 model and observations (gray) throughout the study period. A forecast trajectory is shown for start times spanning all months

Fig. 4

Forecasts (blue) of the CFSv2 model and observations (gray) throughout the study period. A forecast trajectory is shown for start times spanning all months. The forecast anomalies are with respect to climatologies spanning two base periods having differing forecast biases (see the text)

Fig. 5

Forecasts (blue) of the MME and observations (gray) throughout the study period. A forecast trajectory is shown for start times spanning all months

Fig. 6

Time series of mean squared errors of each individual NMME model and of the MME, averaged over the third to sixth lead (2.5 to 5.5 month leads). Note that the top row has larger scale, to accommodate occasional very large squared errors in the 1980s

Fig. 7

MME forecasts, by target month, as a function of lead time (hence, forecasts were made increasingly earlier than the target month toward the upper part of panel). The observations are shown at the bottom of the panel

Fig. 8

MME error, by target month, as a function of lead time (hence, errors are for forecasts made increasingly earlier than the target month toward the upper part of panel). The observations are shown at the bottom of the panel

Fig. 9

Mean squared error skill score (MSESS) of ensemble mean anomaly forecasts as a function of target month and lead in months. Black dots indicate MSESS that is statistically significantly greater than a climatological forecast at 5% significance level, using a sign test

Fig. 10

Anomaly correlation of ensemble mean forecast and observed anomalies as a function of target month and lead time in months. Black dots indicate correlation values that are statistically significantly greater than zero at 5% significance level, using a t test

Fig. 11

Squared amplitude bias as a component of the decomposition of the MSESS. Here the CFSv2 and CCSM4 models are analyzed with a single climatology instead of the dual climatologies as in the other analyses. See text for details

Fig. 12

Mean-squared error skill score (MSESS; left) and anomaly correlation (right), as a function of lead and averaged over (top row) all 12 target months, (middle row) the better forecast target months of November through March, and (bottom row) the more challenging forecast target months of May through September. The Fisher-Z transform is used in averaging the correlations