Object-based verification of a prototype Warn-on-Forecast system

Abstract

An object-based verification methodology for the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e) has been developed and applied to 32 cases between December 2015 and June 2017. NEWS-e forecast objects of composite reflectivity and 30-minute rotation tracks of updraft helicity are matched to corresponding objects in Multi-Radar Multi-Sensor data on space and time scales typical of a National Weather Service warning. Object matching allows contingency table-based verification statistics to be used to establish baseline performance metrics for NEWS-e thunderstorm and mesocyclone forecasts. NEWS-e critical Success Index (CSI) scores of reflectivity (updraft helicity) forecasts decrease from approximately 0.7 (0.4) to 0.4 (0.2) over 3 hours of forecast time. CSI scores decrease through the forecast period, indicating that errors have not saturated and skill is retained at 3 hours of forecast time. Lower verification scores for rotation track forecasts are primarily a result of a high frequency bias. Comparison of different system configurations used in 2016 and 2017 show an increase in skill for 2017 reflectivity forecasts, attributable mainly to improvements in the forecast initial condition. A small decrease in skill in 2017 rotation track forecasts is likely a result of sample differences between 2016 and 2017. Although large case-to-case variation is present, evidence is found that NEWS-e forecast skill improves with increasing object size and intensity, as well as in mesoscale environments in which an enhanced or higher risk of severe thunderstorms was forecast.

Fig. A1.

Time series of (a, d, g, j) probability of detection, (b, e, h, k) false alarm ratio, and (c, f, i, l) critical success index for NEWS-e 2–5 km rotation track forecasts. The intensity threshold used to identify forecast and observed rotation track objects is varied between the (a–c) 99.95th percentile from each year’s climatology (same as Fig. 7), (d–f) the 99.95th percentile from the 2017 climatology only, (g–i) the 99.95th percentile from the 2016 climatology only, and (j–l) the 99.9th percentile from each year’s climatology. Individual ensemble member scores are plotted in thin orange (blue) lines with thick orange (blue) lines representing the ensemble mean for 2017 (2016) NEWS-e forecasts.

Fig. A2.

As in Fig. A1, except for the composite reflectivity objects and the maximum distance threshold for object matching is varied from (a–c) 20 km, (d–f) 30 km, (g–i) 40 km (same as Fig. 5), and (j–l) 60 km.

Fig. 1.

Example NEWS-e domain from 16 May 2017. The map shown corresponds to the HRRRE domain, with the nested NEWS-e domain shaded green. WSR-88D sites whose data are assimilated into NEWS-e are marked by blue dots with 150-km range rings drawn in gray.

Fig. 2.

Schematic of NEWS-e system configuration for 2017.

Fig. 3.

Climatologies of forecast and verification datasets for (blue) 2016 cases and (orange) 2017 cases. Scatter plots show the the 99.1st through 99.98th percentile values for (a) composite reflectivity (dBZ), (b) 2–5 km updraft helicity (m² s⁻²) or azimuthal wind shear (AWS; s⁻¹), and (c) 0–2 km updraft helicity or azimuthal wind shear. Thresholds used for object identification are marked by horizontal and vertical lines.

Fig. 4.

Schematic depicting the object matching and verification process. Initial thresholded fields from the (a) forecast from a single ensemble member and (d) observations are subjected to size and continuity quality control thresholds prior to (b, e) object identification. (c) Forecast objects are matched to verification objects according to prescribed spatiotemporal displacement thresholds with matched pairs being considered, hits, unmatched forecast objects false alarms, and unmatched verification objects misses. This classification of objects allows the (f) standard contingency table metrics probability of detection (POD), false alarm ratio (FAR), frequency bias (BIAS), and critical success index (CSI) to be calculated to quantify forecast skill.

Fig. 5.

Time series of object-based (a) POD, (b) BIAS, (c) FAR, and (d) CSI for composite reflectivity forecasts from (blue) 2016 and (orange) 2017. Individual ensemble members are plotted with thin lines and the ensemble mean in bold. Ensemble means are calculated as the mean of verification metrics from each ensemble member. The first and last 20 minutes of the forecast are masked so that only forecast times where objects could be matched in time as well as space are considered. The total number of objects from each year is annotated.

Fig. 6.

Paintball plots of composite reflectivity objects (a) 30, (b) 60, (c) 90, and (d) 120 minutes into forecasts initialized at 0100 UTC on 28 May 2017. Colored shading indicates NEWS-e member forecast objects, with different colors assigned to each ensemble member, and dark gray shading observed objects. Regions shaded light gray are less than 5 km or greater than 150 km from the nearest WSR-88D and not considered in verification. Ensemble mean POD, FAR, BIAS, and CSI scores are provided in the upper right of each panel.

Fig. 7.

Performance diagrams (Roebber 2009) for 60-minute composite reflectivity forecasts from each case during (a) 2016 and (b) 2017. Small circles indicate scores of individual ensemble members and large circles represent the ensemble mean from each case. Cases are numbered according to the legend provided below each plot and color coded according to maximum SPC risk in the NEWS-e domain and storm mode. The total number of objects identified for each case is provided following each date in the legend.

Fig. 8.

Time series of the object-based ensemble mean (a) POD, (b) FAR, (c) BIAS, and (d) CSI for composite reflectivity forecasts aggregated for each forecast initialization hour between 2000 and 0200 UTC. Scores from 2017 (2016) forecasts are plotted in orange (blue) and every other forecast is plotted using lighter, dashed lines in order to improve readability. As in Fig. 5, the first and last 20 minutes of each forecast are masked. The total number of objects for each forecast initialization hour is annotated in panel a.

Fig. 9.

As in Fig. 7 except for 60-minute composite reflectivity forecasts from (orange) 6 cases in 2017 and (blue) the same 6 cases re-run using Thompson microphysics. The 2016 reflectivity climatology was used to identify objects in the forecasts using Thompson microphysics.

Fig. 10.

As in Fig. 5 except for 2–5 km updraft helicity forecasts.

Fig. 11.

As in Fig. 5 except for 0–2 km updraft helicity forecasts.

Fig. 12.

As in Fig. 7 except for 60-minute 2–5 km updraft helicity forecasts.

Fig. 13.

As in Fig. 9 except for 2–5 km updraft helicity forecasts. Note that the 2017 2–5 km updraft helicity climatology is used to define rotation track objects in both the Thompson and NSSL 2-Moment experiments.

Fig. 14.

As in Fig. 6 except for (a, c) composite reflectivity and (b, d) rotation track objects 60-minutes into forecasts initialized at 2300 UTC on (a, b) 17 May 2017 and (c, d) 23 May 2017. POD, FAR, BIAS, and CSI scores for each forecast are provided in the lower left of each panel. Note that some forecast rotation track objects in (d) are matched to observed objects at different times, resulting in a FAR less than 1.0 despite no observed objects being present at the forecast time plotted.

Fig. 15.

As in Fig. 8 except for 2–5 km updraft helicity forecasts.

Fig. 16.

Scatterplots of the parameter space of object area and maximum intensity for 60-minute NEWS-e forecasts of (a, b) composite reflectivity (dBZ) and (c–f) 2–5 km updraft helicity (m² s⁻²) during (a, c) 2016, (b, d) 2017, and 2017 cases classified as (e) supercell or (f) mixed/linear mode. Matched objects are plotted in orange and false alarm objects in blue with the total number of objects in each category listed in the lower right. Kernel density estimate contours of the 95th, 97.5th, 99th, and 99.9th percentile values of each distribution are overlain to illustrate differences between matched and false alarm distributions. Every third reflectivity object is plotted to improve clarity.

Fig. 17.

Scatterplots of the east-west and north-south centroid displacements (km) of matched objects for 30-minute NEWS-e forecasts of (a) composite reflectivity (dBZ) and (b) 2–5 km updraft helicity (m² s⁻²). Objects from 2016 (2017) are plotted in blue (orange) and the total number of objects for each year is listed in the lower left. Kernel density estimate contours are overlain as in Fig. 16 and every third reflectivity object is plotted to improve clarity.