Development of a Machine Learning Model to Estimate US Firearm Homicides in Near Real Time

Abstract

Importance: Firearm homicides are a major public health concern; lack of timely mortality data presents considerable challenges to effective response. Near real-time data sources offer potential for more timely estimation of firearm homicides.

Objective: To estimate near real-time burden of weekly and annual firearm homicides in the US.

Design, setting, and participants: In this prognostic study, anonymous, longitudinal time series data were obtained from multiple data sources, including Google and YouTube search trends related to firearms (2014-2019), emergency department visits for firearm injuries (National Syndromic Surveillance Program, 2014-2019), emergency medical service activations for firearm-related injuries (biospatial, 2014-2019), and National Domestic Violence Hotline contacts flagged with the keyword firearm (2016-2019). Data analysis was performed from September 2021 to September 2022.

Main outcomes and measures: Weekly estimates of US firearm homicides were calculated using a 2-phase pipeline, first fitting optimal machine learning models for each data stream and then combining the best individual models into a stacked ensemble model. Model accuracy was assessed by comparing predictions of firearm homicides in 2019 to actual firearm homicides identified by National Vital Statistics System death certificates. Results were also compared with a SARIMA (seasonal autoregressive integrated moving average) model, a common method to forecast injury mortality.

Results: Both individual and ensemble models yielded highly accurate estimates of firearm homicides. Individual models' mean error for weekly estimates of firearm homicides (root mean square error) varied from 24.95 for emergency department visits to 31.29 for SARIMA forecasting. Ensemble models combining data sources had lower weekly mean error and higher annual accuracy than individual data sources: the all-source ensemble model had a weekly root mean square error of 24.46 deaths and full-year accuracy of 99.74%, predicting the total number of firearm homicides in 2019 within 38 deaths for the entire year (compared with 95.48% accuracy and 652 deaths for the SARIMA model). The model decreased the time lag of reporting weekly firearm homicides from 7 to 8 months to approximately 6 weeks.

Conclusions and relevance: In this prognostic study of diverse secondary data on machine learning, ensemble modeling produced accurate near real-time estimates of weekly and annual firearm homicides and substantially decreased data source time lags. Ensemble model forecasts can accelerate public health practitioners' and policy makers' ability to respond to unanticipated shifts in firearm homicides.

Figure 1.. Actual and Predicted Weekly Number of Firearm Homicides in 2019 According to Individual Data Sources

Individual prediction models for 2019 firearm homicide deaths were built using 5 individual data sources: search engine and video-sharing platform search trends related to firearms (2014-2019), emergency department visits for firearm injuries (National Syndromic Surveillance Program, 2014-2019), emergency medical service activations for firearm-related injuries (biospatial, 2014-2019), and domestic violence hotline contacts flagged with the keyword firearm (2016-2019).

Figure 2.. Actual and Predicted Weekly Number of Firearm Homicides in 2019 According to the Ensemble Model

Final least absolute shrinkage and selection operator (LASSO) ensemble model combines all data sources (emergency department visits, emergency medical service activations, search engine and video-sharing platform keyword search, and domestic violence hotline calls) trained on 2016-2017 data, validated on 2018 data, and tested on 2019 data to predict weekly firearm homicides using a LASSO ensemble model compared with the actual number of weekly firearm homicides and seasonal autoregressive integrated moving average (SARIMA)–forecasted number of weekly firearm homicides in 2019.