An optimized machine learning model for identifying socio-economic, demographic and health-related variables associated with low vaccination levels that vary across ZIP codes in California

Abstract

There is an urgent need for an in-depth and systematic assessment of a wide range of predictive factors related to populations most at risk for delaying and refusing COVID-19 vaccination as cases of the disease surge across the United States. Many studies have assessed a limited number of general sociodemographic and health-related factors related to low vaccination rates. Machine learning methods were used to assess the association of 151 social and health-related risk factors derived from the American Community Survey 2019 and the Centers for Disease Control and Prevention (CDC) BRFSS with the response variables of vaccination rates and unvaccinated counts in 1,555 ZIP Codes in California. The performance of various analytical models was evaluated according to their ability to regress between predictive variables and vaccination levels. Machine learning modeling identified the Gradient Boosting Regressor (GBR) as the predictive model with a higher percentage of the explained variance than the variance identified through linear and generalized regression models. A set of 20 variables explained 72.90% of the variability of unvaccinated counts among ZIP Codes in California. ZIP Codes were shown to be a more meaningful geo-local unit of analysis than county-level assessments. Modeling vaccination rates was not as effective as modeling unvaccinated counts. The public health utility of this model provides for the analysis of state and local conditions related to COVID-19 vaccination use and future public health problems and pandemics.

Keywords: COVID-19; Machine learning; Sociodemographic determinants; Spatial assessment; Vaccination uptake.

Fig. 1

A and B: Box plots illustrate within-county variations on median household income and the Social Vulnerability Index (SVI) within the counties listed on the x-axis. Significant variations of many of these variables, especially within large counties, make it very challenging to understand how they may play a role in vaccination prevalence if measured only by counties, as is done in much of the literature and media.

Fig. 2A

Correlations between proportions of each feature group in ZIP codes in California and unvaccinated counts. As shown in the table above, these feature variables correlate significantly (p = 0) to the vaccination counts.

Fig. 2B

Correlations between proportions of each feature group in ZIP codes in California and vaccination rates. These feature variables correlate significantly to the vaccination rates, as shown in the table above.

Fig. 3

A and B: ML Method selection - Modeling Unvaccinated Counts and Vaccination Rates in CA. These tables list different machine learning methods we tried on our data set before selecting the best one. We use 5-fold cross-validation to assess the performance of each method through “explained_variance” criteria. 20% of the data is set aside for each fold, and the model is trained on the other 80%. Then the trained model predicts on the 20% and calculates variance explained between the prediction $\widehat{y}$ and the actual y values: $L(y-\widehat{y})=1-\frac{\mathit{Var}(y-\widehat{y})}{\mathit{Var}\left(y\right)}$. The higher the variance explained, the better the model.

Fig. 3

A and B: ML Method selection - Modeling Unvaccinated Counts and Vaccination Rates in CA. These tables list different machine learning methods we tried on our data set before selecting the best one. We use 5-fold cross-validation to assess the performance of each method through “explained_variance” criteria. 20% of the data is set aside for each fold, and the model is trained on the other 80%. Then the trained model predicts on the 20% and calculates variance explained between the prediction $\widehat{y}$ and the actual y values: $L(y-\widehat{y})=1-\frac{\mathit{Var}(y-\widehat{y})}{\mathit{Var}\left(y\right)}$. The higher the variance explained, the better the model.

Fig. 4

A and B: Forward Sequential Selection considering variance explained by each variable starting from largest until a maximum is reached - for Unvaccinated Counts and Vaccination Rates.

Fig. 5

A and B: Modeling Unvaccinated Counts and Vaccination Rates by ZIP codes in CA. The green lines represent the actual values in each ZIP code, and the dotted red predicted. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 6

A and B: Modeling Unvaccinated Counts and Vaccination Rates by ZIP codes in CA. These scatter plots illustrate the correlation between actual values (on the x-axis) and predicted.

Fig. 6

A and B: Modeling Unvaccinated Counts and Vaccination Rates by ZIP codes in CA. These scatter plots illustrate the correlation between actual values (on the x-axis) and predicted.

Fig. 7

A and B: Feature Importances in predicting Unvaccinated Counts and Vaccination Rates using GBR in CA ZIP codes. The importances give us a sense of each variable's contribution in bringing the prediction as close as possible to the actual values.

Fig. 7

A and B: Feature Importances in predicting Unvaccinated Counts and Vaccination Rates using GBR in CA ZIP codes. The importances give us a sense of each variable's contribution in bringing the prediction as close as possible to the actual values.