Toward equitable environmental exposure modeling through convergence of data, open, and citizen sciences: an example of air pollution exposure modeling amidst increasing wildfire smoke

Abstract

Exposure modeling is critical in environmental epidemiology and human health but may face challenges (e.g., skewed data, unequal error, context-insensitive validation, and computational demands). Modeling decisions reflect the intended use of the models and the values that modelers prioritize. We aimed to provide a conceptual framework and machine learning (ML) modeling protocols that address these issues. With 500m-gridded hourly PM_2.5 and O₃ levels in Illinois before, during, and after the 2023 Canadian wildfire season as a motivating example, we conducted modeling experiments to evaluate modeling methods, guided by three domains we propose based on theories of science: 1) Data Diversity, leveraging open and citizen science data to enhance inclusivity, parsimony, and representativeness; 2) Equitable Accuracy, ensuring fairly distributed uncertainties across subpopulations; and 3) Sustainable Modeling, balancing accuracy with reducing computational demands to promote accessibility for under-resourced researchers. We found that ML with publicly available data can achieve high accuracy. Depending on methods, performance may vary substantially, even with identical input data. Large but skewed data may reduce performance. Misuse of cross-validation protocols can underestimate prediction error; although we observed R²s of ∼98 %, the modeled estimates varied significantly, indicating the need for careful model validation. By using new modeling protocols including representativeness-considered training and validation data and a new loss function, we achieved high agreement between estimates and ground-based measurements (e.g., R² = ∼90 % for PM_2.5; ∼80 % for O₃), equally distributed errors across sociodemographic strata and urban-rural divides, and reduction in computation time-from several weeks or months to a few days.

Keywords: Exposure assessment; Exposure modeling; Machine learning; Preferential sampling; Uncertainty; Validation.

Figure 1.. Locations of the air quality monitors/sensors used in this study (A) and categorizations of Zip Code Tabulation Area (ZCTA) based on the locations of the monitors/sensors and sociodemographic characteristics and urban areas in Illinois (B, C).

Note. In panel A, 29 EPA monitors for PM_2.5, 88 PurpleAir Sensor (PAS) for PM_2.5, and 36 EPA monitors for O₃ are presented. In panels B and C, each color represents ZCTAs belonging to the same propensity score stratum. ZCTAs were grouped using stratification of the propensity score for having at least one PM_2.5 monitor/sensor (B) or the propensity score for having a O₃ monitor (C). There are 9 strata for PM_2.5 and 5 strata for O₃

Figure 2.. Our novel modeling protocol.

Note. As environmental monitors and sensors are not randomly distributed, they may correlate with the characteristics of target locations (i.e., (sub-)study population) such as sociodemographic factors and urbanicity. This non-randomness issue could be seen as preferential sampling, over/underrepresentation, and/or data skewness. If this is not considered in model construction and validation, exposure estimates may be differentially biased across the characteristics, and model validation may not be able to detect this bias. Also, if the goal of modeling is to estimate environmental exposures in target locations including unmonitored areas, models should be constructed to be transportable, and a validation protocol should be designed to ensure that errors evaluated using test sets accurately represent errors for target locations. Furthermore, as environmental exposures and their drivers may be highly spatially and temporally autocorrelated, if test sets are autocorrelated with train sets, errors could be highly underestimated, leading to overconfidence, particularly when the test sets were not designed to align with modeling goals. Accordingly, naïve random data split protocols (e.g., K-fold cross-validation) and arbitrary leave-a few monitors-out validation may not be optimal for model training and testing. Therefore, we designed our modeling protocol to increase prediction power and the transportability of models, to reduce both overall, spatial, and temporal errors and potential inequality in these errors across the characteristics, and to accurately evaluate the errors. Our PS stratification resulted in spatial blocks as shown in Figure 1 and Figure S1, which can be viewed as a type of spatial cross-validation. We highlight that whether PS stratification can be a type of spatial cross-validation depends on whether the variables used for PS stratification are spatially clustered.

Figure 3.. Hourly PM_2.5 estimates on May 10, 2024, 20:00 CST and June 28, 2023, 20:00 CST in the Greater Chicago Region.

Note. The black solid line represents the City of Chicago. The black dashed line represents Cook County. The right panel shows elevated levels of PM_2.5 due to Canadian wildfires. Both figures show that PM_2.5 levels were spatially heterogeneous. For example, PM_2.5 levels were high close to/on major expressways and roads with high traffic volumes.

Figure 4.. Results of PM_2.5 modeling experiments.

Note. We evaluated overall accuracy, spatial accuracy, and temporal accuracy. For spatial accuracy, we calculated an average for each monitor/sensor, referred to as the spatial mean. The spatial mean of the estimates was compared to the spatial mean of the measurements. For temporal accuracy, the difference between the estimates and their spatial mean was compared to the difference between the measurements and their spatial mean. RMSE=root mean squared error. Coefficient indicates the slope factor of the linear regression of measurements against estimates. R-sq (R²) indicates the coefficient of determination, which equals squared Pearson correlation.

Figure 5.. Results of PM_2.5 modeling experiments for ‘US Extended 1’, ‘Spatially equal sample & Spatial loss function & Spatiotemporal calibration’ regarding equality in errors.

Note 1. We evaluated overall accuracy, spatial accuracy, and temporal accuracy. For spatial accuracy, we calculated an average for each monitor/sensor, referred to as the spatial mean. The spatial mean of the estimates was compared to the spatial mean of the measurements. For temporal accuracy, the difference between the estimates and their spatial mean was compared to the difference between the measurements and their spatial mean. RMSE=root mean squared error. Coefficient indicates the slope factor of the linear regression of measurements against estimates. R-sq (R²) indicates the coefficient of determination, which equals squared Pearson correlation. Note 2. Error measures in the propensity score stratum-specific test sets were calculated.