Advancing Air Pollution Exposure Models with Open-Vocabulary Object Detection and Semantic Segmentation of Street-View Images

Abstract

Mobile monitoring campaigns combined with land use regression (LUR) models effectively capture fine-scale spatial variations in urban air pollution. However, traditional predictor variables often fail to capture the nuances of the built environment and undocumented emission sources. To address this, we developed a framework integrating customizable object-level and segmentation-level visual features from street-view images into stepwise regression and random-forest-based LUR models. Using 5.7 million mobile air pollution measurements (2019-2020) and 0.37 million street-view images (2008-2024), we mapped nitrogen dioxide (NO₂), black carbon (BC), and ultrafine particles (UFP) across 46,664 road segments in Amsterdam, The Netherlands. Incorporating street-view images improved model performance, increasing R² by 0.01-0.05 and reducing mean absolute errors by 0.7-10.3%. Sensitivity analyses indicated that key street-view-derived visual features remained stable across years and seasons. Using images from nearby years expanded training instances, thereby enhancing alignment with mobile measurements at fine granularity. Our open-vocabulary object detection module identified influential but previously unrecognized object predictors, such as chimneys, traffic lights, and shops. Combined with segmentation-derived features (e.g., walls, roads, grass), street-view images contributed 8-18% feature importance to model predictions. These findings highlight the potential of visual data in enhancing hyperlocal air pollution mapping and exposure assessment.

Keywords: air pollution; deep learning; exposure assessment; land use regression (LUR); mobile sensing; street-view image; vision-language model (VLM); vision-transformer models (ViT).

1

Examples of street-view images in different years and seasons. TikTok AI was used to convert street-view images into an animation style for demonstration purposes.

2

Architecture of the visual land use regression model (VLUR). Object and segmentation-level information is extracted from 0.37 million street-view images. These visual features are integrated with classic land use, traffic, and population data to train land use regression models, supervised by 50 m road-segment aggregated mobile air pollution measurements. TikTok AI was used to convert street-view images into an animation style for demonstration purposes.

3

Examples of open-vocabulary object and semantic segmentation results. Objects were detected by OWL-ViT (Vision Transformer for Open-World Localization). Semantic segmentation was performed using Mask2Former. TikTok AI was used to convert street-view images into an animation style for demonstration purposes.

4

Density plot of mobile training data, model predictions, and long-term validation data for NO₂ at 33 Palmes locations. RF: random-forest-based LUR model. SLR: stepwise linear regression-based LUR model. SpecificY, MostnearY, and Season-weighted represent temporal strategies based on different sets of street-view images. Palmes 33 is a subset of long-term fixed-site measurements of NO₂ in Amsterdam. The LUR models were trained by using mobile NO₂ measurements and aimed to estimate long-term NO₂ distributions represented by Palmes data.