XComposition: multimodal deep learning model to measure body composition using chest radiographs and clinical data

Abstract

Background: Body composition metrics such as visceral fat volume, subcutaneous fat volume, and skeletal muscle volume are important predictors of cardiovascular disease, diabetes, and cancer prognosis.

Purpose: We explore the use of deep learning to estimate body composition metrics from chest radiographs and a small set of easily obtainable clinical variables.

Materials and methods: A retrospective cohort of patients with concurrent noncontrast abdominal CT's and frontal chest radiographs within 3 months of each other was selected. A multitask, multimodal, deep learning model using chest radiographs and clinical variables (age, sex at birth, height and weight extracted from electronic medical records) was trained to estimate the body composition metrics. Reference standard was body composition, including subcutaneous fat volume, measured on CT.

Results: Our final cohort consisted of 1118 patients (582 female and 538 male subjects) from 30 health systems across the United States with imaging performed from 2010 to 2024. The mean age at imaging was 67 years (SD: 17), mean height was 1.67 meters (SD: 0.2), and mean weight was 78 kg (SD: 20). Average values for visceral fat, subcutaneous fat, and skeletal muscle indices were 59.39 cm²/m² (SD: 39.26), 88.13 cm²/m² (SD: 58.52), and 44.81 cm²/m² (SD: 15.49). The best-performing model achieved a Pearson correlation of 0.85 (95% CI: 0.81-0.88) for subcutaneous fat volume, 0.76 (0.65-0.80) for visceral fat volume, and 0.58 (0.49-0.67) for skeletal muscle volume with the multimodal model outperforming unimodal models (P = .0001 for subcutaneous fat volume). Mean absolute errors of the best performing models for subcutaneous and visceral fat volumes were 1054 cm³/m² and 667 cm³/m², respectively.

Conclusion: We introduced a multimodal deep learning model leveraging chest radiographs to estimate body composition. Our model can facilitate large-scale studies by estimating body composition using a chest radiograph and commonly available clinical variables.

Keywords: body composition; chest radiographs; deep learning; multimodal data fusion.

Figure 1.

Cohort flowchart. We initially request 3000 cases. Of these, 1118 cases met all our inclusion criteria and were included in our study. Individuals with missing clinical data were excluded if their age or sex was missing or if both height and weight were missing. Inaccurate clinical data includes height and weight measurements outside acceptable ranges (height 100–280 cm, weight: 30–600 kg). ROI = region of interest.

Figure 2.

A diagram showing model architecture, input features, and the various fusion strategies used in our study. We used a resnet18 model for our convolutional neural network.

Figure 3.

Scatter plots showing multimodal model predictions versus ground truth values (ie, values calculated from abdominal CT) for 3 of the top performing body composition metrics.

Figure 4.

Bland-Altman plots showing the distribution of prediction errors with respect to ground truth (ie, values calculated from abdominal CT) for subcutaneous and visceral fat index, vertebral bone volume, and skeletal muscle volume.

Figure 5.

Saliency maps for feature importance in prediction of the top performing body composition metrics. For the first 3, occlusion-sensitivity saliency maps are depicted, whereas for the vertebral body volume, the saliency map is based on the integrated gradients method. The white box highlights the region of increased importance for vertebral body volume calculation that coincides with the region containing the lower thoracic and upper lumbar vertebral bodies. The images are aggregates of individual chest radiographs and their respective saliency maps in the hold-out test cohort.

Figure 6.

Three samples of the radiographs in which the model had poor performance. (A) Parts of the chest radiograph that are important to the model are missing from the image. eSFV vs SFV: 1525 cm³ vs 6320 cm³, eVFV vs VFV: 660 cm³ vs 4202 cm³, eSMI vs SMI: 150 cm³/m² vs 1003 cm³/m² B. This chest radiograph is dark and does not encompass some important regions for prediction. eSFV vs SFV: 6234 cm³ vs 1741 cm³, eVFV vs VFV: 3153 cm³ vs 5234 cm³, eSMI vs SMI: 929 cm³/m² vs 1286 cm³/m² C. The patient is rotated, and the lungs are not properly inflated. eSFV vs SFV: 4003 cm³ vs 5940 cm³, eVFV vs VFV: 2046 cm³ vs 3151 cm³, eSMI vs SMI: 2501 cm³/m² vs 1050 cm³/m². (e)SMI = (estimated) skeletal muscle index, (e)SFV = (estimated) subcutaneous fat volume, (e)VFV = (estimated) visceral fat volume.