External Validation of the Machine Learning-Based Thermographic Indices for Rheumatoid Arthritis: A Prospective Longitudinal Study

Abstract

External validation is crucial in developing reliable machine learning models. This study aimed to validate three novel indices-Thermographic Joint Inflammation Score (ThermoJIS), Thermographic Disease Activity Index (ThermoDAI), and Thermographic Disease Activity Index-C-reactive protein (ThermoDAI-CRP)-based on hand thermography and machine learning to assess joint inflammation and disease activity in rheumatoid arthritis (RA) patients. A 12-week prospective observational study was conducted with 77 RA patients recruited from rheumatology departments of three hospitals. During routine care visits, indices were obtained at baseline and week 12 visits using a pre-trained machine learning model. The performance of these indices was assessed cross-sectionally and longitudinally using correlation coefficients, the area under the receiver operating curve (AUROC), sensitivity, specificity, and positive and negative predictive values. ThermoDAI and ThermoDAI-CRP correlated with CDAI, SDAI, and DAS28-CRP cross-sectionally (ρ = 0.81; ρ = 0.83; ρ = 0.78) and longitudinally (ρ = 0.55; ρ = 0.61; ρ = 0.60), all p < 0.001. ThermoDAI and ThermoDAI-CRP also outperformed Patient Global Assessment (PGA) and PGA + C-reactive protein (CRP) in detecting changes in 28-swollen joint counts (SJC28). ThermoJIS had an AUROC of 0.67 (95% CI, 0.58 to 0.76) for detecting patients with swollen joints and effectively identified patients transitioning from SJC28 > 1 at baseline visit to SJC28 ≤ 1 at week 12 visit. These results support the effectiveness of ThermoJIS in assessing joint inflammation, as well as ThermoDAI and ThermoDAI-CRP in evaluating disease activity in RA patients.

Keywords: artificial intelligence; external validation; machine learning; rheumatoid arthritis; thermography.

Figure 1

Association between the Thermographic Joint Inflammation Score (ThermoJIS) and the 28-swollen joint counts (SJC28). (a) Area under the receiver operating curve (AUROC) of the ThermoJIS for detecting swollen joints considered as SJC28 > 1 (AUROC, 0.67; 95% CI 0.58–0.76, p < 0.001). (b) Spearman’s correlation between ThermoJIS and SJC28 (ρ = 0.30, p < 0.001). Aggregated data from baseline visit and week 12 visit (n = 150).

Figure 2

Indirect analysis of the Thermographic Joint Inflammation Score (ThermoJIS) performance in detecting synovitis that is not evident through swollen joint count by comparing the ThermoJIS values of patients with 28-swollen joint counts (SJC28) = 0 and Patient Global Assessment (PGA) >5 (n = 11) versus patients with SJC28 = 0 and PGA ≤ 5 (n = 62). The Mann–Whitney U test was used to assess statistical significance.

Figure 3

The area under the receiver operating curve (AUROC) of the Thermographic Joint Inflammation Score (ThermoJIS) for detecting transitions from 28-swollen joint counts (SJC28) > 1 at the baseline visit to SJC28 ≤ 1 at the week 12 visit (AUROC, 0.71; 95% CI 0.52–0.90, p = 0.053) (n = 31).

Figure 4

Correlation between the Thermographic Disease Activity Index (ThermoDAI) and Thermographic Disease Activity Index-C-reactive protein (ThermoDAI-CRP) with common indices used in clinical practice. (a) Spearman’s correlation between the ThermoDAI and Clinical Disease Activity Index (CDAI) (ρ = 0.81; p < 0.001); (b) Spearman’s correlation between the ThermoDAI-CRP and Simplified Disease Activity Index (SDAI) (ρ = 0.83; p < 0.001); (c) Spearman’s correlation between the ThermoDAI-CRP and 28-joint Disease Activity Score C-reactive protein (DAS28-CRP) (ρ = 0.78; p < 0.001). Aggregated data from baseline visit and week 12 visit (n = 150).

Figure 5

Correlation of the change in Thermographic Disease Activity Index (ThermoDAI) and Thermographic Disease Activity Index-C-reactive protein (ThermoDAI-CRP) between the baseline visit and the week 12 visit with the Clinical Disease Activity Index (CDAI), Simplified Disease Activity Index (SDAI), and 28-joint Disease Activity Score C-reactive protein (DAS28-CRP) (n = 73). (a) Spearman’s correlation between the ΔThermoDAI and ΔCDAI (ρ = 0.55, p < 0.001); (b) Spearman’s correlation between the ΔThermoDAI-CRP and ΔSDAI (ρ = 0.61, p < 0.001); (c) Spearman’s correlation between ΔThermoDAI-CRP and ΔDAS28-CRP (ρ = 0.60, p < 0.001).

Figure 6

The area under the receiver operating curve (AUROC) of the Thermographic Disease Activity Index (ThermoDAI), Patient Global Assessment (PGA), and C-reactive protein (CRP) for detecting transitions from 28-swollen joint counts (SJC28) > 1 at the baseline visit to SJC28 ≤ 1 at the week 12 visit (n = 31). (a) AUROC of the ΔThermoDAI [0.73 (95% CI, 0.55 to 0.91, p = 0.028)] and ΔPGA [0.61 (95% CI, 0.41 to 0.81, p = 0.273)]; (b) AUROC of the ΔThermoDAI-CRP [0.66 (95% CI, 0.46 to 0.86, p = 0.128)] and ΔPGA + CRP [0.55 (95% CI, 0.33 to 0.76, p = 0.678)].

Figure 7

Detection of patients who met the American College of Rheumatology 20% response (ACR20), the American College of Rheumatology 50% response (ACR50), and European Alliance of Associations for Rheumatology (EULAR) treatment response criteria using the Thermographic Disease Activity Index (ThermoDAI) and Thermographic Disease Activity Index-C-reactive protein (ThermoDAI-CRP) (n = 32). (a) Area under the receiver operating curve (AUROC) of the ThermoDAI for detecting patients who met ACR20 [0.73 (95% CI, 0.55 to 0.91, p = 0.031)]; (b) area under the receiver operating curve (AUROC) of the ThermoDAI for detecting patients who met ACR50 [0.79 (95% CI, 0.47 to 1.00, p = 0.062)]; (c) area under the receiver operating curve (AUROC) of the ThermoDAI for detecting patients who met EULAR treatment response [0.74 (95% CI, 0.56 to 0.91, p = 0.023)]; (d) area under the receiver operating curve (AUROC) of the ThermoDAI-CRP for detecting patients who met ACR20 [0.80 (95% CI, 0.64 to 0.95, p = 0.006)]; (e) area under the receiver operating curve (AUROC) of the ThermoDAI-CRP for detecting patients who met ACR50 [0.91 (95% CI, 0.80 to 1.00, p = 0.005)]; (f) area under the receiver operating curve (AUROC) of the ThermoDAI-CRP for detecting patients who met EULAR treatment response [0.77 (95% CI, 0.60 to 0.94, p = 0.010)].