Dorsal Finger Fold Recognition by Convolutional Neural Networks for the Detection and Monitoring of Joint Swelling in Patients with Rheumatoid Arthritis

Abstract

Digital biomarkers such as wearables are of increasing interest in monitoring rheumatic diseases, but they usually lack disease specificity. In this study, we apply convolutional neural networks (CNN) to real-world hand photographs in order to automatically detect, extract, and analyse dorsal finger fold lines as a correlate of proximal interphalangeal (PIP) joint swelling in patients with rheumatoid arthritis (RA). Hand photographs of RA patients were taken by a smartphone camera in a standardized manner. Overall, 190 PIP joints were categorized as either swollen or not swollen based on clinical judgement and ultrasound. Images were automatically preprocessed by cropping PIP joints and extracting dorsal finger folds. Subsequently, metrical analysis of dorsal finger folds was performed, and a CNN was trained to classify the dorsal finger lines into swollen versus non-swollen joints. Representative horizontal finger folds were also quantified in a subset of patients before and after resolution of PIP swelling and in patients with disease flares. In swollen joints, the number of automatically extracted deep skinfold imprints was significantly reduced compared to non-swollen joints (1.3, SD 0.8 vs. 3.3, SD 0.49, p < 0.01). The joint diameter/deep skinfold length ratio was significantly higher in swollen (4.1, SD 1.4) versus non-swollen joints (2.1, SD 0.6, p < 0.01). The CNN model successfully differentiated swollen from non-swollen joints based on finger fold patterns with a validation accuracy of 0.84, a sensitivity of 88%, and a specificity of 75%. A heatmap of the original images obtained by an extraction algorithm confirmed finger folds as the region of interest for correct classification. After significant response to disease-modifying antirheumatic drug ± corticosteroid therapy, longitudinal metrical analysis of eight representative deep finger folds showed a decrease in the mean diameter/finger fold length (finger fold index, FFI) from 3.03 (SD 0.68) to 2.08 (SD 0.57). Conversely, the FFI increased in patients with disease flares. In conclusion, automated preprocessing and the application of CNN algorithms in combination with longitudinal metrical analysis of dorsal finger fold patterns extracted from real-world hand photos might serve as a digital biomarker in RA.

Keywords: Digital biomarker; Disease activity; Neural networks; Rheumatoid arthritis; Swelling.

Fig. 1

Automated finger fold recognition to monitor RA. Hand photographs are taken by a smartphone in a standardized manner (a). Hands, and subsequently PIP joints, are automatically recognized and extracted. Finger fold lines are isolated from the images, measured and related to the joint diameter (b, c). A convolutional deep neural network was used to train a model for classification of extracted finger fold patterns into swollen versus non swollen joints (d) A heatmap of the cropped PIP joints confirmed finger folds as the region of interest for correct classification (e).

Fig. 2

Longitudinal metrical analysis of horizontal finger folds. a Representative images of finger fold patterns before and after treatment or arthritis flares. (b) The joint diameter/finger fold length ratio reduced in a panel of patients after successful treatment with disease modifying antirheumatic drugs and/or cortisone.