Computer vision detects inflammatory arthritis in standardized smartphone photographs in an Indian patient cohort

Abstract

Introduction: Computer vision extracts meaning from pixelated images and holds promise in automating various clinical tasks. Convolutional neural networks (CNNs), a deep learning network used therein, have shown promise in analyzing X-ray images and joint photographs. We studied the performance of a CNN on standardized smartphone photographs in detecting inflammation in three hand joints and compared it to a rheumatologist's diagnosis.

Methods: We enrolled 100 consecutive patients with inflammatory arthritis with an onset period of less than 2 years, excluding those with deformities. Each patient was examined by a rheumatologist, and the presence of synovitis in each joint was recorded. Hand photographs were taken in a standardized manner, anonymized, and cropped to include joints of interest. A ResNet-101 backbone modified for two class outputs (inflamed or not) was used for training. We also tested a hue-augmented dataset. We reported accuracy, sensitivity, and specificity for three joints: wrist, index finger proximal interphalangeal (IFPIP), and middle finger proximal interphalangeal (MFPIP), taking the rheumatologist's opinion as the gold standard.

Results: The cohort consisted of 100 individuals, of which 22 of them were men, with a mean age of 49.7 (SD 12.9) years. The majority of the cohort (n = 68, 68%) had rheumatoid arthritis. The wrist (125/200, 62.5%), MFPIP (94/200, 47%), and IFPIP (83/200, 41.5%) were the three most commonly inflamed joints. The CNN achieved the highest accuracy, sensitivity, and specificity in detecting synovitis in the MFPIP (83, 77, and 88%, respectively), followed by the IFPIP (74, 74, and 75%, respectively) and the wrist (62, 90, and 21%, respectively).

Discussion: We have demonstrated that computer vision was able to detect inflammation in three joints of the hand with reasonable accuracy on standardized photographs despite a small dataset. Feature engineering was not required, and the CNN worked despite a diversity in clinical diagnosis. Larger datasets are likely to improve accuracy and help explain the basis of classification. These data suggest a potential use of computer vision in screening and follow-up of inflammatory arthritis.

Keywords: artificial intelligence; computer vision; digital health; inflammatory arthritis; screening.

Figure 1

Schema of photograph processing and outputs from the convolutional neural network.

Figure 2

Hue augmentation for skin tone: representative hue-augmented pairs with random hue variation between 6 and 34 degrees. (A) Index finger proximal interphalangeal joints, (B) middle finger interphalangeal joints, and (C) wrist joints. Note the henna on some images did not change the results.

Figure 3

Representative image showing the performance of Convolutional Neural Network (CNN) for three joints in contingency tables. Accuracy denotes the overall probability that a patient is correctly classified. MFPIP, middle finger proximal interphalangeal joint; IFPIP, index finger proximal interphalangeal joint; PPV, positive predictive value; NPV, negative predictive value.

Figure 4

(A) Representative training graph for naive (no hue augmentation) middle finger PIP. On this run, accuracy = 83%, sensitivity = 77%, and specificity = 88%. (B) Representative training graph for naive (with hue augmentation) MFPIP. On this run, accuracy = 81%, sensitivity = 65%, and specificity = 92%.