Deep Learning-Based Classification of Inflammatory Arthritis by Identification of Joint Shape Patterns-How Neural Networks Can Tell Us Where to "Deep Dive" Clinically

Abstract

Objective: We investigated whether a neural network based on the shape of joints can differentiate between rheumatoid arthritis (RA), psoriatic arthritis (PsA), and healthy controls (HC), which class patients with undifferentiated arthritis (UA) are assigned to, and whether this neural network is able to identify disease-specific regions in joints.

Methods: We trained a novel neural network on 3D articular bone shapes of hand joints of RA and PsA patients as well as HC. Bone shapes were created from high-resolution peripheral-computed-tomography (HR-pQCT) data of the second metacarpal bone head. Heat maps of critical spots were generated using GradCAM. After training, we fed shape patterns of UA into the neural network to classify them into RA, PsA, or HC.

Results: Hand bone shapes from 932 HR-pQCT scans of 617 patients were available. The network could differentiate the classes with an area-under-receiver-operator-curve of 82% for HC, 75% for RA, and 68% for PsA. Heat maps identified anatomical regions such as bare area or ligament attachments prone to erosions and bony spurs. When feeding UA data into the neural network, 86% were classified as "RA," 11% as "PsA," and 3% as "HC" based on the joint shape.

Conclusion: We investigated neural networks to differentiate the shape of joints of RA, PsA, and HC and extracted disease-specific characteristics as heat maps on 3D joint shapes that can be utilized in clinical routine examination using ultrasound. Finally, unspecific diseases such as UA could be grouped using the trained network based on joint shape.

Keywords: arthritis; artificial intelligence; bone; deep learning; joint.

Figure 1

Deep learning model for the classification of joint shapes. (A) Proposed deep learning model termed Convolutional Supervised Auto-Encoder (CSAE) model consists of five stages each for the encoding and decoding branch. Stages closer to the linear classification layer have an increasing number of channels. (B) A single encoder consists of two 3 × 3 × 3 convolution followed by a Leaky ReLU activation function and three-dimensional dropout. Maximum pooling with a factor of two is used for down-sampling. The decoding branch is used to generate features in the bottleneck that are discriminative of the image.

Figure 2

Training and validation of the neural network, visualization of the regions influencing the networks decisions and application of network to undetermined arthritis cases. (A) Training and validation of the neural network using the three-dimesional articular bone shape (assessed by high-resolution peripheral computed tomography) of defined conditions such as rheumatoid arthritis (RA), psoriatic arthritis (PsA), and healthy controls (HC). (B) Left: Location of the measurement region (red) of high-resolution peripheral computed tomography (CT) scans as data source; center: Three different segmentation bone masks with the respective heat maps from healthy controls as well as RA patients and PsA patients; each patient segmentation mask is shown in the palmar view (top row) and in the dorsal view (bottom row); right: Preparation of anatomical specimen to correlate to heat maps detected by the neural network with anatomical regions. (C) Application of the neural network using undifferentiated arthritis patients to classify them into either RA, PsA, and HC according to the neural network defined in (A). (D) Ultrasound image, dorsal scan of a healthy metacarpophalangeal joint. Here, we illustrate the transfer of our findings to arthrosonography. The outline of the capsule is marked yellow. The articular entheseal regions are marked in red. Based on the findings of the neural network, alterations of these articular entheseal regions (red) are specific for PsA and should be paid attention in clinical routine, especially in patients who are suspected for PsA. Patients provided written consent to the depiction of their images.