Development of a deep learning model for automated detection of calcium pyrophosphate deposition in hand radiographs

Abstract

Background: Calcium pyrophosphate deposition (CPPD) disease is a leading cause of arthritis, which can mimic or strongly interfere with other rheumatic diseases such as gout, osteoarthritis (OA) or rheumatoid arthritis (RA). In the recently established ACR/EULAR CPPD classification criteria, calcification and OA features of the wrist and hand joints are substantial features.

Objectives: To develop and test a deep-learning algorithm for automatically and reliably detecting CPPD features in hand radiographs, focusing on calcification of the triangular fibrocartilage complex (TFCC) and metacarpophalangeal (MCP)-2 and -3 joints, in separate or combined models.

Methods: Two radiologists independently labeled a dataset of 926 hand radiographs, yielding 319 CPPD positive and 607 CPPD negative cases across the three sites of interest after adjudicating discrepant cases. CPPD presence was then predicted using a convolutional neural network. We tested seven CPPD models, each with a different combination of sites out of TFCC, MCP-2 and MCP-3. The model performance was assessed using the area under the receiver operating characteristic (AUROC) and area under the precision-recall (AUPR) curves, with heatmaps (Grad-CAM) aiding in case discrimination.

Results: All models trialed gave good class separation, with the combined TFCC, MCP-2 and MCP-3 model showing the most robust performance with a mean AUROC of 0.86, mean AUPR of 0.77, sensitivity of 0.77, specificity of 0.80, and precision of 0.67. The TFCC-alone model had a slightly lower mean AUROC of 0.85 with a mean AUPR of 0.73. The MCP-2-alone and MCP-3-alone models exhibited mean AUROCs of 0.78-0.87, but lower mean AUPRs of 0.29-0.47. Heatmap analysis revealed activation in the regions of interest for positive cases (true and false positives), but unexpected highlights were encountered possibly due to correlated features in different hand regions.

Conclusion: A combined deep-learning model detecting CPPD at the TFCC and MCP-2/3 joints in hand radiographs provides the highest diagnostic performance. The algorithm could be used to screen larger OA or RA databases or electronic medical records for CPPD cases. Future work includes dataset expansion and validation with external datasets.

Keywords: CPPD; automated; chondrocalcinosis; detection; image recoginiton; machine learning; radiograph (X-ray).

Figure 1

Preprocessing steps before training the model. (A) Hand splitting. (B) Thresholding to generate a mask of the hand region (original image on left, mask on the right). Other features were sometimes retained, as in the case of the circled ‘G’ (center), thus smaller objects needed to be removed (right). (C) Cropping reduced the region of interest to part of the hand containing the TFCC, MCP-2, and MCP-3, and contrast enhancement was performed on this reduced region. (D) Further trimming to two smaller regions: on the left, the TFCC region; on the right, the MCP-2 and MCP-3 joints.

Figure 2

ROC curves for each of the seven different potential CPPD models, using an identical training and testing dataset.

Figure 3

ROC curves for the model predicting whether any site (TFCC, MCP-2, MCP-3), TFCC alone, or MCP2 or MCP3 alone are positive for CPPD after further trimming of the image (test set, according to Figure 1D). On the right, precision to recall curves are shown. Different colors correspond to five-fold cross-validation.

Figure 4

Confusion matrix corresponding to a threshold of 0.7 on one of the test folds of the combined model, giving a sensitivity (recall) of 0.77, specificity of 0.80, precision of 0.67, and F1 score of 0.72. Increasing the threshold increases the specificity and precision but decreases the sensitivity (recall), thus the threshold needs to be selected based on the clinical requirements.

Figure 5

Interpretability results using Grad-CAM on the last convolutional layer for classifying any site (out of TFCC, MCP-2, MCP-3) as positive. In the first row, we see examples of images classified correctly as positive; in the second row, images classified incorrectly as positive; in the third row, images classified correctly as negative; in the fourth row, images classified incorrectly as negative.TP, True positive; FP, false positive; TN, true negative; FN, false negative.