Neural networks for automatic scoring of arthritis disease activity on ultrasound images

Abstract

Background: The development of standardised methods for ultrasound (US) scanning and evaluation of synovitis activity by the OMERACT-EULAR Synovitis Scoring (OESS) system is a major step forward in the use of US in the diagnosis and monitoring of patients with inflammatory arthritis. The variation in interpretation of disease activity on US images can affect diagnosis, treatment and outcomes in clinical trials. We, therefore, set out to investigate if we could utilise neural network architecture for the interpretation of disease activity on Doppler US images, using the OESS scoring system.

Methods: Two state-of-the-art neural networks were used to extract information from 1342 Doppler US images from patients with rheumatoid arthritis (RA). One neural network divided images as either healthy (Doppler OESS score 0 or 1) or diseased (Doppler OESS score 2 or 3). The other to score images across all four of the OESS systems Doppler US scores (0-3). The neural networks were hereafter tested on a new set of RA Doppler US images (n=176). Agreement between rheumatologist's scores and network scores was measured with the kappa statistic.

Results: For the neural network assessing healthy/diseased score, the highest accuracies compared with an expert rheumatologist were 86.4% and 86.9% with a sensitivity of 0.864 and 0.875 and specificity of 0.864 and 0.864, respectively. The other neural network developed to four class Doppler OESS scoring achieved an average per class accuracy of 75.0% and a quadratically weighted kappa score of 0.84.

Conclusion: This study is the first to show that neural network technology can be used in the scoring of disease activity on Doppler US images according to the OESS system.

Keywords: OMERACT-EULAR Synovitis Scoring system; artificial intelligence; automatic scoring; convolutional neural networks; deep learning; disease activity; rheumatoid arthritis; ultrasound.

Figure 1

Illustration of convolutional neural network (CNN) automatically scoring Outcome Measures in Rheumatology (OMERACT)-EULAR disease activity on an ultrasound (US) image. Neural network scoring rheumatoid arthritis (RA) disease activity, according to the OMERACT-EULAR Synovitis Scoring (OESS) system, on an US Colour Doppler image of the wrist. An US expert has given the OESS score of 3. The US image is passed through the network where specialised neurons extract increasingly complex information. Each neuron constructs an information map that represents how much of each information at different levels of complexity is present in the image. At the end of the network, classifier neurons map the information to probability scores for each class (OESS system scores). The class with the highest probability represents the RA disease activity score given by the network.