Towards Robust and Accurate Detection of Abnormalities in Musculoskeletal Radiographs with a Multi-Network Model

Abstract

This study proposes a novel multi-network architecture consisting of a multi-scale convolution neural network (MSCNN) with fully connected graph convolution network (GCN), named MSCNN-GCN, for the detection of musculoskeletal abnormalities via musculoskeletal radiographs. To obtain both detailed and contextual information for a better description of the characteristics of the radiographs, the designed MSCNN contains three subnetwork sequences (three different scales). It maintains high resolution in each sub-network, while fusing features with different resolutions. A GCN structure was employed to demonstrate global structure information of the images. Furthermore, both the outputs of MSCNN and GCN were fused through the concat of the two feature vectors from them, thus making the novel framework more discriminative. The effectiveness of this model was verified by comparing the performance of radiologists and three popular CNN models (DenseNet169, CapsNet, and MSCNN) with three evaluation metrics (Accuracy, F1 score, and Kappa score) using the MURA dataset (a large dataset of bone X-rays). Experimental results showed that the proposed framework not only reached the highest accuracy, but also demonstrated top scores on both F1 metric and kappa metric. This indicates that the proposed model achieves high accuracy and strong robustness in musculoskeletal radiographs, which presents strong potential for a feasible scheme with intelligent medical cases.

Keywords: CNN; GCN; abnormality detection; fusion; multi-network; musculoskeletal radiographs.

Figure 1

Preprocessing of images (a) Original image, (b) Padded image, and (c) Resized image.

Figure 2

Proposed network architecture.

Figure 3

The proposed architecture consists of four blocks. Block I shows the preprocessing procedure for the original image. Block II shows a multi-scale convolution neural network (MSCNN) block, consisting of three branches, which obtains more detailed information from the preprocessed image. Block III shows a graph convolution network (GCN) block to extract global structure information from the downsampled image. Block IV shows a fusion block to demonstrate the classification results through the concat of the two feature vectors from MSCNN and GCN, respectively.

Figure 4

F1 scores of radiologists, DenseNet169 and the proposed model.

Figure 5

Examples of heatmaps for the shoulder’s radiographys predicted by the proposed framework. (a) is an abnormal sample where the green arrow on the right pointed to the salient features corresponding to model’s correct prediction. (b) is a normal sample where the red arrow on the right pointed to the salient features corresponding to the model’s wrong prediction.

Figure 6

Kappa score of DenseNet169, CapsNet and MSCNN-GCN.

Figure 7

Comparison between MSCNN and MSCNN-GCN.