QSMRim-Net: Imbalance-aware learning for identification of chronic active multiple sclerosis lesions on quantitative susceptibility maps

Abstract

Background and purpose: Chronic active multiple sclerosis (MS) lesions are characterized by a paramagnetic rim at the edge of the lesion and are associated with increased disability in patients. Quantitative susceptibility mapping (QSM) is an MRI technique that is sensitive to chronic active lesions, termed rim + lesions on the QSM. We present QSMRim-Net, a data imbalance-aware deep neural network that fuses lesion-level radiomic and convolutional image features for automated identification of rim + lesions on QSM.

Methods: QSM and T2-weighted-Fluid-Attenuated Inversion Recovery (T2-FLAIR) MRI of the brain were collected at 3 T for 172 MS patients. Rim + lesions were manually annotated by two human experts, followed by consensus from a third expert, for a total of 177 rim + and 3986 rim negative (rim-) lesions. Our automated rim + detection algorithm, QSMRim-Net, consists of a two-branch feature extraction network and a synthetic minority oversampling network to classify rim + lesions. The first network branch is for image feature extraction from the QSM and T2-FLAIR, and the second network branch is a fully connected network for QSM lesion-level radiomic feature extraction. The oversampling network is designed to increase classification performance with imbalanced data.

Results: On a lesion-level, in a five-fold cross validation framework, the proposed QSMRim-Net detected rim + lesions with a partial area under the receiver operating characteristic curve (pROC AUC) of 0.760, where clinically relevant false positive rates of less than 0.1 were considered. The method attained an area under the precision recall curve (PR AUC) of 0.704. QSMRim-Net out-performed other state-of-the-art methods applied to the QSM on both pROC AUC and PR AUC. On a subject-level, comparing the predicted rim + lesion count and the human expert annotated count, QSMRim-Net achieved the lowest mean square error of 0.98 and the highest correlation of 0.89 (95% CI: 0.86, 0.92).

Conclusion: This study develops a novel automated deep neural network for rim + MS lesion identification using T2-FLAIR and QSM images.

Keywords: Chronic active lesions; Convolutional neural network; Multiple sclerosis; Quantitative susceptibility mapping; Radiomic features.

Fig. 1

Example of MS lesions on an axial slice of the QSM (left) and corresponding axial slice of the T2-FLAIR (right). The digit 1 marked with red indicates a rim + lesion and the digit 2 marked with green indicates a rim- lesion. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 2

Schematic of the rim + and rim- lesion annotation process. First, we use LST (Schmidt, 2012) to obtain an initial lesion segmentation mask. Second, a human expert performs manual correction and confluent lesion separation, followed by mask edits based on QSM. Third, rim + lesions are manually annotated by two human experts, followed by consensus from a third expert.

Fig. 3

Distribution of the number of paramagnetic rim lesions (rim + lesions) per subject in the Weill Cornell dataset. The plot is colored by the groups used for the stratified five-fold cross validation.

Fig. 4

Schematic of the proposed QSMRim-Net for paramagnetic rim lesion identification. (Top Row) The deep residual network takes in both QSM and FLAIR images to extract convolutional features. (Bottom Row) The QSM image and the lesion mask are used to extract radiomic features, followed by feature extraction of an MLP. A tensor concatenation operation is performed to fuse convolutional and radiomic features, and a DeepSMOTE layer is used to perform synthetic minority feature over-sampling during the training phase.

Fig. 5

Schematic of the DeepSMOTE network layer. $N$ is the number of samples in a training mini-batch, and $n$ is the number of rim + samples in the mini-batch. The input features go through an MLP for feature transformation, followed by selecting rim + samples from the mini-batch. The transformed rim + features are used to generate a Euclidean distance-based similarity followed by latent feature interpolation. The original features and the oversampled features are concatenated, resulting in a total of $N+2n$ samples in the output of DeepSMOTE.

Fig. 6

The partial receiver operating characteristic (pROC) curve and precision-recall (PR) curves for the proposed (QSMRim-Net) and comparator methods. AUC denotes the area under the curve. We use clinically relevant false positive rates of less than 0.1 to compute the pROC AUC, in order to account for the rare nature of rim + lesions. Our QSMRim-Net algorithm outperformed all other algorithms on pROC AUC (FPR < 0.1) and PR AUC.

Fig. 7

The predicted count of rim + lesions from QSMRim-Net versus the expert human count ($\rho =0.89\left(95\%CI:0.86,0.92\right),MSE=0.98$). Points in the plot have been jittered for better visualization. The linear regression line for the predicted count versus the gold standard count with 95% CI is also shown (solid blue) along with the identity line (dashed blue). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 8

Visual examples of a true positive, a false positive, a false negative and a true negative produced by QSMRim-Net. The QSM is shown on the left and the T2-FLAIR on the right. The lesion of interest indicated with a red arrow. (A) A rim + lesion that is correctly identified. (B) A rim- lesion with a vein forming a rim-like shape that was falsely identified as rim + by QSMRim-Net. (C) A rim + lesion with that was missed by QSMRim-Net. (D) A rim- lesion that is correctly identified. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)