MR-Class: A Python Tool for Brain MR Image Classification Utilizing One-vs-All DCNNs to Deal with the Open-Set Recognition Problem

Abstract

Background: MR image classification in datasets collected from multiple sources is complicated by inconsistent and missing DICOM metadata. Therefore, we aimed to establish a method for the efficient automatic classification of MR brain sequences.

Methods: Deep convolutional neural networks (DCNN) were trained as one-vs-all classifiers to differentiate between six classes: T1 weighted (w), contrast-enhanced T1w, T2w, T2w-FLAIR, ADC, and SWI. Each classifier yields a probability, allowing threshold-based and relative probability assignment while excluding images with low probability (label: unknown, open-set recognition problem). Data from three high-grade glioma (HGG) cohorts was assessed; C1 (320 patients, 20,101 MRI images) was used for training, while C2 (197, 11,333) and C3 (256, 3522) were for testing. Two raters manually checked images through an interactive labeling tool. Finally, MR-Class' added value was evaluated via radiomics model performance for progression-free survival (PFS) prediction in C2, utilizing the concordance index (C-I).

Results: Approximately 10% of annotation errors were observed in each cohort between the DICOM series descriptions and the derived labels. MR-Class accuracy was 96.7% [95% Cl: 95.8, 97.3] for C2 and 94.4% [93.6, 96.1] for C3. A total of 620 images were misclassified; manual assessment of those frequently showed motion artifacts or alterations of anatomy by large tumors. Implementation of MR-Class increased the PFS model C-I by 14.6% on average, compared to a model trained without MR-Class.

Conclusions: We provide a DCNN-based method for the sequence classification of brain MR images and demonstrate its usability in two independent HGG datasets.

Keywords: artificial intelligence (AI); content-based image classification; convolutional neural networks (CNN); data curation and preparation; deep learning.

Figure 1

Sample images of the different MR images present in the three datasets C1–C3.

Figure 2

MR-Class training workflow. MR-Class comprises five one-vs-all DCNNs, one for each class, and the T1w-vs-T1wce binary DCNN. After MR image preprocessing, each DCNN was trained with an 80%/20% training/validation split, with class 1 representing the DCNNs’ target class and 0 for the rest. For the T1w-vs-T1wce DCNN, class 0 was assigned to T1w and 1 to T1wce. T2w-FL: T2w-FLAIR, T1wce: T1w contrast-enhanced.

Figure 3

MR-Class inference workflow. C2 and C3 were used for testing. After preprocessing, MR images are passed to the 5 one-vs-all DCNN classifiers. A classification probability threshold of 0.95 was used. If none of the classifiers labels an image, it is rendered as other. If more than one classifier labels a specific image, then the image is labeled by the classifier with the highest probability.

Figure 4

The one-vs-all ResNet-18 architecture. An alternating stack of convolutional activations and pooling layers. The skip connections (indicated by arrows) fit the unmodified input from the previous layer to the next, preserving the original image signal. FC (2) refers to a fully connected layer with two neurons as output, representing the sequence and the other possible sequences.

Figure 5

Distribution of the probabilities of correct and wrong labeled images for all three cohorts in the study when inferred to MR-Class. Based on the distributions of C1, a cutoff classification threshold probability of 0.95 was used. Histogram bin width = 0.01.

Figure 6

Confusion matrices of the six DCNNs for C2 (A) and C3 (B). The upper panels in (A,B) show the confusion matrices for datasets C2 and C3. The lower panels in (A,B) show MR-Class normalized confusion matrices for datasets C2 and C3, i.e., the percentages (%) of correct classification results per class. SE: sensitivity; SP: specificity. Class ‘Other’: when none of the DCNNs labels an image; n: number of scans per class, T2w-FL: T2w-FLAIR.

Figure 7

Examples of misclassified images. The first two images are examples of a misclassified MR, possibly due to blurry images (left) and alterations in expected anatomy (displaced ventricles, large tumor, right). The next three MR images show incorrect predictions due to different MR artifacts (shading, motion, aliasing). All of these images are falsely classified as “other”. The last image is a diffusion-weighted image (DWI), specifically a trace DWI, misclassified as T2w.

Figure 8

Box plots of the 1st–99th percentile C-Is attained by the MR-class and DICOM series description (SD) curated dataset models fitted by the respective signatures after three resampling approaches. MCCV: Monte Carlo cross-validation, BStrap: bootstrapping, CV: cross-validation.