Performance evaluation of radiologists with artificial neural network for differential diagnosis of intra-axial cerebral tumors on MR images

Abstract

Background and purpose: Previous studies have suggested that use of an artificial neural network (ANN) system is beneficial for radiological diagnosis. Our purposes in this study were to construct an ANN for the differential diagnosis of intra-axial cerebral tumors on MR images and to evaluate the effect of ANN outputs on radiologists' diagnostic performance.

Materials and methods: We collected MR images of 126 patients with intra-axial cerebral tumors (58 high-grade gliomas, 37 low-grade gliomas, 19 metastatic tumors, and 12 malignant lymphomas). We constructed a single 3-layer feed-forward ANN with a Levenberg-Marquardt algorithm. The ANN was designed to differentiate among 4 categories of tumors (high-grade gliomas, low-grade gliomas, metastases, and malignant lymphomas) with use of 2 clinical parameters and 13 radiologic findings in MR images. Subjective ratings for the 13 radiologic findings were provided independently by 2 attending radiologists. All 126 cases were used for training and testing of the ANN based on a leave-one-out-by-case method. In the observer test, MR images were viewed by 9 radiologists, first without and then with ANN outputs. Each radiologist's performance was evaluated through a receiver operating characteristic (ROC) analysis on a continuous rating scale.

Results: The averaged area under the ROC curve for ANN alone was 0.949. The diagnostic performance of the 9 radiologists increased from 0.899 to 0.946 (P < .001) when they used ANN outputs.

Conclusions: The ANN can provide useful output as a second opinion to improve radiologists' diagnostic performance in the differential diagnosis of intra-axial cerebral tumors seen on MR imaging.

Fig 1.

Diagram of the basic structure of the ANN. Although only 10 input units and 8 hidden units are shown for illustration, the ANN consists of 15 input units and 9 hidden units.

Fig 2.

MR images of 2 actual cases. A, Case 1: MR images of a 44-year-old woman with a glioblastoma confirmed on pathologic examination (WHO grade IV). Left image: T2WI shows a heterogeneously hyperintense mass with central necrosis and surrounding signal intensity abnormality likely related to tumor extension and edema. Middle and right images: Precontrast and postcontrast T1WIs show hemorrhagic mass and peripheral enhancement with central necrosis, characteristic of glioblastoma. B, Case 2: MR images of a 62-year-old woman with proved metastatic brain tumor from lung cancer. Left image: T2WI shows a cystic frontoparietal mass with mixed-aged hemorrhage. Middle and right images: Precontrast and postcontrast T1WIs show a thin layer of peripheral enhancement. Surgery disclosed adenocarcinoma.

Fig 3.

ANN output obtained on the basis of 2 radiologists' ratings of MR features and clinical information for the 2 cases shown in Fig 2. Each graph shows the largest output values among the 4 categories corresponding to the correct diagnoses. A, Case 1: The likelihood of high-grade glioma is very high. ANN led to the correct diagnosis. B, Case 2: The likelihood of metastasis is approximately equivalent to high-grade glioma and malignant lymphoma. ANN might fail to lead to the correct diagnosis.

Fig 4.

Average AUC values and binormal ROC curves for observers with and without ANN output (averaged plot values for all readers). Those for ANN alone are also indicated. Note that observer performance improves significantly with ANN output.

Fig 5.

The number of correctly diagnosed cases for which observers' rankings changed because of ANN output. Positive values indicate that ANN was beneficial, whereas negative values indicate that ANN was detrimental. ANN output clearly improved the performance.