Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Jan;57(1):155-163.
doi: 10.1002/uog.23530.

Ultrasound image analysis using deep neural networks for discriminating between benign and malignant ovarian tumors: comparison with expert subjective assessment

F Christiansen  1 , E L Epstein  1 , E Smedberg  2 , M Åkerlund  3 , K Smith  4 , E Epstein  2
Affiliations

Ultrasound image analysis using deep neural networks for discriminating between benign and malignant ovarian tumors: comparison with expert subjective assessment

F Christiansen et al. Ultrasound Obstet Gynecol. 2021 Jan.

Abstract
in English, Spanish, Chinese

Objectives: To develop and test the performance of computerized ultrasound image analysis using deep neural networks (DNNs) in discriminating between benign and malignant ovarian tumors and to compare its diagnostic accuracy with that of subjective assessment (SA) by an ultrasound expert.

Methods: We included 3077 (grayscale, n = 1927; power Doppler, n = 1150) ultrasound images from 758 women with ovarian tumors, who were classified prospectively by expert ultrasound examiners according to IOTA (International Ovarian Tumor Analysis) terms and definitions. Histological outcome from surgery (n = 634) or long-term (≥ 3 years) follow-up (n = 124) served as the gold standard. The dataset was split into a training set (n = 508; 314 benign and 194 malignant), a validation set (n = 100; 60 benign and 40 malignant) and a test set (n = 150; 75 benign and 75 malignant). We used transfer learning on three pre-trained DNNs: VGG16, ResNet50 and MobileNet. Each model was trained, and the outputs calibrated, using temperature scaling. An ensemble of the three models was then used to estimate the probability of malignancy based on all images from a given case. The DNN ensemble classified the tumors as benign or malignant (Ovry-Dx1 model); or as benign, inconclusive or malignant (Ovry-Dx2 model). The diagnostic performance of the DNN models, in terms of sensitivity and specificity, was compared to that of SA for classifying ovarian tumors in the test set.

Results: At a sensitivity of 96.0%, Ovry-Dx1 had a specificity similar to that of SA (86.7% vs 88.0%; P = 1.0). Ovry-Dx2 had a sensitivity of 97.1% and a specificity of 93.7%, when designating 12.7% of the lesions as inconclusive. By complimenting Ovry-Dx2 with SA in inconclusive cases, the overall sensitivity (96.0%) and specificity (89.3%) were not significantly different from using SA in all cases (P = 1.0).

Conclusion: Ultrasound image analysis using DNNs can predict ovarian malignancy with a diagnostic accuracy comparable to that of human expert examiners, indicating that these models may have a role in the triage of women with an ovarian tumor. © 2020 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Análisis de imágenes ecográficas utilizando redes neurales profundas para distinguir entre tumores ováricos benignos y malignos: comparación con la evaluación subjetiva de expertos OBJETIVOS: Desarrollar y probar el desempeño del análisis de imágenes ecográficas computarizadas utilizando redes neurales profundas (RNP) para distinguir entre tumores ováricos benignos y malignos y comparar su precisión en el diagnóstico con la de la evaluación subjetiva (ES) por especialistas expertos en ecografía. MÉTODOS: Se incluyeron 3077 (escala de grises, n=1927; power Doppler, n=1150) imágenes de ultrasonido de 758 mujeres con tumores ováricos, que fueron clasificadas prospectivamente por examinadores especialistas en ecografía, de acuerdo con los términos y definiciones de la IOTA (Análisis Internacional de Tumores Ováricos). El resultado histológico de la cirugía (n=634) o el seguimiento a largo plazo (≥3 años) (n=124) sirvieron como el estándar de referencia. El conjunto de datos se dividió en un subconjunto de formación (n=508; 314 benignos y 194 malignos), un subconjunto de validación (n=100; 60 benignos y 40 malignos) y un subconjunto de pruebas (n=150; 75 benignos y 75 malignos). Se utilizó el aprendizaje de transferencia en tres RNP pre-formadas: VGG16, ResNet50 y MobileNet. Cada modelo fue formado primero mediante escalas de temperatura, al igual que los la calibración de los outputs. A continuación, se utilizó una combinación de los tres modelos para estimar la probabilidad de que el tumor fuera maligno con base en la totalidad de las imágenes de un caso determinado. La combinación de RNP permitió clasificar los tumores como benignos o malignos (modelo Ovry-Dx1); o como benignos, no concluyentes o malignos (modelo Ovry-Dx2). Se comparó el desempeño para el diagnóstico de los modelos de RNP, en términos de sensibilidad y de especificidad, con el de la ES para la clasificación de los tumores ováricos en el subconjunto de formación. RESULTADOS: Con una sensibilidad del 96,0%, Ovry-Dx1 tuvo una especificidad similar a la de la ES (86,7% frente a 88,0%; P=1,0). Ovry-Dx2 tuvo una sensibilidad del 97,1% y una especificidad del 93,7%, y designaron un 12,7% de las lesiones como no concluyentes. Cuando se complementó Ovry-Dx2 con ES en los casos no concluyentes, la sensibilidad general (96,0%) y la especificidad (89,3%) no fueron significativamente diferentes de la utilización de ES en todos los casos (P=1,0). CONCLUSIÓN: El análisis de imágenes ecográficas mediante RNP puede predecir el cáncer de ovario con una precisión en el diagnóstico igual a la de los especialistas expertos humanos, lo que indica que estos modelos pueden jugar un papel en el triaje de mujeres con un tumor de ovario. © 2020 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

采用基于深度神经网络的超声图像分析来辨别良性和恶性卵巢肿瘤:与专家主观评价对比 目的: 开发并检测采用基于深度神经网络(DNN)的电脑化超声图像分析来辨别良性和恶性卵巢肿瘤的性能，并将其诊断准确性与由一名超声专家进行主观评价(SA)的诊断准确性进行对比。 方法: 我们包含了3077张(灰度图，n=1927；能量多普勒，n=1150)来自758名有卵巢肿瘤妇女的超声图像，她们已由专家超声检查人员根据IOTA(国际卵巢肿瘤分析)术语和定义进行预期分类。以手术的组织学预后(n=634)或长期(≥3年)随访(n=124)作为金标准。数据集被分成训练集(n=508；314个良性和194个恶性)、验证集(n=100；60个良性和40个L恶性)和测试集(n=150；75个良性和75个恶性)。我们将迁移学习用于三个预先训练的DNN:VGG16、ResNet50和MobileNet。每个模型都受过训练，对结果进行过标准化，采用温度定标。然后，根据给定病例的所有图像，三个模型整体被用于判断恶性的可能性。DNN整体将肿瘤分类为良性或恶性(Ovry-Dx1模型)；或良性、不确定或恶性(Ovry-Dx2模型)。从敏感性和特异性角度，就测试集中卵巢肿瘤的分类将DNN模型的诊断性能与SA的诊断性能进行对比。 结果: 当敏感度为96.0%时，Ovry-Dx1的特异性类似于SA的特异性(86.7% vs 88.0%；P=1.0)。 在将12.7%的病灶指定为不确定时，Ovry-Dx2的敏感度为97.1%，特异性为93.7%。在不确定的病例中用Ovry-Dx2来补充SA，总体敏感度(96.0%)和特异性(89.3%)与在所有病例中使用SA(P=1.0)无显著差异。 结论: 基于DNN的超声图像分析可以用来预测卵巢恶性肿瘤，其诊断准确性能够达到人类专家检查人员的程度，表明这些模型可能在卵巢肿瘤妇女的分诊中起到作用。© 2020 作者。威利父子公司(John Wiley & Sons Ltd)代表国际妇产科超声学会(ISUOG)出版《国际妇产超声杂志》(Ultrasound in Obstetrics & Gynecology)。.

Keywords: classification; computer-aided diagnosis; deep learning; machine learning; ovarian neoplasm; ovarian tumor; transfer learning; ultrasonography.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Diagnostic performance for distinguishing between benign and malignant ovarian tumors of deep‐learning models VGG16 (formula image), ResNet50 (formula image), MobileNet (formula image) and ensemble of the three models (Ovry‐Dx1 in (a) and Ovry‐Dx2 in (b); (formula image)), in all patients in test set (a) and when excluding cases with predicted probability of malignancy between 0.4 and 0.6, corresponding to high uncertainty (b). Percentage of cases excluded was 10.7% for VGG16, 22.7% for ResNet50, 14.0% for MobileNet and 12.7% for Ovry‐Dx2. AUC, area under receiver‐operating‐characteristics curve.
Figure 2
Figure 2
Receiver‐operating‐characteristics (ROC) curves for distinguishing between benign and malignant ovarian tumors of deep‐learning models VGG16 (formula image), ResNet50 (formula image) and MobileNet (formula image) and ensemble of the three models (Ovry‐Dx1 in (a) and Ovry‐Dx2 in (b); (formula image)), in all patients in test set (a) and when excluding inconclusive cases (predicted probability of malignancy between 0.4 and 0.6) (b). In (a), ROC curve of IOTA simple‐rules risk (formula image) and operating point for expert subjective assessment (formula image) are also shown.
Figure 3
Figure 3
(a,b) Ten ovarian lesions that were difficult to classify by both expert subjective assessment and deep neural network model Ovry‐Dx2; final histological diagnosis was malignant or borderline in four (a) and benign in six (b). (c,d) Further nine ovarian lesions classified as inconclusive by Ovry‐Dx2; final histological diagnosis was malignant or borderline in three (c) and benign in six (d). Images marked with (formula image) were also classified as inconclusive by IOTA simple rules.
Figure 4
Figure 4
Six cases misdiagnosed by deep neural network model Ovry‐Dx2: (a,b) malignant cases classified as benign; (c–f) benign cases classified as malignant. (a) Clear‐cell carcinoma with 6‐mm papillary projection; case was also misdiagnosed by expert subjective assessment. (b) Mucinous intestinal borderline tumor. (c,d) Dermoid. (e) Fibroma; case was also misdiagnosed by expert subjective assessment. (f) Leydig cell tumor.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Webb PM, Jordan SJ. Epidemiology of epithelial ovarian cancer. Best Pract Res Clin Obstet Gynaecol 2017; 41: 3–14. - PubMed
    1. Sharma A, Apostolidou S, Burnell M, Campbell S, Habib M, Gentry‐Maharaj A, Amso N, Seif MW, Fletcher G, Singh N, Benjamin E, Brunell C, Turner G, Rangar R, Godfrey K, Oram D, Herod J, Williamson K, Jenkins H, Mould T, Woolas R, Murdoch J, Dobbs S, Leeson S, Cruickshank D, Fourkala EO, Ryan A, Parmar M, Jacobs I, Menon U. Risk of epithelial ovarian cancer in asymptomatic women with ultrasound‐detected ovarian masses: a prospective cohort study within the UK collaborative trial of ovarian cancer screening (UKCTOCS). Ultrasound Obstet Gynecol 2012; 40: 338–344. - PubMed
    1. Froyman W, Landolfo C, De Cock B, Wynants L, Sladkevicius P, Testa AC, Van Holsbeke C, Domali E, Fruscio R, Epstein E, Dos Santos Bernardo MJ, Franchi D, Kudla MJ, Chiappa V, Alcazar JL, Leone FPG, Buonomo F, Hochberg L, Coccia ME, Guerriero S, Deo N, Jokubkiene L, Kaijser J, Coosemans A, Vergote I, Verbakel JY, Bourne T, Van Calster B, Valentin L, Timmerman D. Risk of complications in patients with conservatively managed ovarian tumours (IOTA5): a 2‐year interim analysis of a multicentre, prospective, cohort study. Lancet Oncol 2019; 20: 448–458. - PubMed
    1. Engelen MJ, Kos HE, Willemse PH, Aalders JG, de Vries EG, Schaapveld M, Otter R, van der Zee AG. Surgery by consultant gynecologic oncologists improves survival in patients with ovarian carcinoma. Cancer 2006; 106: 589–598. - PubMed
    1. Meys EM, Kaijser J, Kruitwagen RF, Slangen BF, Van Calster B, Aertgeerts B, Verbakel JY, Timmerman D, Van Gorp T. Subjective assessment versus ultrasound models to diagnose ovarian cancer: A systematic review and meta‐analysis. Eur J Cancer 2016; 58: 17–29. - PubMed

Publication types