. 2020 Apr 29;2(3):e190211.

doi: 10.1148/ryai.2020190211. eCollection 2020 May.

Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge

Adam E Flanders¹, Luciano M Prevedello¹, George Shih¹, Safwan S Halabi¹, Jayashree Kalpathy-Cramer¹, Robyn Ball¹, John T Mongan¹, Anouk Stein¹, Felipe C Kitamura¹, Matthew P Lungren¹, Gagandeep Choudhary¹, Lesley Cala¹, Luiz Coelho¹, Monique Mogensen¹, Fanny Morón¹, Elka Miller¹, Ichiro Ikuta¹, Vahe Zohrabian¹, Olivia McDonnell¹, Christie Lincoln¹, Lubdha Shah¹, David Joyner¹, Amit Agarwal¹, Ryan K Lee¹, Jaya Nath¹; RSNA-ASNR 2019 Brain Hemorrhage CT Annotators

Collaborators, Affiliations

Collaborators

RSNA-ASNR 2019 Brain Hemorrhage CT Annotators:
Amit Agarwal, Fernando Aparici, Ahmad Atassi, Yulia Bronstein, Karen Buch, Rafael Burbano, Harsha Chadaga, Christopher Chan, Amy Chen, Arthur Cortez, Bradley Delman, Cliff Eskey, Luciano Farage, Wende Gibbs, Gloria Guzman Perez-Carrillo, Bronwyn Hamilton, Christopher Hancock, Alvand Hassankhani, Mark Herbst, Tarek Hijaz, Michael Hollander, Samuel Hou, Rajan Jain, David Joyner, Jinsuh Kim, Xiaojie Luo, Ben McGuinness, Anthony Miller, Todd Mulderink, Momin Muzaffar, Thomas O’Neill, Mark Oswood, David Pastel, Boyan Petrovic, Vaishali Phalke, Douglas Phillips, Sanjay Prabhu, Gary Press, Rafael Rojas, Zoran Rumboldt, Lubdha Shah, Dinesh Sharma, Achint Singh, Shira Slasky, James Smirniotopoulos, Joel Stein, Mary Tenenbaum, Vinson Uytana

Affiliation

¹ Department of Radiology/Division of Neuroradiology, Thomas Jefferson University Hospital, 132 S Tenth St, Suite 1080B Main Building, Philadelphia, PA 19107 (A.E.F.); Department of Radiology, The Ohio State University, Columbus, Ohio (L.M.P.); Department of Radiology, Weill Cornell Medical College, New York, NY (G.S.); Department of Radiology, Stanford University, Stanford, Calif (S.S.H.); Department of Radiology and Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, Mass (J.K.); Quantitative Sciences Unit, Stanford University, Stanford, Calif (R.B.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); MD.ai, New York, NY (A.S.); Department of Diagnostic Imaging, Universidade Federal de São Paulo, São Paulo, Brazil (F.C.K.); Department of Radiology, Stanford University, Stanford, Calif (M.P.L.); Department of Radiology, University of Alabama at Birmingham, Birmingham, Ala (G.C.); Faculty of Health and Medical Sciences, University of Western Australia, Perth, Australia (L. Cala); Advanced Diagnostic Imaging, Clínica DAPI, Curitiba, Brazil (L. Coelho); Department of Radiology, University of Washington, Seattle, Wash (M.M.); Department of Radiology, Baylor College of Medicine, Houston, Tex (F.M., C.L.); Department of Radiology, University of Ottawa, Ottawa, Canada (E.M.); Department of Radiology & Biomedical Imaging, Yale University, New Haven, Conn (I.I., V.Z.); Department of Medical Imaging, Gold Coast University Hospital, Southport, Australia (O.M.); Department of Neuroradiology, University of Utah Health Sciences Center, Salt Lake City, Utah (L.S.); Department of Radiology and Medical Imaging, University of Virginia Health, Charlottesville, Va (D.J.); Division of Neuroradiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.A.); Department of Radiology, Albert Einstein Healthcare Network, Philadelphia, Pa (R.K.L.); and Department of Radiology, SUNY Downstate Medical Center, Albany, NY (J.N.).

PMID: 33937827
PMCID: PMC8082297
DOI: 10.1148/ryai.2020190211

Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge

Adam E Flanders et al. Radiol Artif Intell. 2020.

. 2020 Apr 29;2(3):e190211.

doi: 10.1148/ryai.2020190211. eCollection 2020 May.

Authors

Collaborators

RSNA-ASNR 2019 Brain Hemorrhage CT Annotators:
Amit Agarwal, Fernando Aparici, Ahmad Atassi, Yulia Bronstein, Karen Buch, Rafael Burbano, Harsha Chadaga, Christopher Chan, Amy Chen, Arthur Cortez, Bradley Delman, Cliff Eskey, Luciano Farage, Wende Gibbs, Gloria Guzman Perez-Carrillo, Bronwyn Hamilton, Christopher Hancock, Alvand Hassankhani, Mark Herbst, Tarek Hijaz, Michael Hollander, Samuel Hou, Rajan Jain, David Joyner, Jinsuh Kim, Xiaojie Luo, Ben McGuinness, Anthony Miller, Todd Mulderink, Momin Muzaffar, Thomas O’Neill, Mark Oswood, David Pastel, Boyan Petrovic, Vaishali Phalke, Douglas Phillips, Sanjay Prabhu, Gary Press, Rafael Rojas, Zoran Rumboldt, Lubdha Shah, Dinesh Sharma, Achint Singh, Shira Slasky, James Smirniotopoulos, Joel Stein, Mary Tenenbaum, Vinson Uytana

Affiliation

¹ Department of Radiology/Division of Neuroradiology, Thomas Jefferson University Hospital, 132 S Tenth St, Suite 1080B Main Building, Philadelphia, PA 19107 (A.E.F.); Department of Radiology, The Ohio State University, Columbus, Ohio (L.M.P.); Department of Radiology, Weill Cornell Medical College, New York, NY (G.S.); Department of Radiology, Stanford University, Stanford, Calif (S.S.H.); Department of Radiology and Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, Mass (J.K.); Quantitative Sciences Unit, Stanford University, Stanford, Calif (R.B.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); MD.ai, New York, NY (A.S.); Department of Diagnostic Imaging, Universidade Federal de São Paulo, São Paulo, Brazil (F.C.K.); Department of Radiology, Stanford University, Stanford, Calif (M.P.L.); Department of Radiology, University of Alabama at Birmingham, Birmingham, Ala (G.C.); Faculty of Health and Medical Sciences, University of Western Australia, Perth, Australia (L. Cala); Advanced Diagnostic Imaging, Clínica DAPI, Curitiba, Brazil (L. Coelho); Department of Radiology, University of Washington, Seattle, Wash (M.M.); Department of Radiology, Baylor College of Medicine, Houston, Tex (F.M., C.L.); Department of Radiology, University of Ottawa, Ottawa, Canada (E.M.); Department of Radiology & Biomedical Imaging, Yale University, New Haven, Conn (I.I., V.Z.); Department of Medical Imaging, Gold Coast University Hospital, Southport, Australia (O.M.); Department of Neuroradiology, University of Utah Health Sciences Center, Salt Lake City, Utah (L.S.); Department of Radiology and Medical Imaging, University of Virginia Health, Charlottesville, Va (D.J.); Division of Neuroradiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.A.); Department of Radiology, Albert Einstein Healthcare Network, Philadelphia, Pa (R.K.L.); and Department of Radiology, SUNY Downstate Medical Center, Albany, NY (J.N.).

PMID: 33937827
PMCID: PMC8082297
DOI: 10.1148/ryai.2020190211

Erratum in

Erratum: Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge.
Flanders AE, Prevedello LM, Shih G, Halabi SS, Kalpathy-Cramer J, Ball R, Mongan JT, Stein A, Kitamura FC, Lungren MP, Choudhary G, Cala L, Coelho L, Mogensen M, Morón F, Miller E, Ikuta I, Zohrabian V, McDonnell O, Lincoln C, Shah L, Joyner D, Agarwal A, Lee RK, Nath J; RSNA-ASNR 2019 Brain Hemorrhage CT Annotators. Flanders AE, et al. Radiol Artif Intell. 2020 Jul 29;2(4):e209002. doi: 10.1148/ryai.2020209002. eCollection 2020 Jul. Radiol Artif Intell. 2020. PMID: 33939782 Free PMC article.

Abstract

This dataset is composed of annotations of the five hemorrhage subtypes (subarachnoid, intraventricular, subdural, epidural, and intraparenchymal hemorrhage) typically encountered at brain CT.

2020 by the Radiological Society of North America, Inc.

PubMed Disclaimer

Conflict of interest statement

Disclosures of Conflicts of Interest: A.E.F. disclosed no relevant relationships. L.M.P. Activities related to the present article: institution receives time-limited loaner access by OSU AI Lab to NVIDIA GPUs via Master Research Agreement between The Ohio State University and NVIDIA, no money transfer; time-limited access by OSU AI Lab to WIP postprocessing software via Master Research Agreement between OSU and Siemens Healthineers. No money transfer; unrestrictive support of OSU AI Lab by DeBartolo Family Funds. Activities not related to the present article: disclosed no relevant relationships. Other relationships: disclosed no relevant relationships. G.S. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: board member and shareholder of MD.ai but no money paid to author or MD.ai. Other relationships: disclosed no relevant relationships. S.S.H. disclosed no relevant relationships. J.K. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: author is consultant for INFOTECH, Soft; institution receives grants from Genentech and GE. Other relationships: disclosed no relevant relationships. R.B. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: previously employed by Roam Analytics (employment ended prior to working on this study); stockholder in Roam Analytics. Other relationships: disclosed no relevant relationships. J.T.M. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: consultant for Siemens (PAMA clinical decision support related); institution receives grants from GE (past and pending grants related to PTX detection); author may receive future royalties from GE related to PTX AI detector licensed to GE. Other relationships: disclosed no relevant relationships. A.S. Activities related to the present article: employee and shareholder in MD.ai. Activities not related to the present article: employee and shareholder in MD.ai. Other relationships: disclosed no relevant relationships. F.C.K. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: consultant for MD.ai; employed by Diagnósticos da América (DASA). Other relationships: disclosed no relevant relationships. M.P.L. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: consultant for and stockholder in Nines, SegMed, and Bunker Hill. Other relationships: disclosed no relevant relationships. G.C. disclosed no relevant relationships. L. Cala disclosed no relevant relationships. L. Coelho disclosed no relevant relationships. M.M. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: consultant for Cerebrotech Medical Systems. Other relationships: disclosed no relevant relationships. F.M. disclosed no relevant relationships. E.M. disclosed no relevant relationships. I.I. disclosed no relevant relationships. V.Z. disclosed no relevant relationships. O.M. disclosed no relevant relationships. C.L. disclosed no relevant relationships. L.S. disclosed no relevant relationships. D.J. disclosed no relevant relationships. A.A. disclosed no relevant relationships. R.K.L. disclosed no relevant relationships. J.N. disclosed no relevant relationships.

Figures

Workflow diagram for image data query, extraction, curation,
anonymization, and exportation by the three contributing institutions. DICOM
= Digital Imaging and Communications in Medicine, ID = identification,
MIRC-CTP = Medical Image Resource Center-Clinical Trials Processor, PACS
= picture archiving and communication system, RSNA = Radiological
Society of North America. — **Figure 1:**
Workflow diagram for image data query, extraction, curation, anonymization, and exportation by the three contributing institutions. DICOM = Digital Imaging and Communications in Medicine, ID = identification, MIRC-CTP = Medical Image Resource Center-Clinical Trials Processor, PACS = picture archiving and communication system, RSNA = Radiological Society of North America.

Workflow process diagram illustrates the steps to creation of the final
brain CT hemorrhage dataset starting from solicitation from respective
institutions to creation of the final collated and balanced datasets. ASNR
= American Society of Neuroradiology, DICOM = Digital Imaging and
Communications in Medicine, UIDs = unique identifiers. — **Figure 2:**
Workflow process diagram illustrates the steps to creation of the final brain CT hemorrhage dataset starting from solicitation from respective institutions to creation of the final collated and balanced datasets. ASNR = American Society of Neuroradiology, DICOM = Digital Imaging and Communications in Medicine, UIDs = unique identifiers.

A complex multicompartmental cerebral hemorrhage on a single axial CT
image displayed using the annotation tool in a single portal window. Hemorrhage
labels (left column) relevant to the image display on the bottom of the image
once selected. ASNR = American Society of Neuroradiology, RSNA
=Radiological Society of North America. — **Figure 3:**
A complex multicompartmental cerebral hemorrhage on a single axial CT image displayed using the annotation tool in a single portal window. Hemorrhage labels (left column) relevant to the image display on the bottom of the image once selected. ASNR = American Society of Neuroradiology, RSNA =Radiological Society of North America.

Distribution of examination labels in the final training (blue) and test
(orange) datasets. The “any hemorrhage” designation represents
when one or more of the hemorrhage subclasses were present in the entire
examination. — **Figure 4:**
Distribution of examination labels in the final training (blue) and test (orange) datasets. The “any hemorrhage” designation represents when one or more of the hemorrhage subclasses were present in the entire examination.

Distribution of image-based labels in the final training (blue) and test
(orange) datasets. The “any hemorrhage” designation represents
when one or more of the hemorrhage subclasses were present on an
image. — **Figure 5:**
Distribution of image-based labels in the final training (blue) and test (orange) datasets. The “any hemorrhage” designation represents when one or more of the hemorrhage subclasses were present on an image.

See this image and copyright information in PMC

References

1. Cordonnier C , Demchuk A , Ziai W , Anderson CS . Intracerebral haemorrhage: current approaches to acute management . Lancet 2018. ; 392 ( 10154 ): 1257 – 1268 . - PubMed
1. Majumdar A , Brattain L , Telfer B , Farris C , Scalera J . Detecting Intracranial Hemorrhage with Deep Learning . Conf Proc IEEE Eng Med Biol Soc 2018. ; 2018 : 583 – 587 . - PubMed
1. Kuo W , Hӓne C , Mukherjee P , Malik J , Yuh EL . Expert-level detection of acute intracranial hemorrhage on head computed tomography using deep learning . Proc Natl Acad Sci U S A 2019. ; 116 ( 45 ): 22737 – 22745 . - PMC - PubMed
1. Dawud AM , Yurtkan K , Oztoprak H . Application of Deep Learning in Neuroradiology: Brain Haemorrhage Classification Using Transfer Learning . Comput Intell Neurosci 2019. ; 2019 : 4629859 . - PMC - PubMed
1. Ginat DT . Analysis of head CT scans flagged by deep learning software for acute intracranial hemorrhage . Neuroradiology 2020. ; 62 ( 3 ): 335 – 340 . - PubMed

Grants and funding

UL1 TR001863/TR/NCATS NIH HHS/United States

LinkOut - more resources

Full Text Sources
Other Literature Sources
- The Lens - Patent Citations Database

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge

Collaborators

Affiliation

Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge

Authors

Collaborators

Affiliation

Erratum in

Abstract

Conflict of interest statement

Figures

References

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources