Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2017 May:69:177-187.
doi: 10.1016/j.jbi.2017.04.011. Epub 2017 Apr 18.

Automated annotation and classification of BI-RADS assessment from radiology reports

Sergio M Castro  1 , Eugene Tseytlin  2 , Olga Medvedeva  3 , Kevin Mitchell  4 , Shyam Visweswaran  5 , Tanja Bekhuis  6 , Rebecca S Jacobson  7
Affiliations

Automated annotation and classification of BI-RADS assessment from radiology reports

Sergio M Castro et al. J Biomed Inform. 2017 May.

Abstract

The Breast Imaging Reporting and Data System (BI-RADS) was developed to reduce variation in the descriptions of findings. Manual analysis of breast radiology report data is challenging but is necessary for clinical and healthcare quality assurance activities. The objective of this study is to develop a natural language processing (NLP) system for automated BI-RADS categories extraction from breast radiology reports. We evaluated an existing rule-based NLP algorithm, and then we developed and evaluated our own method using a supervised machine learning approach. We divided the BI-RADS category extraction task into two specific tasks: (1) annotation of all BI-RADS category values within a report, (2) classification of the laterality of each BI-RADS category value. We used one algorithm for task 1 and evaluated three algorithms for task 2. Across all evaluations and model training, we used a total of 2159 radiology reports from 18 hospitals, from 2003 to 2015. Performance with the existing rule-based algorithm was not satisfactory. Conditional random fields showed a high performance for task 1 with an F-1 measure of 0.95. Rules from partial decision trees (PART) algorithm showed the best performance across classes for task 2 with a weighted F-1 measure of 0.91 for BIRADS 0-6, and 0.93 for BIRADS 3-5. Classification performance by class showed that performance improved for all classes from Naïve Bayes to Support Vector Machine (SVM), and also from SVM to PART. Our system is able to annotate and classify all BI-RADS mentions present in a single radiology report and can serve as the foundation for future studies that will leverage automated BI-RADS annotation, to provide feedback to radiologists as part of a learning health system loop.

Keywords: Breast Imaging Reporting and Data System (BI-RADS); Imaging informatics; Information extraction; Machine learning; Natural language processing.

PubMed Disclaimer

Conflict of interest statement

9. Conflicts of Interest

At the time of this publication, the authors do not report any conflict of interest.

Figures

Figure 1
Figure 1
Methods for development of the two-step machine learning annotator.
Figure 2
Figure 2
Example radiology reports of increasing complexity with multiple BIRADS statements depicting the two separate tasks.
Figure 3
Figure 3
Corpus annotation process and inter-annotator agreement
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Siegel RL, Miller KD, Jemal A. Cancer statistics, 2016: Cancer Statistics, 2016. CA Cancer J Clin. 2016;66:7–30. - PubMed
    1. Torre LA, Bray F, Siegel RL, Ferlay J, Lortet-Tieulent J, Jemal A. Global cancer statistics, 2012: Global Cancer Statistics, 2012. CA Cancer J Clin. 2015;65:87–108. - PubMed
    1. Berry DA, Cronin KA, Plevritis SK, Fryback DG, Clarke L, Zelen M, et al. Effect of Screening and Adjuvant Therapy on Mortality from Breast Cancer. New England Journal of Medicine. 2005;353:1784–92. - PubMed
    1. U.S. Preventive Services Task Force. Screening for breast cancer: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med. 2009;151:716–26. W-236. - PubMed
    1. Oeffinger KC, Fontham EH, Etzioni R, et al. Breast cancer screening for women at average risk: 2015 guideline update from the american cancer society. Journal of the American Medical Association. 2015;314:1599–614. - PMC - PubMed