Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 May 23:2022:486-495.
eCollection 2022.

Prior Knowledge Enhances Radiology Report Generation

Song Wang  1 , Liyan Tang  1 , Mingquan Lin  2 , George Shih  3 , Ying Ding  1 , Yifan Peng  2
Affiliations

Prior Knowledge Enhances Radiology Report Generation

Song Wang et al. AMIA Jt Summits Transl Sci Proc. .

Abstract

Radiology report generation aims to produce computer-aided diagnoses to alleviate the workload of radiologists and has drawn increasing attention recently. However, previous deep learning methods tend to neglect the mutual influences between medical findings, which can be the bottleneck that limits the quality of generated reports. In this work, we propose to mine and represent the associations among medical findings in an informative knowledge graph and incorporate this prior knowledge with radiology report generation to help improve the quality of generated reports. Experiment results demonstrate the superior performance of our proposed method on the IU X-ray dataset with a ROUGE-L of 0.384±0.007 and CIDEr of 0.340±0.011. Compared with previous works, our model achieves an average of 1.6% improvement (2.0% and 1.5% improvements in CIDEr and ROUGE-L, respectively). The experiments suggest that prior knowledge can bring performance gains to accurate radiology report generation. We will make the code publicly available at https://github.com/bionlplab/report_generation_amia2022.

PubMed Disclaimer

Figures

Figure 1:
Figure 1:
An example of the generated report. Concepts marked in red are the concept nodes in our knowledge graph. Chest X-ray images are encoded first, the image representations will then be used to generate texts.
Figure 2:
Figure 2:
The proposed framework.
Figure 3:
Figure 3:
Average sentence number of reports of different BLEU scores.
Figure 4:
Figure 4:
Three examples where our model generates low-scored reports and high-scored reports.
See this image and copyright information in PMC

References

    1. Pahadia M, Khurana S, Geha H, Deahl STI. Radiology report writing skills: A linguistic and technical guide for early-career oral and maxillofacial radiologists. Imaging Science in Dentistry. 2020 Sep;50(3):269–272. - PMC - PubMed
    1. Shin HC, Lu L, Kim L, Seff A, Yao J, Summers RM. Interleaved text/image deep mining on a large-scale radiology database for automated image interpretation. The Journal of Machine Learning Research. 2016;17(1):3729–3759.
    1. Rennie SJ, Marcheret E, Mroueh Y, Ross J, Goel V. Self-Critical Sequence Training for Image Captioning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA, USA: IEEE Computer Society; 2017. p. 1179-1195. Available from: https://doi.ieeecomputersociety.org/ 10.1109/CVPR.2017.131.
    1. Lu J, Xiong C, Parikh D, Socher R. Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017. p. 3242-3250.
    1. Anderson P, He X, Buehler C, Teney D, Johnson M, Gould S, et al. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2018. p. 6077-6086.

LinkOut - more resources