Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Jun 20;31(7):1596-1607.
doi: 10.1093/jamia/ocae108.

Deep learning with noisy labels in medical prediction problems: a scoping review

Yishu Wei  1   2 , Yu Deng  3 , Cong Sun  1 , Mingquan Lin  1   4 , Hongmei Jiang  5 , Yifan Peng  1
Affiliations

Deep learning with noisy labels in medical prediction problems: a scoping review

Yishu Wei et al. J Am Med Inform Assoc. .

Abstract

Objectives: Medical research faces substantial challenges from noisy labels attributed to factors like inter-expert variability and machine-extracted labels. Despite this, the adoption of label noise management remains limited, and label noise is largely ignored. To this end, there is a critical need to conduct a scoping review focusing on the problem space. This scoping review aims to comprehensively review label noise management in deep learning-based medical prediction problems, which includes label noise detection, label noise handling, and evaluation. Research involving label uncertainty is also included.

Methods: Our scoping review follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We searched 4 databases, including PubMed, IEEE Xplore, Google Scholar, and Semantic Scholar. Our search terms include "noisy label AND medical/healthcare/clinical," "uncertainty AND medical/healthcare/clinical," and "noise AND medical/healthcare/clinical."

Results: A total of 60 papers met inclusion criteria between 2016 and 2023. A series of practical questions in medical research are investigated. These include the sources of label noise, the impact of label noise, the detection of label noise, label noise handling techniques, and their evaluation. Categorization of both label noise detection methods and handling techniques are provided.

Discussion: From a methodological perspective, we observe that the medical community has been up to date with the broader deep-learning community, given that most techniques have been evaluated on medical data. We recommend considering label noise as a standard element in medical research, even if it is not dedicated to handling noisy labels. Initial experiments can start with easy-to-implement methods, such as noise-robust loss functions, weighting, and curriculum learning.

Keywords: deep learning; label uncertainty; noisy label.

PubMed Disclaimer

Conflict of interest statement

There are no competing interests to declare.

Figures

Figure 1.
Figure 1.
Flowchart of the search strategy.
Figure 2.
Figure 2.
Summary of noisy label detection methods. Terminologies in the figure are explained in the Glossary.
Figure 3.
Figure 3.
Methods applied to handle label noise.
Figure 4.
Figure 4.
Evaluation method summary.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Salahuddin Z, Woodruff HC, Chatterjee A, Lambin P. Transparency of deep neural networks for medical image analysis: a review of interpretability methods. Comput Biol Med. 2022;140:105111. - PubMed
    1. Wallace DK, Quinn GE, Freedman SF, Chiang MF. Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS. 2008;12(4):352-356. - PMC - PubMed
    1. Frénay B, Kabán A. A comprehensive introduction to label noise. In: The European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. i6doc.com publ; 2014:23-25.
    1. Song H, Kim M, Park D, Shin Y, Lee JG. Learning from noisy labels with deep neural networks: a survey. IEEE Trans Neural Netw Learn Syst. 2023; 34(11):8135-8153. - PubMed
    1. Algan G, Ulusoy I. Image classification with deep learning in the presence of noisy labels: a survey. Knowledge-Based Syst. 2021;215:106771.

Publication types