Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Nov 1;26(11):1247-1254.
doi: 10.1093/jamia/ocz149.

Evaluating shallow and deep learning strategies for the 2018 n2c2 shared task on clinical text classification

Michel Oleynik  1 , Amila Kugic  1 , Zdenko Kasáč  1 , Markus Kreuzthaler  1   2
Affiliations

Evaluating shallow and deep learning strategies for the 2018 n2c2 shared task on clinical text classification

Michel Oleynik et al. J Am Med Inform Assoc. .

Abstract

Objective: Automated clinical phenotyping is challenging because word-based features quickly turn it into a high-dimensional problem, in which the small, privacy-restricted, training datasets might lead to overfitting. Pretrained embeddings might solve this issue by reusing input representation schemes trained on a larger dataset. We sought to evaluate shallow and deep learning text classifiers and the impact of pretrained embeddings in a small clinical dataset.

Materials and methods: We participated in the 2018 National NLP Clinical Challenges (n2c2) Shared Task on cohort selection and received an annotated dataset with medical narratives of 202 patients for multilabel binary text classification. We set our baseline to a majority classifier, to which we compared a rule-based classifier and orthogonal machine learning strategies: support vector machines, logistic regression, and long short-term memory neural networks. We evaluated logistic regression and long short-term memory using both self-trained and pretrained BioWordVec word embeddings as input representation schemes.

Results: Rule-based classifier showed the highest overall micro F1 score (0.9100), with which we finished first in the challenge. Shallow machine learning strategies showed lower overall micro F1 scores, but still higher than deep learning strategies and the baseline. We could not show a difference in classification efficiency between self-trained and pretrained embeddings.

Discussion: Clinical context, negation, and value-based criteria hindered shallow machine learning approaches, while deep learning strategies could not capture the term diversity due to the small training dataset.

Conclusion: Shallow methods for clinical phenotyping can still outperform deep learning methods in small imbalanced data, even when supported by pretrained embeddings.

Keywords: data mining; deep learning; machine learning; natural language processing.

PubMed Disclaimer

References

    1. Meystre SM, Savova GK, Kipper-Schuler KC, et al. Extracting information from textual documents in the electronic health record: a review of recent research. Yearb Med Inform 2008; 17: 128–44. - PubMed
    1. Hebal F, Nanney E, Stake C, et al. Automated data extraction: merging clinical care with real-time cohort-specific research and quality improvement data. J Pediatr Surg 2017; 521: 149–52. - PubMed
    1. Safran C, Bloomrosen M, Hammond WE, et al. Toward a national framework for the secondary use of health data: an American medical informatics association white paper. J Am Med Inform Assoc 2007; 141: 1–9. - PMC - PubMed
    1. Mann CJ. Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emerg Med J 2003; 201: 54–60. - PMC - PubMed
    1. Geneletti S, Richardson S, Best N.. Adjusting for selection bias in retrospective, case–control studies. Biostatistics 2008; 101: 17–31. - PubMed

Publication types