Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011 Dec;18 Suppl 1(Suppl 1):i150-6.
doi: 10.1136/amiajnl-2011-000431. Epub 2011 Sep 21.

Developing a natural language processing application for measuring the quality of colonoscopy procedures

Henk Harkema  1 , Wendy W Chapman, Melissa Saul, Evan S Dellon, Robert E Schoen, Ateev Mehrotra
Affiliations

Developing a natural language processing application for measuring the quality of colonoscopy procedures

Henk Harkema et al. J Am Med Inform Assoc. 2011 Dec.

Abstract

Objective: The quality of colonoscopy procedures for colorectal cancer screening is often inadequate and varies widely among physicians. Routine measurement of quality is limited by the costs of manual review of free-text patient charts. Our goal was to develop a natural language processing (NLP) application to measure colonoscopy quality.

Materials and methods: Using a set of quality measures published by physician specialty societies, we implemented an NLP engine that extracts 21 variables for 19 quality measures from free-text colonoscopy and pathology reports. We evaluated the performance of the NLP engine on a test set of 453 colonoscopy reports and 226 pathology reports, considering accuracy in extracting the values of the target variables from text, and the reliability of the outcomes of the quality measures as computed from the NLP-extracted information.

Results: The average accuracy of the NLP engine over all variables was 0.89 (range: 0.62-1.0) and the average F measure over all variables was 0.74 (range: 0.49-0.89). The average agreement score, measured as Cohen's κ, between the manually established and NLP-derived outcomes of the quality measures was 0.62 (range: 0.09-0.86).

Discussion: For nine of the 19 colonoscopy quality measures, the agreement score was 0.70 or above, which we consider a sufficient score for the NLP-derived outcomes of these measures to be practically useful for quality measurement.

Conclusion: The use of NLP for information extraction from free-text colonoscopy and pathology reports creates opportunities for large scale, routine quality measurement, which can support quality improvement in colonoscopy care.

PubMed Disclaimer

Conflict of interest statement

Competing interests: None.

Figures

Figure 1
Figure 1
General architecture of the NLP-based system for measuring the quality of colonoscopy procedures from free-text clinical reports. NLP, natural language processing.
See this image and copyright information in PMC

References

    1. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348:2635–45 - PubMed
    1. Chassin MR, Galvin RW. The urgent need to improve health care quality. Institute of Medicine National Roundtable on Health Care Quality. JAMA 1998;280:1000–5 - PubMed
    1. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: Institute of Medicine of The National Academies, 2001 - PubMed
    1. Performance Measurement: Accelerating Improvement. Washington, DC: Institute of Medicine of The National Academies, 2005
    1. Diamond CC, Rask KJ, Kohler SA. Use of paper medical records versus administrative data for measuring and improving health care quality: are we still searching for a gold standard? Dis Manag 2001;4:121–30

Publication types