Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Sep 20:26:e58278.
doi: 10.2196/58278.

Evaluating a Natural Language Processing-Driven, AI-Assisted International Classification of Diseases, 10th Revision, Clinical Modification, Coding System for Diagnosis Related Groups in a Real Hospital Environment: Algorithm Development and Validation Study

Hong-Jie Dai  1   2   3 , Chen-Kai Wang  1   4   5 , Chien-Chang Chen  6 , Chong-Sin Liou  1 , An-Tai Lu  7 , Chia-Hsin Lai  1 , Bo-Tsz Shain  1 , Cheng-Rong Ke  1 , William Yu Chung Wang  8 , Tatheer Hussain Mir  1 , Mutiara Simanjuntak  1 , Hao-Yun Kao #  9 , Ming-Ju Tsai #  10 , Vincent S Tseng #  4
Affiliations

Evaluating a Natural Language Processing-Driven, AI-Assisted International Classification of Diseases, 10th Revision, Clinical Modification, Coding System for Diagnosis Related Groups in a Real Hospital Environment: Algorithm Development and Validation Study

Hong-Jie Dai et al. J Med Internet Res. .

Abstract

Background: International Classification of Diseases codes are widely used to describe diagnosis information, but manual coding relies heavily on human interpretation, which can be expensive, time consuming, and prone to errors. With the transition from the International Classification of Diseases, Ninth Revision, to the International Classification of Diseases, Tenth Revision (ICD-10), the coding process has become more complex, highlighting the need for automated approaches to enhance coding efficiency and accuracy. Inaccurate coding can result in substantial financial losses for hospitals, and a precise assessment of outcomes generated by a natural language processing (NLP)-driven autocoding system thus assumes a critical role in safeguarding the accuracy of the Taiwan diagnosis related groups (Tw-DRGs).

Objective: This study aims to evaluate the feasibility of applying an International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM), autocoding system that can automatically determine diagnoses and codes based on free-text discharge summaries to facilitate the assessment of Tw-DRGs, specifically principal diagnosis and major diagnostic categories (MDCs).

Methods: By using the patient discharge summaries from Kaohsiung Medical University Chung-Ho Memorial Hospital (KMUCHH) from April 2019 to December 2020 as a reference data set we developed artificial intelligence (AI)-assisted ICD-10-CM coding systems based on deep learning models. We constructed a web-based user interface for the AI-assisted coding system and deployed the system to the workflow of the certified coding specialists (CCSs) of KMUCHH. The data used for the assessment of Tw-DRGs were manually curated by a CCS with the principal diagnosis and MDC was determined from discharge summaries collected at KMUCHH from February 2023 to April 2023.

Results: Both the reference data set and real hospital data were used to assess performance in determining ICD-10-CM coding, principal diagnosis, and MDC for Tw-DRGs. Among all methods, the GPT-2 (OpenAI)-based model achieved the highest F1-score, 0.667 (F1-score 0.851 for the top 50 codes), on the KMUCHH test set and a slightly lower F1-score, 0.621, in real hospital data. Cohen κ evaluation for the agreement of MDC between the models and the CCS revealed that the overall average κ value for GPT-2 (κ=0.714) was approximately 12.2 percentage points higher than that of the hierarchy attention network (κ=0.592). GPT-2 demonstrated superior agreement with the CCS across 6 categories of MDC, with an average κ value of approximately 0.869 (SD 0.033), underscoring the effectiveness of the developed AI-assisted coding system in supporting the work of CCSs.

Conclusions: An NLP-driven AI-assisted coding system can assist CCSs in ICD-10-CM coding by offering coding references via a user interface, demonstrating the potential to reduce the manual workload and expedite Tw-DRG assessment. Consistency in performance affirmed the effectiveness of the system in supporting CCSs in ICD-10-CM coding and the judgment of Tw-DRGs.

Keywords: International Classification of Diseases; Taiwan diagnosis related groups; deep learning; electronic medical record; natural language processing.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Architecture of the hierarchical attention network implementation in this study. ICD-10-CM: International Classification of Diseases, 10th Revision, Clinical Modification.
Figure 2
Figure 2
An example of fine-tuned data for GPT-2.
Figure 3
Figure 3
User interface of a natural language processing–driven artificial intelligence–assisted coding system at Kaohsiung Medical University Chung-Ho Memorial Hospital.
Figure 4
Figure 4
Visualization of the attentive key terms highlighted in a given discharge summary by the attention mechanism.
Figure 5
Figure 5
Workflow of the natural language processing–driven artificial intelligence–assisted coding system deployed in Kaohsiung Medical University Chung-Ho Memorial Hospital. ICD-10-CM: International Classification of Diseases, 10th Revision, Clinical Modification; SQL: structured query language.
See this image and copyright information in PMC

References

    1. International classification of diseases:[9th] ninth revision, basic tabulation list with alphabetic index. World Health Organization. 1978. [2024-04-29]. https://iris.who.int/handle/10665/39473 .
    1. Sivashankaran S, Borsi JP, Yoho A. Have ICD-10 coding practices changed since 2015? AMIA Annu Symp Proc. 2019;2019:804–11. https://europepmc.org/abstract/MED/32308876 - PMC - PubMed
    1. de Lissovoy G. Codes, coding, and COVID-19. Med Care. 2020 Dec;58(12):1035–6. doi: 10.1097/MLR.0000000000001430.00005650-202012000-00001 - DOI - PubMed
    1. Steindel SJ. International classification of diseases, 10th edition, clinical modification and procedure coding system: descriptive overview of the next generation HIPAA code sets. J Am Med Inform Assoc. 2010 May 01;17(3):274–82. doi: 10.1136/jamia.2009.001230. https://europepmc.org/abstract/MED/20442144 17/3/274 - DOI - PMC - PubMed
    1. Mills RE, Butler RR, McCullough EC, Bao MZ, Averill RF. Impact of the transition to ICD-10 on Medicare inpatient hospital payments. Medicare Medicaid Res Rev. 2011 Jun 06;1(2):E1–13. doi: 10.5600/mmrr.001.02.a02. https://europepmc.org/abstract/MED/22340773 - DOI - PMC - PubMed

Publication types

LinkOut - more resources