Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Nov 22:2:e40965.
doi: 10.2196/40965.

Real-Time Classification of Causes of Death Using AI: Sensitivity Analysis

Patrícia Pita Ferreira  1   2   3 , Diogo Godinho Simões  1   4 , Constança Pinto de Carvalho  1   5 , Francisco Duarte  6 , Eugénia Fernandes  1 , Pedro Casaca Carvalho  1 , José Francisco Loff  7 , Ana Paula Soares  1 , Maria João Albuquerque  1 , Pedro Pinto-Leite  1 , André Peralta-Santos  1   8   9
Affiliations

Real-Time Classification of Causes of Death Using AI: Sensitivity Analysis

Patrícia Pita Ferreira et al. JMIR AI. .

Abstract

Background: In 2021, the European Union reported >270,000 excess deaths, including >16,000 in Portugal. The Portuguese Directorate-General of Health developed a deep neural network, AUTOCOD, which determines the primary causes of death by analyzing the free text of physicians' death certificates (DCs). Although AUTOCOD's performance has been established, it remains unclear whether its performance remains consistent over time, particularly during periods of excess mortality.

Objective: This study aims to assess the sensitivity and other performance metrics of AUTOCOD in classifying underlying causes of death compared with manual coding to identify specific causes of death during periods of excess mortality.

Methods: We included all DCs between 2016 and 2019. AUTOCOD's performance was evaluated by calculating various performance metrics, such as sensitivity, specificity, positive predictive value (PPV), and F1-score, using a confusion matrix. This compared International Statistical Classification of Diseases and Health-Related Problems, 10th Revision (ICD-10), classifications of DCs by AUTOCOD with those by human coders at the Directorate-General of Health (gold standard). Subsequently, we compared periods without excess mortality with periods of excess, severe, and extreme excess mortality. We defined excess mortality as 2 consecutive days with a Z score above the 95% baseline limit, severe excess mortality as 2 consecutive days with a Z score >4 SDs, and extreme excess mortality as 2 consecutive days with a Z score >6 SDs. Finally, we repeated the analyses for the 3 most common ICD-10 chapters focusing on block-level classification.

Results: We analyzed a large data set comprising 330,098 DCs classified by both human coders and AUTOCOD. AUTOCOD demonstrated high sensitivity (≥0.75) for 10 ICD-10 chapters examined, with values surpassing 0.90 for the more prevalent chapters (chapter II-"Neoplasms," chapter IX-"Diseases of the circulatory system," and chapter X-"Diseases of the respiratory system"), accounting for 67.69% (223,459/330,098) of all human-coded causes of death. No substantial differences were observed in these high-sensitivity values when comparing periods without excess mortality with periods of excess, severe, and extreme excess mortality. The same holds for specificity, which exceeded 0.96 for all chapters examined, and for PPV, which surpassed 0.75 in 9 chapters, including the more prevalent ones. When considering block classification within the 3 most common ICD-10 chapters, AUTOCOD maintained a high performance, demonstrating high sensitivity (≥0.75) for 13 ICD-10 blocks, high PPV for 9 blocks, and specificity of >0.98 in all blocks, with no significant differences between periods without excess mortality and those with excess mortality.

Conclusions: Our findings indicate that, during periods of excess and extreme excess mortality, AUTOCOD's performance remains unaffected by potential text quality degradation because of pressure on health services. Consequently, AUTOCOD can be dependably used for real-time cause-specific mortality surveillance even in extreme excess mortality situations.

Keywords: AI; artificial intelligence; deep learning; deep neural networks; evaluation; machine learning; mortality; mortality statistics; underlying cause of death.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Flowchart of the study population inclusion criteria. DC: death certificate; DGS: Directorate-General of Health.
Figure 2
Figure 2
Comparison between performance metrics of AUTOCOD during periods of excess mortality, severe excess mortality, and extreme excess mortality and periods without excess mortality for International Statistical Classification of Diseases and Health-Related Problems, 10th Revision (ICD-10), chapters. DGS: Directorate-General of Health; SICO: Death Certificate Information System.
Figure 3
Figure 3
Comparison between performance metrics of AUTOCOD during periods of excess mortality and periods without excess mortality for International Statistical Classification of Diseases and Health-Related Problems, 10th Revision (ICD-10), blocks. DGS: Directorate-General of Health; SICO: Death Certificate Information System.
See this image and copyright information in PMC

Similar articles

See all similar articles

References

    1. Graphs and maps. EuroMOMO. [2023-04-13]. https://www.euromomo.eu/graphs-and-maps/
    1. Vestergaard LS, Nielsen J, Richter L, Schmid D, Bustos N, Braeye T, Denissov G, Veideman T, Luomala O, Möttönen T, Fouillet A, Caserio-Schönemann C, An der Heiden M, Uphoff H, Lytras T, Gkolfinopoulou K, Paldy A, Domegan L, O'Donnell J, De' Donato F, Noccioli F, Hoffmann P, Velez T, England K, van Asten L, White RA, Tønnessen R, da Silva SP, Rodrigues AP, Larrauri A, Delgado-Sanz C, Farah A, Galanis I, Junker C, Perisa D, Sinnathamby M, Andrews N, O'Doherty M, Marquess DF, Kennedy S, Olsen SJ, Pebody R, Krause TG, Mølbak K. Excess all-cause mortality during the COVID-19 pandemic in Europe - preliminary pooled estimates from the EuroMOMO network, March to April 2020. Euro Surveill. 2020 Jul;25(26):2001214. doi: 10.2807/1560-7917.ES.2020.25.26.2001214. http://www.eurosurveillance.org/content/10.2807/1560-7917.ES.2020.25.26.... - DOI - DOI - PMC - PubMed
    1. Pinto CS, Anderson RN, Martins H, Marques C, Maia C, Borralho MC. Mortality Information System in Portugal: transition to e-death certification. Eurohealth (Lond) 2016;22(2):1–53. https://europepmc.org/abstract/MED/32336930 - PMC - PubMed
    1. SICO - eVM. Vigilância de Mortalidade. [2021-11-08]. https://evm.min-saude.pt/#shiny-tab-info_eVM .
    1. International Statistical Classification of Diseases and Health-Related Problems, 10th Revision, 5th Edition, 2016. World Health Organization. 2015. [2021-11-09]. https://apps.who.int/iris/handle/10665/246208 .

LinkOut - more resources