Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Dec:11:100497.
doi: 10.1016/j.ejro.2023.100497. Epub 2023 Jun 19.

Performance of an AI algorithm during the different phases of the COVID pandemics: what can we learn from the AI and vice versa

Michele Catalano  1 , Chandra Bortolotto  1 , Giovanna Nicora  2 , Marina Francesca Achilli  1 , Alessio Consonni  1 , Lidia Ruongo  1 , Giovanni Callea  1 , Antonio Lo Tito  1 , Carla Biasibetti  1 , Antonella Donatelli  1 , Sara Cutti  3 , Federico Comotto  4 , Giulia Maria Stella  5 , Angelo Corsico  5 , Stefano Perlini  6 , Riccardo Bellazzi  2 , Raffaele Bruno  7 , Andrea Filippi  8 , Lorenzo Preda  1
Affiliations

Performance of an AI algorithm during the different phases of the COVID pandemics: what can we learn from the AI and vice versa

Michele Catalano et al. Eur J Radiol Open. 2023 Dec.

Abstract

Background: Artificial intelligence (AI) has proved to be of great value in diagnosing and managing Sars-Cov-2 infection. ALFABETO (ALL-FAster-BEtter-TOgether) is a tool created to support healthcare professionals in the triage, mainly in optimizing hospital admissions.

Methods: The AI was trained during the pandemic's "first wave" (February-April 2020). Our aim was to assess the performance during the "third wave" of the pandemics (February-April 2021) and evaluate its evolution. The neural network proposed behavior (hospitalization vs home care) was compared with what was actually done. If there were discrepancies between ALFABETO's predictions and clinicians' decisions, the disease's progression was monitored. Clinical course was defined as "favorable/mild" if patients could be managed at home or in spoke centers and "unfavorable/severe" if patients need to be managed in a hub center.

Results: ALFABETO showed accuracy of 76%, AUROC of 83%; specificity was 78% and recall 74%. ALFABETO also showed high precision (88%). 81 hospitalized patients were incorrectly predicted to be in "home care" class. Among those "home-cared" by the AI and "hospitalized" by the clinicians, 3 out of 4 misclassified patients (76.5%) showed a favorable/mild clinical course. ALFABETO's performance matched the reports in literature.

Conclusions: The discrepancies mostly occurred when the AI predicted patients could stay at home but clinicians hospitalized them; these cases could be handled in spoke centers rather than hubs, and the discrepancies may aid clinicians in patient selection. The interaction between AI and human experience has the potential to improve both AI performance and our comprehension of pandemic management.

Keywords: Artificial intelligence; COVID 19; Machine learning; Resource allocation; X-rays.

PubMed Disclaimer

Conflict of interest statement

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Preda Prof Lorenzo reports financial support was provided by Lombardy Region.

Figures

Fig. 1
Fig. 1
A Conceptual view of ALFABETO Model. Clinical data and deep learning features extracted from CXR are provided as input for a machine learning classifier, which then predicts whether a specific patient should be hospitalized or not.
Fig. 2
Fig. 2
Prognosis of patients during the third wave. Each pie chart refers to a specific group of patients, identified by the true outcome and ALFABETO predicted outcome. For instance, the lower right pie chart reports the percentage of hospitalized patients correctly predicted as “Hospital” by ALFABETO. The different slices are proportional to the percentage of patients with mild, severe or moderate prognosis in each group.
See this image and copyright information in PMC

References

    1. Zhou P., Yang X.L., Wang X.G., et al. A pneumonia outbreak associated with a new coronavirus of probable bat origin. Nature. 2020;579(7798):270–273. doi: 10.1038/s41586-020-2012-7. - DOI - PMC - PubMed
    1. Mallah S.I., Ghorab O.K., Al-Salmi S., et al. COVID-19: breaking down a global health crisis. Ann. Clin. Microbiol Antimicrob. 2021;20(1):35. doi: 10.1186/s12941-021-00438-7. - DOI - PMC - PubMed
    1. Alimadadi A., Aryal S., Manandhar I., Munroe P.B., Joe B., Cheng X. Artificial intelligence and machine learning to fight COVID-19. Physiol. Genom. 2020;52(4):200–202. doi: 10.1152/physiolgenomics.00029.2020. - DOI - PMC - PubMed
    1. Brat G.A., Weber G.M., Gehlenborg N., et al. International electronic health record-derived COVID-19 clinical course profiles: the 4CE consortium. NPJ Digit Med. 2020;3:109. doi: 10.1038/s41746-020-00308-0. - DOI - PMC - PubMed
    1. Subudhi S., Verma A., Patel A.B., et al. Comparing machine learning algorithms for predicting ICU admission and mortality in COVID-19. NPJ Digit Med. 2020;4(1):87. doi: 10.1038/s41746-021-00456-x. May 21. - DOI - PMC - PubMed