Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients

doi:10.1007/s00330-022-08969-z

. 2023 Jan;33(1):23-33.

doi: 10.1007/s00330-022-08969-z. Epub 2022 Jul 2.

Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients

Michael D Kuo^{1

2}, Keith W H Chiu³, David S Wang⁴, Anna Rita Larici^{5

6}, Dmytro Poplavskiy⁷, Adele Valentini⁸, Alessandro Napoli⁹, Andrea Borghesi¹⁰, Guido Ligabue^{11

12}, Xin Hao B Fang¹³, Hing Ki C Wong¹⁴, Sailong Zhang³, John R Hunter⁴, Abeer Mousa¹⁵, Amato Infante^{6

16}, Lorenzo Elia^{5

6}, Salvatore Golemi¹⁰, Leung Ho P Yu¹⁷, Christopher K M Hui^{18

19}, Bradley J Erickson¹⁵

Affiliations

¹ Medical Artificial Intelligence Laboratory Program, Department of Diagnostic Radiology, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong SAR, China. mikedkuo@gmail.com.
² Ensemble Group Holdings, Ensemblehealth.ai, Scottsdale, AZ, USA. mikedkuo@gmail.com.
³ Medical Artificial Intelligence Laboratory Program, Department of Diagnostic Radiology, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong SAR, China.
⁴ Department of Radiology, Stanford Health Care, Stanford, CA, USA.
⁵ Section of Radiology, Department of Radiological and Hematological Sciences, Università Cattolica del Sacro Cuore, Rome, Italy.
⁶ Department of Diagnostic Imaging, Oncological Radiotherapy and Hematology, Fondazione Policlinico Universitario "A. Gemelli" IRCCS, Rome, Italy.
⁷ Ensemble Group Holdings, Ensemblehealth.ai, Scottsdale, AZ, USA.
⁸ Department of Radiology, Fondazione IRCCS Policlinico San Matteo, Pavia, Italy.
⁹ Department of Radiological, Oncological and Pathological Sciences, Sapienza University of Rome, Rome, Italy.
¹⁰ Department of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of Brescia, ASST Spedali Civili of Brescia, Brescia, Italy.
¹¹ Department of Medical and Surgical Sciences for Children & Adults, Modena and Reggio Emilia University, Modena, Italy.
¹² Division of Radiology, Azienda Ospedaliero-Universitaria Policlinico di Modena, Modena, Italy.
¹³ Radiology Department, Queen Mary Hospital, Hong Kong SAR, China.
¹⁴ Radiology Department, United Christian Hospital, Hong Kong SAR, China.
¹⁵ Radiology Department, Mayo Clinic, Rochester, MN, USA.
¹⁶ Columbus Covid 2 Hospital, Rome, Italy.
¹⁷ Department of Mathematics and Information Technology, The Education University of Hong Kong, Hong Kong SAR, China.
¹⁸ Department of Medicine, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong SAR, China.
¹⁹ Department of Respiratory & Critical Care Medicine, Matilda & War Memorial Hospital, Hong Kong SAR, China.

PMID: 35779089
DOI: 10.1007/s00330-022-08969-z

Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients

Michael D Kuo et al. Eur Radiol. 2023 Jan.

. 2023 Jan;33(1):23-33.

doi: 10.1007/s00330-022-08969-z. Epub 2022 Jul 2.

Authors

Affiliations

¹ Medical Artificial Intelligence Laboratory Program, Department of Diagnostic Radiology, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong SAR, China. mikedkuo@gmail.com.
² Ensemble Group Holdings, Ensemblehealth.ai, Scottsdale, AZ, USA. mikedkuo@gmail.com.
³ Medical Artificial Intelligence Laboratory Program, Department of Diagnostic Radiology, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong SAR, China.
⁴ Department of Radiology, Stanford Health Care, Stanford, CA, USA.
⁵ Section of Radiology, Department of Radiological and Hematological Sciences, Università Cattolica del Sacro Cuore, Rome, Italy.
⁶ Department of Diagnostic Imaging, Oncological Radiotherapy and Hematology, Fondazione Policlinico Universitario "A. Gemelli" IRCCS, Rome, Italy.
⁷ Ensemble Group Holdings, Ensemblehealth.ai, Scottsdale, AZ, USA.
⁸ Department of Radiology, Fondazione IRCCS Policlinico San Matteo, Pavia, Italy.
⁹ Department of Radiological, Oncological and Pathological Sciences, Sapienza University of Rome, Rome, Italy.
¹⁰ Department of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of Brescia, ASST Spedali Civili of Brescia, Brescia, Italy.
¹¹ Department of Medical and Surgical Sciences for Children & Adults, Modena and Reggio Emilia University, Modena, Italy.
¹² Division of Radiology, Azienda Ospedaliero-Universitaria Policlinico di Modena, Modena, Italy.
¹³ Radiology Department, Queen Mary Hospital, Hong Kong SAR, China.
¹⁴ Radiology Department, United Christian Hospital, Hong Kong SAR, China.
¹⁵ Radiology Department, Mayo Clinic, Rochester, MN, USA.
¹⁶ Columbus Covid 2 Hospital, Rome, Italy.
¹⁷ Department of Mathematics and Information Technology, The Education University of Hong Kong, Hong Kong SAR, China.
¹⁸ Department of Medicine, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong SAR, China.
¹⁹ Department of Respiratory & Critical Care Medicine, Matilda & War Memorial Hospital, Hong Kong SAR, China.

PMID: 35779089
DOI: 10.1007/s00330-022-08969-z

Abstract

Objectives: While chest radiograph (CXR) is the first-line imaging investigation in patients with respiratory symptoms, differentiating COVID-19 from other respiratory infections on CXR remains challenging. We developed and validated an AI system for COVID-19 detection on presenting CXR.

Methods: A deep learning model (RadGenX), trained on 168,850 CXRs, was validated on a large international test set of presenting CXRs of symptomatic patients from 9 study sites (US, Italy, and Hong Kong SAR) and 2 public datasets from the US and Europe. Performance was measured by area under the receiver operator characteristic curve (AUC). Bootstrapped simulations were performed to assess performance across a range of potential COVID-19 disease prevalence values (3.33 to 33.3%). Comparison against international radiologists was performed on an independent test set of 852 cases.

Results: RadGenX achieved an AUC of 0.89 on 4-fold cross-validation and an AUC of 0.79 (95%CI 0.78-0.80) on an independent test cohort of 5,894 patients. Delong's test showed statistical differences in model performance across patients from different regions (p < 0.01), disease severity (p < 0.001), gender (p < 0.001), and age (p = 0.03). Prevalence simulations showed the negative predictive value increases from 86.1% at 33.3% prevalence, to greater than 98.5% at any prevalence below 4.5%. Compared with radiologists, McNemar's test showed the model has higher sensitivity (p < 0.001) but lower specificity (p < 0.001).

Conclusion: An AI model that predicts COVID-19 infection on CXR in symptomatic patients was validated on a large international cohort providing valuable context on testing and performance expectations for AI systems that perform COVID-19 prediction on CXR.

Key points: • An AI model developed using CXRs to detect COVID-19 was validated in a large multi-center cohort of 5,894 patients from 9 prospectively recruited sites and 2 public datasets. • Differences in AI model performance were seen across region, disease severity, gender, and age. • Prevalence simulations on the international test set demonstrate the model's NPV is greater than 98.5% at any prevalence below 4.5%.

Keywords: Artificial intelligence; COVID-19; Public health; Radiology; Thoracic.

PubMed Disclaimer

Cited by

Clinical Implication and Prognostic Value of Artificial-Intelligence-Based Results of Chest Radiographs for Assessing Clinical Outcomes of COVID-19 Patients.
Shin HJ, Kim MH, Son NH, Han K, Kim EK, Kim YC, Park YS, Lee EH, Kyong T. Shin HJ, et al. Diagnostics (Basel). 2023 Jun 16;13(12):2090. doi: 10.3390/diagnostics13122090. Diagnostics (Basel). 2023. PMID: 37370985 Free PMC article.
Artificial Intelligence in the Intensive Care Unit: Present and Future in the COVID-19 Era.
Kołodziejczak MM, Sierakowska K, Tkachenko Y, Kowalski P. Kołodziejczak MM, et al. J Pers Med. 2023 May 25;13(6):891. doi: 10.3390/jpm13060891. J Pers Med. 2023. PMID: 37373880 Free PMC article. Review.
FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare.
Lekadir K, Frangi AF, Porras AR, Glocker B, Cintas C, Langlotz CP, Weicken E, Asselbergs FW, Prior F, Collins GS, Kaissis G, Tsakou G, Buvat I, Kalpathy-Cramer J, Mongan J, Schnabel JA, Kushibar K, Riklund K, Marias K, Amugongo LM, Fromont LA, Maier-Hein L, Cerdá-Alberich L, Martí-Bonmatí L, Cardoso MJ, Bobowicz M, Shabani M, Tsiknakis M, Zuluaga MA, Fritzsche MC, Camacho M, Linguraru MG, Wenzel M, De Bruijne M, Tolsgaard MG, Goisauf M, Cano Abadía M, Papanikolaou N, Lazrak N, Pujol O, Osuala R, Napel S, Colantonio S, Joshi S, Klein S, Aussó S, Rogers WA, Salahuddin Z, Starmans MPA; FUTURE-AI Consortium. Lekadir K, et al. BMJ. 2025 Feb 5;388:e081554. doi: 10.1136/bmj-2024-081554. BMJ. 2025. PMID: 39909534 Free PMC article.
CT-based Assessment at 6-Month Follow-up of COVID-19 Pneumonia patients in China.
Fang X, Lv Y, Lv W, Liu L, Feng Y, Liu L, Pan F, Zhang Y. Fang X, et al. Sci Rep. 2024 Feb 29;14(1):5028. doi: 10.1038/s41598-024-54920-1. Sci Rep. 2024. PMID: 38424447 Free PMC article.
Generalizable disease detection using model ensemble on chest X-ray images.
Abad M, Casas-Roma J, Prados F. Abad M, et al. Sci Rep. 2024 Mar 11;14(1):5890. doi: 10.1038/s41598-024-56171-6. Sci Rep. 2024. PMID: 38467705 Free PMC article.

References

1. Gottlieb RL, Vaca CE, Paredes R et al (2021) Early remdesivir to prevent progression to severe Covid-19 in outpatients. N Engl J Med. https://doi.org/10.1056/NEJMoa2116846
1. Kucharski AJ, Klepac P, Conlan AJK et al (2020) Effectiveness of isolation, testing, contact tracing, and physical distancing on reducing transmission of SARS-CoV-2 in different settings: a mathematical modelling study. Lancet Infect Dis 20:1151–1160 - DOI
1. Dryden-Peterson S, Velásquez GE, Stopka TJ, Davey S, Lockman S, Ojikutu BO (2021) Disparities in SARS-CoV-2 testing in Massachusetts during the COVID-19 pandemic. JAMA Netw Open 4:e2037067 - DOI
1. Quilty BJ, Clifford S, Hellewell J et al (2021) Quarantine and testing strategies in contact tracing for SARS-CoV-2: a modelling study. Lancet Public Health 6:e175–e183 - DOI
1. Mina MJ, Parker R, Larremore DB (2020) Rethinking Covid-19 test sensitivity — a strategy for containment. N Engl J Med 383:e120 - DOI

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
- Springer
Medical
- MedlinePlus Health Information
Miscellaneous
- NCI CPTAC Assay Portal

[1] Gottlieb RL, Vaca CE, Paredes R et al (2021) Early remdesivir to prevent progression to severe Covid-19 in outpatients. N Engl J Med. https://doi.org/10.1056/NEJMoa2116846

[2] Gottlieb RL, Vaca CE, Paredes R et al (2021) Early remdesivir to prevent progression to severe Covid-19 in outpatients. N Engl J Med. https://doi.org/10.1056/NEJMoa2116846

[3] Kucharski AJ, Klepac P, Conlan AJK et al (2020) Effectiveness of isolation, testing, contact tracing, and physical distancing on reducing transmission of SARS-CoV-2 in different settings: a mathematical modelling study. Lancet Infect Dis 20:1151–1160 - DOI

[4] Kucharski AJ, Klepac P, Conlan AJK et al (2020) Effectiveness of isolation, testing, contact tracing, and physical distancing on reducing transmission of SARS-CoV-2 in different settings: a mathematical modelling study. Lancet Infect Dis 20:1151–1160 - DOI

[5] Dryden-Peterson S, Velásquez GE, Stopka TJ, Davey S, Lockman S, Ojikutu BO (2021) Disparities in SARS-CoV-2 testing in Massachusetts during the COVID-19 pandemic. JAMA Netw Open 4:e2037067 - DOI

[6] Dryden-Peterson S, Velásquez GE, Stopka TJ, Davey S, Lockman S, Ojikutu BO (2021) Disparities in SARS-CoV-2 testing in Massachusetts during the COVID-19 pandemic. JAMA Netw Open 4:e2037067 - DOI

[7] Quilty BJ, Clifford S, Hellewell J et al (2021) Quarantine and testing strategies in contact tracing for SARS-CoV-2: a modelling study. Lancet Public Health 6:e175–e183 - DOI

[8] Quilty BJ, Clifford S, Hellewell J et al (2021) Quarantine and testing strategies in contact tracing for SARS-CoV-2: a modelling study. Lancet Public Health 6:e175–e183 - DOI

[9] Mina MJ, Parker R, Larremore DB (2020) Rethinking Covid-19 test sensitivity — a strategy for containment. N Engl J Med 383:e120 - DOI

[10] Mina MJ, Parker R, Larremore DB (2020) Rethinking Covid-19 test sensitivity — a strategy for containment. N Engl J Med 383:e120 - DOI

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients

Affiliations

Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients

Authors

Affiliations

Abstract

Similar articles

Cited by

References

MeSH terms

LinkOut - more resources

Full Text Sources

Medical

Miscellaneous

Abstract

Similar articles

Cited by

References

MeSH terms

Related information

LinkOut - more resources

Full Text Sources

Medical

Miscellaneous