Real-world evaluation of rapid and laboratory-free COVID-19 triage for emergency care: external validation and pilot deployment of artificial intelligence driven screening

Abstract

Background: Uncertainty in patients' COVID-19 status contributes to treatment delays, nosocomial transmission, and operational pressures in hospitals. However, the typical turnaround time for laboratory PCR remains 12-24 h and lateral flow devices (LFDs) have limited sensitivity. Previously, we have shown that artificial intelligence-driven triage (CURIAL-1.0) can provide rapid COVID-19 screening using clinical data routinely available within 1 h of arrival to hospital. Here, we aimed to improve the time from arrival to the emergency department to the availability of a result, do external and prospective validation, and deploy a novel laboratory-free screening tool in a UK emergency department.

Methods: We optimised our previous model, removing less informative predictors to improve generalisability and speed, developing the CURIAL-Lab model with vital signs and readily available blood tests (full blood count [FBC]; urea, creatinine, and electrolytes; liver function tests; and C-reactive protein) and the CURIAL-Rapide model with vital signs and FBC alone. Models were validated externally for emergency admissions to University Hospitals Birmingham, Bedfordshire Hospitals, and Portsmouth Hospitals University National Health Service (NHS) trusts, and prospectively at Oxford University Hospitals, by comparison with PCR testing. Next, we compared model performance directly against LFDs and evaluated a combined pathway that triaged patients who had either a positive CURIAL model result or a positive LFD to a COVID-19-suspected clinical area. Lastly, we deployed CURIAL-Rapide alongside an approved point-of-care FBC analyser to provide laboratory-free COVID-19 screening at the John Radcliffe Hospital (Oxford, UK). Our primary improvement outcome was time-to-result, and our performance measures were sensitivity, specificity, positive and negative predictive values, and area under receiver operating characteristic curve (AUROC).

Findings: 72 223 patients met eligibility criteria across the four validating hospital groups, in a total validation period spanning Dec 1, 2019, to March 31, 2021. CURIAL-Lab and CURIAL-Rapide performed consistently across trusts (AUROC range 0·858-0·881, 95% CI 0·838-0·912, for CURIAL-Lab and 0·836-0·854, 0·814-0·889, for CURIAL-Rapide), achieving highest sensitivity at Portsmouth Hospitals (84·1%, Wilson's 95% CI 82·5-85·7, for CURIAL-Lab and 83·5%, 81·8-85·1, for CURIAL-Rapide) at specificities of 71·3% (70·9-71·8) for CURIAL-Lab and 63·6% (63·1-64·1) for CURIAL-Rapide. When combined with LFDs, model predictions improved triage sensitivity from 56·9% (51·7-62·0) for LFDs alone to 85·6% with CURIAL-Lab (81·6-88·9; AUROC 0·925) and 88·2% with CURIAL-Rapide (84·4-91·1; AUROC 0·919), thereby reducing missed COVID-19 cases by 65% with CURIAL-Lab and 72% with CURIAL-Rapide. For the prospective deployment of CURIAL-Rapide, 520 patients were enrolled for point-of-care FBC analysis between Feb 18 and May 10, 2021, of whom 436 received confirmatory PCR testing and ten (2·3%) tested positive. Median time from arrival to a CURIAL-Rapide result was 45 min (IQR 32-64), 16 min (26·3%) sooner than with LFDs (61 min, 37-99; log-rank p<0·0001), and 6 h 52 min (90·2%) sooner than with PCR (7 h 37 min, 6 h 5 min to 15 h 39 min; p<0·0001). Classification performance was high, with sensitivity of 87·5% (95% CI 52·9-97·8), specificity of 85·4% (81·3-88·7), and negative predictive value of 99·7% (98·2-99·9). CURIAL-Rapide correctly excluded infection for 31 (58·5%) of 53 patients who were triaged by a physician to a COVID-19-suspected area but went on to test negative by PCR.

Interpretation: Our findings show the generalisability, performance, and real-world operational benefits of artificial intelligence-driven screening for COVID-19 over standard-of-care in emergency departments. CURIAL-Rapide provided rapid, laboratory-free screening when used with near-patient FBC analysis, and was able to reduce the number of patients who tested negative for COVID-19 but were triaged to COVID-19-suspected areas.

Funding: The Wellcome Trust, University of Oxford Medical and Life Sciences Translational Fund.

Figure 1

Overview of study design Overview shows the timeline of model development, evaluation, and deployment (A); successive elimination of less informative predictors from CURIAL-1.0 to optimise for generalisability (CURIAL-Lab) and result-time (CURIAL-Rapide; B); and a proposed novel rapid screening pathway for COVID-19 in emergency departments, which combines lateral flow device testing with artificial intelligence screening (C). Routine blood tests and vital signs recordings are done on arrival to the emergency department, either using rapid point-of-care haematology analysers (about 10 min; CURIAL-Rapide) or the existing laboratory pathway (about 1 h; CURIAL-Lab). Real-time algorithmic analysis allows early, high-confidence identification of patients who are negative for safe triage to COVID-19-free clinical areas. Patients with positive CURIAL results are admitted to enhanced precautions (amber) areas, pending confirmatory PCR. Patients testing positive with a lateral flow test are streamed directly to COVID-19 (red) clinical areas. Arrow thickness represents patient flow. ALT=alanine aminotransferase. APTT=activated partial thromboplastin time. CRP=C-reactive protein. eGFR=estimated glomerular filtration rate. INR=international normalised ratio. NHS=National Health Service. p50=pressure at which haemoglobin is 50% bound to oxygen. * CURIAL-Lab used data collected from routine blood tests (full blood count; urea, creatinine, and electrolytes; liver function tests; and C-reactive protein) and vital signs, whereas CURIAL-Rapide used data that could be collected at the patient's bedside (full blood count and vital signs).

Figure 2

Performance of CURIAL-1.0, CURIAL-Lab, and CURIAL-Rapide during external validation at three independent UK hospitals trusts All models were calibrated during training to achieve 90% sensitivity. Error bars show 95% CIs. Numerical results are shown in the appendix (pp 9–10). AUROC=area under receiver operating characteristic curve. NHS=National Health Service.

Figure 3

Performance characteristics of Innova SARS-CoV-2 LFD (A), CURIAL-Rapide and CURIAL-Lab (B) calibrated during training to a sensitivity of 80%, and combined clinical pathways (C) Combined clinical pathways consider either a positive CURIAL model (CURIAL-Rapide or CURIAL-Lab) result or a positive LFD test as a COVID-19 suspected case, at Oxford University Hospitals National Health Service Foundation Trust between Dec 23, 2020, and March 6, 2021. Error bars show 95% CIs. Numerical results are shown in the appendix (p 10). AUROC=area under receiver operating characteristic curve. LFD=antigen testing with lateral flow device. NPV=negative predictive value. PPV=positive predictive value.

Figure 4

Time-to-result from patient arrival in the emergency department (A) and performance against a PCR standard (B) (A) Kaplan-Meier plots of time-to-result in h from patient arrival in the emergency department for CURIAL-Rapide, Innova SARS-CoV-2 LFD, and PCR swabs tests, alongside number of results awaited, CURIAL-Rapide results that were available sooner than LFD testing (log rank test, p<0·0001), and PCR test results (p<0·0001). (B) Receiver operating characteristic curve showing performance of CURIAL-Rapide, clinical triage done by the first-attending physician, and Innova SARS-CoV-2 LFD, against a PCR reference standard. LFD=antigen testing with lateral flow device.