Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Jul;3(7):e425-e433.
doi: 10.1016/S2589-7500(21)00080-7. Epub 2021 May 25.

An external validation of the QCovid risk prediction algorithm for risk of mortality from COVID-19 in adults: a national validation cohort study in England

Vahé Nafilyan  1 , Ben Humberstone  2 , Nisha Mehta  3 , Ian Diamond  2 , Carol Coupland  4 , Luke Lorenzi  2 , Piotr Pawelek  2 , Ryan Schofield  2 , Jasper Morgan  2 , Paul Brown  2 , Ronan Lyons  5 , Aziz Sheikh  6 , Julia Hippisley-Cox  7
Affiliations

An external validation of the QCovid risk prediction algorithm for risk of mortality from COVID-19 in adults: a national validation cohort study in England

Vahé Nafilyan et al. Lancet Digit Health. 2021 Jul.

Abstract

Background: Public policy measures and clinical risk assessments relevant to COVID-19 need to be aided by risk prediction models that are rigorously developed and validated. We aimed to externally validate a risk prediction algorithm (QCovid) to estimate mortality outcomes from COVID-19 in adults in England.

Methods: We did a population-based cohort study using the UK Office for National Statistics Public Health Linked Data Asset, a cohort of individuals aged 19-100 years, based on the 2011 census and linked to Hospital Episode Statistics, the General Practice Extraction Service data for pandemic planning and research, and radiotherapy and systemic chemotherapy records. The primary outcome was time to COVID-19 death, defined as confirmed or suspected COVID-19 death as per death certification. Two periods were used: (1) Jan 24 to April 30, 2020, and (2) May 1 to July 28, 2020. We assessed the performance of the QCovid algorithms using measures of discrimination and calibration. Using predicted 90-day risk of COVID-19 death, we calculated r2 values, Brier scores, and measures of discrimination and calibration with corresponding 95% CIs over the two time periods.

Findings: We included 34 897 648 adults aged 19-100 years resident in England. 26 985 (0·08%) COVID-19 deaths occurred during the first period and 13 177 (0·04%) during the second. The algorithms had good discrimination and calibration in both periods. In the first period, they explained 77·1% (95% CI 76·9-77·4) of the variation in time to death in men and 76·3% (76·0-76·6) in women. The D statistic was 3·761 (3·732-3·789) for men and 3·671 (3·640-3·702) for women and Harrell's C was 0·935 (0·933-0·937) for men and 0·945 (0·943-0·947) for women. Similar results were obtained for the second time period. In the top 5% of patients with the highest predicted risks of death, the sensitivity for identifying deaths in the first period was 65·94% for men and 71·67% for women.

Interpretation: The QCovid population-based risk algorithm performed well, showing high levels of discrimination for COVID-19 deaths in men and women for both time periods. QCovid has the potential to be dynamically updated as the pandemic evolves and, therefore, has potential use in guiding national policy.

Funding: UK National Institute for Health Research.

PubMed Disclaimer

Conflict of interest statement

Declaration of interests JH-C reports grants from the National Institute for Health Research Biomedical Research Centre, Oxford, UK; John Fell Oxford University Press Research Fund; Cancer Research UK, through the Cancer Research UK Oxford Centre; and the Oxford Wellcome Institutional Strategic Support Fund, during the conduct of the study. JH-C is an unpaid director of QResearch, a not-for-profit organisation that is a partnership between the University of Oxford, Oxford, UK, and EMIS Health, who supplied the QResearch database used for this work. JH-C is a founder and shareholder of ClinRisk and was the company's medical director until May 31, 2019. ClinRisk produces open and closed source software to implement clinical risk algorithms (outside this work) into clinical computer systems. All other authors declare no competing interests.

Figures

Figure 1
Figure 1
Harrell's C statistic by age group for men and women in the first period and second period (A) Results for the first period (Jan 24, to April 30, 2020). (B) Results for the second period (May 1, to July 28, 2020). Bars represent 95% CIs.
Figure 2
Figure 2
Predicted and observed risk of COVID-19-related death in the first study period First study period was Jan 24, to April 30, 2020.
Figure 3
Figure 3
Sensitivity for COVID-19-related death in the validation cohort for the first and second study periods The first study period was Jan 24, to April 30, 2020, and the second study period was May 1, to July 28, 2020. Centiles were based on predicted absolute risks in men and women in each period. Sensitivity (cumulative percentage of deaths) is percentage of total deaths in the period that occurred within the group of patients above the predicted risk threshold.
See this image and copyright information in PMC

References

    1. Zhou F, Yu T, Du R. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. Lancet. 2020;395:1054–1062. - PMC - PubMed
    1. Yancy CW. COVID-19 and African Americans. JAMA. 2020;323:1891–1892. - PubMed
    1. Chen T, Wu DI, Chen H. Clinical characteristics of 113 deceased patients with coronavirus disease 2019: retrospective study. BMJ. 2020;368 - PMC - PubMed
    1. Weiss P, Murdoch DR. Clinical course and mortality risk of severe COVID-19. Lancet. 2020;395:1014–1015. - PMC - PubMed
    1. Wadhera RK, Wadhera P, Gaba P. Variation in COVID-19 hospitalizations and deaths across New York City boroughs. JAMA. 2020;323:2192–2195. - PMC - PubMed

Uncited References

    1. Williamson EJ, Walker AJ, Bhaskaran K. OpenSAFELY: factors associated with COVID-19 death in 17 million patients. Nature. 2020;584:430–436. - PMC - PubMed
    1. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162:55–63. - PubMed
    1. Benchimol EI, Smeeth L, Guttmann A. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med. 2015;12 - PMC - PubMed
    1. Hippisley-Cox J, Clift AK, Coupland CAC. Protocol for the development and evaluation of a tool for predicting risk of short-term adverse outcomes due to COVID-19 in the general UK population. medRxiv. 2020 doi: 10.1101/2020.06.28.20141986. published online June 29. (preprint). - DOI

Publication types