Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Jun 18;20(1):111.
doi: 10.1186/s12911-020-01144-8.

Early warning score validation methodologies and performance metrics: a systematic review

Andrew Hao Sen Fang  1 , Wan Tin Lim  2 , Tharmmambal Balakrishnan  2
Affiliations

Early warning score validation methodologies and performance metrics: a systematic review

Andrew Hao Sen Fang et al. BMC Med Inform Decis Mak. .

Erratum in

Abstract

Background: Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation.

Methods: A systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity.

Results: The key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies.

Conclusions: Methodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.

Keywords: Early warning score; Methodology; Validation.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no conflict of interest. The authors would like to highlight that they included a paper they wrote in this review.

Figures

Fig. 1
Fig. 1
Flow chart describing inclusion of articles for full-text review from search result list
Fig. 2
Fig. 2
Summary of studies with various combinations of outcomes
Fig. 3
Fig. 3
Illustration of how different case definition affect EWS validation for 2 patients
See this image and copyright information in PMC

References

    1. DeVita MA, Hillman K. Why RRS? Where RRS? Crit Care Clin. 2018 Apr;34(2):xi–xii. - PubMed
    1. Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PW. The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review. Resuscitation. 2014;85(5):587–94. 10.1016/j.resuscitation.2014.01.013. - PubMed
    1. Douw G, Huisman-de Waal G, van Zanten AR, van der Hoeven JG, Schoonhoven L. Nurses' 'worry' as predictor of deteriorating surgical ward patients: A prospective cohort study of the Dutch-Early-Nurse-Worry-Indicator-Score. Int J Nurs Stud. 2016;59:134–40. 10.1016/j.ijnurstu.2016.04.006. - PubMed
    1. Morgan RJM, Williams F, Wright MM. An early warning scoring system for detecting developing critical illness. Clin Intensive Care. 1997;8:100.
    1. National Institute for Health and Clinical Excellence. Acute ill patients in hospital: recognition of and response to acute illness in adults in hospital. In: NICE clinical guideline No. 50. London; 2007. - PubMed

Publication types

LinkOut - more resources