Development and validation of risk prediction models for COVID-19 positivity in a hospital setting

Abstract

Objectives: To develop: (1) two validated risk prediction models for coronavirus disease-2019 (COVID-19) positivity using readily available parameters in a general hospital setting; (2) nomograms and probabilities to allow clinical utilisation.

Methods: Patients with and without COVID-19 were included from 4 Hong Kong hospitals. The database was randomly split into 2:1: for model development database (n = 895) and validation database (n = 435). Multivariable logistic regression was utilised for model creation and validated with the Hosmer-Lemeshow (H-L) test and calibration plot. Nomograms and probabilities set at 0.1, 0.2, 0.4 and 0.6 were calculated to determine sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV).

Results: A total of 1330 patients (mean age 58.2 ± 24.5 years; 50.7% males; 296 COVID-19 positive) were recruited. The first prediction model developed had age, total white blood cell count, chest x-ray appearances and contact history as significant predictors (AUC = 0.911 [CI = 0.880-0.941]). The second model developed has the same variables except contact history (AUC = 0.880 [CI = 0.844-0.916]). Both were externally validated on the H-L test (p = 0.781 and 0.155, respectively) and calibration plot. Models were converted to nomograms. Lower probabilities give higher sensitivity and NPV; higher probabilities give higher specificity and PPV.

Conclusion: Two simple-to-use validated nomograms were developed with excellent AUCs based on readily available parameters and can be considered for clinical utilisation.

Keywords: COVID-19; Chest x-ray; Nomogram; Prediction model; White cell count.

Figure 1

Consort diagram showing the process of identification of cases and exclusion of cases.

Figure 2

Calibration plot for the Overall Cohort Model and Unknown Contact History Model for the risk of COVID-19.

Figure 3

Nomogram of the Overall Cohort in Table 2. A total score is calculated from the addition of the scores for the variables chest x-ray (CXR) consolidation/ground glass opacity (GGO), contact history, white cell count and age. Note that age has two steps, whilst other variables have only 1 step. The total score can then be marked on the bottom row and compared with the probability scale above. For example, a patient suspected to have COVID-19 aged 50 (step 1: allocate 2 points and step 2: allocate 7.5 points), has no contact history (step 3: allocate 0 points), total white cell count (WCC) of 2 × 10⁹ cells/L (step 4: allocate 7 points) and a CXR with no consolidation/GGO and absent pleural effusion (PE) (step 5: allocated 0 points), would receive a total score of 16.5 points, which equates to a probability of between 0.6 and 0.7. A clinician then refers to the probability table (Table 3) and decides what degree of sensitivity, specificity, positive predictive value or negative predictive value is adequate for their setting.

Figure 4

Nomogram of the Overall Cohort in Table 4. A total score is calculated from the addition of the scores for the variables pleural effusion, chest x-ray (CXR) consolidation/ground glass opacity (GGO), white cell count, age and vomiting symptom. Note that age has two steps, whilst other variables have only 1 step. The total score can then be marked on the bottom row and compared with the probability scale above. For example, a patient suspected to have COVID-19 aged 40 (step 1: allocate 2 points and step 2: allocate 8.5 points), total white cell count (WCC) of 8 × 10⁹ cells/L (step 3: allocate 3 points) and a CXR with consolidation/GGO and absent pleural effusion (PE) (step 4: allocated 1.5 points), would receive a total score of 15 points which equates to a probability of between 0.6 and 0.7. A clinician then refers to the probability table (Table 3) and decides what degree of sensitivity, specificity, positive predictive value or negative predictive value is adequate for their setting.