JURaSSiC: accuracy of clinician vs risk score prediction of ischemic stroke outcomes

Abstract

Objective: We compared the accuracy of clinicians and a risk score (iScore) to predict observed outcomes following an acute ischemic stroke.

Methods: The JURaSSiC (Clinician JUdgment vs Risk Score to predict Stroke outComes) study assigned 111 clinicians with expertise in acute stroke care to predict the probability of outcomes of 5 ischemic stroke case scenarios. Cases (n = 1,415) were selected as being representative of the 10 most common clinical presentations from a pool of more than 12,000 stroke patients admitted to 12 stroke centers. The primary outcome was prediction of death or disability (modified Rankin Scale [mRS] ≥3) at discharge within the 95% confidence interval (CI) of observed outcomes. Secondary outcomes included 30-day mortality and death or institutionalization at discharge.

Results: Clinicians made 1,661 predictions with overall accuracy of 16.9% for death or disability at discharge, 46.9% for 30-day mortality, and 33.1% for death or institutionalization at discharge. In contrast, 90% of the iScore-based estimates were within the 95% CI of observed outcomes. Nearly half (n = 53 of 111; 48%) of participants were unable to accurately predict the probability of the primary outcome in any of the 5 rated cases. Less than 1% (n = 1) provided accurate predictions in 4 of the 5 cases and none accurately predicted all 5 case outcomes. In multivariable analyses, the presence of patient characteristics associated with poor outcomes (mRS ≥3 or death) in previous studies (older age, high NIH Stroke Scale score, and nonlacunar subtype) were associated with more accurate clinician predictions of death at 30 days (odds ratio [OR] 2.40, 95% CI 1.57-3.67) and with a trend for more accurate predictions of death or disability at discharge (OR 1.85, 95% CI 0.99-3.46).

Conclusions: Clinicians with expertise in stroke performed poorly compared to a validated tool in predicting the outcomes of patients with an acute ischemic stroke. Use of the risk stroke outcome tool may be superior for decision-making following an acute ischemic stroke.

Figure 1. Scatterplot of clinicians' estimations, iScore predictions, and actual outcomes

Scatterplot of observed outcomes (95% confidence interval [CI]) (red triangle), the iScore (blue X), and clinicians' (black dot) estimations by each stroke case for the outcomes of interest. (A) Death or disability. (B) The 30-day mortality. (C) Institutionalization or death. Each black dot represents the individual clinician's estimation. Multiple black dots horizontally aligned represent more clinicians providing similar estimations. Note the wide range of clinicians' estimations by case, with the great majority outside of the 95% CI of the observed outcome.

Figure 2. Forest plot representing variables associated with clinicians' accuracy

Forest plot representing variables associated with clinicians' accuracy for outcomes of interest. (A) Death or disability (modified Rankin Scale [mRS] 3–5) at discharge. (B) Death at 30 days. (C) Institutionalization or death. Note that high risk cases were associated with clinicians' outcome predictions. Further details are explained in the text. CI = confidence interval; OR = odds ratio.