Precision-Optimised Post-Stroke Prognoses

Abstract

Background: Current medicine cannot confidently predict who will recover from post-stroke impairments. Researchers have sought to bridge this gap by treating the post-stroke prognostic problem as a machine learning problem, reporting prediction error metrics across samples of patients whose outcomes are known. This approach effectively shares prediction error equally among the patients, which is contrary to the long-held clinical intuition that some patients' outcomes are more predictable than other patients' outcomes. Here, we test that intuition empirically, by asking whether those 'more predictable' patients can be identified before their outcomes are known.

Methods: Drawing on lesion location and demographic data, we use ensemble classifiers to predict the presence of a variety of different language impairments in a large sample of stroke patients. We tune these models to maximise their Positive Predictive Value (or precision): that is, the probability that patients assigned to a class are really members of that class. We test whether those tuned models have high precision on independent data.

Results: Precision-tuned models might only classify a subset of patients, but for that reduced set, the classifications are very likely to be correct: typically > 90% and sometimes > 95%. Small reductions of target precision could rapidly raise the proportion of patients for whom 'high enough precision' predictions can be made.

Conclusions: High precision prognoses are possible when predicting language outcomes after stroke. Providing such predictions for subsets of patients might be a reasonable intermediate step on the way to providing them for all.

Keywords: cognition; confidence; language; lesions; machine learning; stroke.

FIGURE 1

A schematic illustration of the data pre‐processing and analysis procedure. After selecting the sample, we extract MRI data, non‐brain demographic variables, and language outcome scores for the selected patients. Tables 1 and 2 report the results of the outer loop cross validation, emphasising those patients for whom ‘high precision’ classifications were made.

FIGURE 2

Average validation set classification performance versus training set confidence. For both NPV (top left) and PPV (top right), the validation set values are generally lower than the training set values indicating some absolute loss of performance. Nevertheless, validation set NPV and PPV are both high in the best case. The bottom row shows the proportions of the correct class that receive each type of ‘high confidence’ prediction. Note that when the predictions are made with the most permissive NPV and PPV thresholds, these proportions are often > 100% because these predictions are also being made for a fraction of the sample that compose the other (wrong) class. Notably, the functions relating validation set coverage to training set PPV are convex (bottom row). This suggests that the coverage for these predictions would increase rapidly with only a comparatively minor reduction in threshold precision.