Training replicable predictors in multiple studies

Abstract

This article considers replicability of the performance of predictors across studies. We suggest a general approach to investigating this issue, based on ensembles of prediction models trained on different studies. We quantify how the common practice of training on a single study accounts in part for the observed challenges in replicability of prediction performance. We also investigate whether ensembles of predictors trained on multiple studies can be combined, using unique criteria, to design robust ensemble learners trained upfront to incorporate replicability into different contexts and populations.

Keywords: cross-study validation; ensemble learning; machine learning; replicability; validation.

Fig. 1.

The architecture of a CSL, illustrated with six studies divided into three subsets, two SSLs, and general weights.

Fig. 2.

Ratios of validation rms errors (rmses) to the rmse of the Reg-a weighting strategy, averaged over 100 simulation iterations, as we vary the coefficient perturbation window. Top seven panels correspond to different choices of SSL; the colors correspond to different weighting schemes. Bottom displays average validation rmse of the best-performing scheme (indicated with color) for each SSL (indicated by letter) at each perturbation window.

Fig. 3.

Differential discrimination of alternative classifiers. For each classifier we compute the hazard ratio associated with a change of one unit in the score vector, as evaluated in the validation datasets. The vertical scale is the ratio of this performance measure to that of the Reg-a CSL. Colors indicate classes of learning strategies: White is weighted CSLs with weights addressing cross-study prediction, purple is CSLs with fixed weights, orange is merging and meta-analysis, and blue is a SSL trained on the TCGA dataset. Horizontal lines are at $y=1$ and at median performance of CS-Avg.