Hierarchical resampling for bagging in multistudy prediction with applications to human neurochemical sensing

Abstract

We propose the "study strap ensemble", which combines advantages of two common approaches to fitting prediction models when multiple training datasets ("studies") are available: pooling studies and fitting one model versus averaging predictions from multiple models each fit to individual studies. The study strap ensemble fits models to bootstrapped datasets, or "pseudo-studies." These are generated by resampling from multiple studies with a hierarchical resampling scheme that generalizes the randomized cluster bootstrap. The study strap is controlled by a tuning parameter that determines the proportion of observations to draw from each study. When the parameter is set to its lowest value, each pseudo-study is resampled from only a single study. When it is high, the study strap ignores the multi-study structure and generates pseudo-studies by merging the datasets and drawing observations like a standard bootstrap. We empirically show the optimal tuning value often lies in between, and prove that special cases of the study strap draw the merged dataset and the set of original studies as pseudo-studies. We extend the study strap approach with an ensemble weighting scheme that utilizes information in the distribution of the covariates of the test dataset. Our work is motivated by neuroscience experiments using real-time neurochemical sensing during awake behavior in humans. Current techniques to perform this kind of research require measurements from an electrode placed in the brain during awake neurosurgery and rely on prediction models to estimate neurotransmitter concentrations from the electrical measurements recorded by the electrode. These models are trained by combining multiple datasets that are collected in vitro under heterogeneous conditions in order to promote accuracy of the models when applied to data collected in the brain. A prevailing challenge is deciding how to combine studies or ensemble models trained on different studies to enhance model generalizability. Our methods produce marked improvements in simulations and in this application. All methods are available in the studyStrap CRAN package.

Keywords: Domain Adaptation; Domain Generalization; Neuroscience; Primary 62P10; Transfer Learning.

Fig 1:

Illustration of the effect of the bag size tuning parameter b on the resulting pseudo-studies. This example assumes K = 10, and observed study sample sizes: n₁ = ... = n₁₀ = 1000. We show four example pseudo-studies. The black rectangles show the bag size, b, and example study bags. The remainder of the column is the study strap replicate, with gray blocks denoting a pseudo-study. $n_j^{(r)}$ is the number of observations resampled from the j^th observed study in the r^th pseudo-study.

Fig 2:

Simulation results for different levels of between-study heterogeneity in true model coefficients, ${\sigma }_{\mathbf{\beta }}^2$ (varies across columns), and distribution of covariates, ${\sigma }_X^2$ (varies across rows). Each observation in a plot is the log ratio of the out-of-study-RMSE from a single test study (from the corresponding method) to the out-of-study-RMSE of the TOM algorithm (RMSE_TOM). Each box plot is comprised of 100 iterations.

Fig 3:

Average test performance as a function of bag size. Points are connected with lines for clarity. The vertical scale is the log of the RMSE in the test study, and divided by the corresponding RMSE of the TOM. Horizontal line indicates performance of the TOM. (a) AR with average weights. Vertical lines indicate the optimal bag size which increases with ${\sigma }_X^2$. (b) SSE with stacking weights achieves optimal performance at intermediate values of bag size for larger values of ${\sigma }_X^2$.

Fig 4:

Average covariate profiles. Each curve corresponds to a study (electrode) and is colored by the overall average current. Studies exhibit both variation and clustering in average current.

Fig 5:

Predictive performance of methods on data using raw covariates (left) and the derivative (right). Dotted line indicates relative performance of the TOM algorithm (Mean raw: RMSE_TOM = 416.53; Mean derivative: RMSE_TOM = 359.64).

Fig 6:

Average Accept/Reject performance on test set as a function of bag size. Derivative shifts the optimal bag size to higher values. Vertical lines indicate optimal bag size. RMSEs are standardized to the RMSE of the TOM algorithm.