Synthetic Model Combination: A new machine-learning method for pharmacometric model ensembling

Abstract

When aiming to make predictions over targets in the pharmacological setting, a data-focused approach aims to learn models based on a collection of labeled examples. Unfortunately, data sharing is not always possible, and this can result in many different models trained on disparate populations, leading to the natural question of how best to use and combine them when making a new prediction. Previous work has focused on global model selection or ensembling, with the result of a single final model across the feature space. Machine-learning models perform notoriously poorly on data outside their training domain, however, due to a problem known as covariate shift, and so we argue that when ensembling models the weightings for individual instances must reflect their respective domains-in other words, models that are more likely to have seen information on that instance should have more attention paid to them. We introduce a method for such an instance-wise ensembling of models called Synthetic Model Combination (SMC), including a novel representation learning step for handling sparse high-dimensional domains. We demonstrate the use of SMC on an example with dosing predictions for vancomycin, although emphasize the applicability of the method to any scenario involving the use of multiple models.

FIGURE 1

Instance‐wise ensembles. (a) Here, we represent the density of the training features for three separate models: M1, M2, and M3. Given new test points A, B, and C, we need to construct predictions from these models. A is well‐represented by both M2 and M3, whereas B only has significant density under M3. C looks like none of the models will be able to make confident predictions. (b) Different models are useful for new patients. Population pharmacokinetic models are often trained on certain demographic groups given the studies that are designed for data collection. For a new patient who does not necessarily fit into one of the existing demographics, different models may be more or less relevant and accurate. Naive ensembles ignore this fact and always incorporate evenly the predictions of each model, SMC on the other hand aims to up‐weight the models that would appear to be more relevant. SMC, Synthetic Model Combination.

FIGURE 2

Synthetic Model Combination training algorithm. (a) Algorithm outlining the main steps in training for SMC. (b) Inference flow diagram. As a new case comes in, the first step is to calculate all of the individual models' predictions, using NONMEM, for example. Then, like with any model averaging algorithm the weights must be calculated. Performance‐based model averaging methods have a set of weights independent of the new case, whereas SMC maps the new case features to a latent space that is then used to calculate individual weights for that case. BMA, Bayesian Model Averaging; SMC, Synthetic Model Combination.

FIGURE 3

Methods based on varying information. A selection of methods from the spectrum of information available to a practitioner. SMC lies quite far toward the little information end, aiming to only take use of some demographic information from each of the models and not require any labeled training points. SMC, Synthetic Model Combination.