Evaluating the Effectiveness of Personalized Medicine With Software

Abstract

We present methodological advances in understanding the effectiveness of personalized medicine models and supply easy-to-use open-source software. Personalized medicine involves the systematic use of individual patient characteristics to determine which treatment option is most likely to result in a better average outcome for the patient. Why is personalized medicine not done more in practice? One of many reasons is because practitioners do not have any easy way to holistically evaluate whether their personalization procedure does better than the standard of care, termed improvement. Our software, "Personalized Treatment Evaluator" (the R package PTE), provides inference for improvement out-of-sample in many clinical scenarios. We also extend current methodology by allowing evaluation of improvement in the case where the endpoint is binary or survival. In the software, the practitioner inputs 1) data from a single-stage randomized trial with one continuous, incidence or survival endpoint and 2) an educated guess of a functional form of a model for the endpoint constructed from domain knowledge. The bootstrap is then employed on data unseen during model fitting to provide confidence intervals for the improvement for the average future patient (assuming future patients are similar to the patients in the trial). One may also test against a null scenario where the hypothesized personalization are not more useful than a standard of care. We demonstrate our method's promise on simulated data as well as on data from a randomized comparative trial investigating two treatments for depression.

Keywords: bootstrap; inference; personalized medicine; randomized comparative trial; statistical software; treatment regimes.

FIGURE 1

A graphical illustration of (1) our proposed method for estimation and (2) our proposed method for inference on the population mean improvement of an allocation procedure and (3) our proposed future allocation procedure (top left of the illustration). To compute the best estimate of the improvement ${\widehat{I}}_0$, the RCT data goes through the K-fold cross validation procedure of Section 3.4 (depicted in the top center). The black slices of the data frame represent the test data. To draw inference, we employ the non-parametric bootstrap procedure of Section 3.5 by sampling the RCT data with replacement and repeating the K-fold CV to produce ${\widehat{I}}_0^1,{\widehat{I}}_0^2,\dots ,{\widehat{I}}_0^B$ (bottom). The gray slices of the data frame represent the duplicate rows in the original data due to sampling with replacement. The confidence interval and significance of $H_0:{\mu }_{I_0}\le 0$ is computed from the bootstrap distribution (middle center). Finally, the practitioner receives $\widehat{f}$ which is built with the complete RCT data (top left).

FIGURE 2

Histograms of the bootstrap samples of the out-of-sample improvement measures for $d_0$ random (left column) and $d_0$ best (right column) for the response model of Eq. 11 for different values of n. ${\widehat{I}}_0$ is illustrated with a thick black line. The ${\text{CI}}_{{\mu }_{I_0},95\%}$ computed by the percentile method is illustrated by thin black lines.

FIGURE 3

Histograms of the bootstrap samples of the cross-validated improvement measures for $d_0$ random (left column) and $d_0$ best (right column) for the response model of Eq. 12 for different values of n. ${\widehat{I}}_0$ is illustrated with a thick black line. The ${\text{CI}}_{{\mu }_{I_0},95\%}$ computed via the percentile method is illustrated by thin black lines. The true population improvement ${\mu }_{I_0}^{\text{*}}$ given the optimal rule $d^{\text{*}}$ is illustrated with a dotted black line.

FIGURE 4

Histograms of the bootstrap samples of ${\tilde{I}}_{\text{Rand}}$ i.e. for the random$d_0$ business-as-usual allocation procedure. The thick black line is the best estimate of ${\widehat{I}}_0$, the thin black lines are the confidence interval computed via the percentile method. More negative values are “better” as improvement is defined as lowering the HSRD composite score corresponding to a patient being less depressed.