Reclassification of predictions for uncovering subgroup specific improvement

Abstract

Risk prediction models play an important role in prevention and treatment of several diseases. Models that are in clinical use are often refined and improved. In many instances, the most efficient way to improve a successful model is to identify subgroups for which there is a specific biological rationale for improvement and tailor the improved model to individuals in these subgroups, an approach especially in line with personalized medicine. At present, we lack statistical tools to evaluate improvements targeted to specific subgroups. Here, we propose simple tools to fill this gap. First, we extend a recently proposed measure, the Integrated Discrimination Improvement, using a linear model with covariates representing the subgroups. Next, we develop graphical and numerical tools that compare reclassification of two models, focusing only on those subjects for whom the two models reclassify differently. We apply these approaches to BRCAPRO, a genetic risk prediction model for breast and ovarian cancer, using data from MD Anderson Cancer Center. We also conduct a simulation study to investigate properties of the new reclassification measure and compare it with currently used measures. Our results show that the proposed tools can successfully uncover subgroup specific model improvements.

Keywords: BRCAPRO; Integrated Discrimination Improvement; breast cancer; reclassification methods; risk prediction.

Figure 1

Proportion of correct reclassification (PCR^c) comparing the “ER/PR, Her-2” model to the “ER/PR” model. The figures below each point are the total number of probands reclassified $(n_R^c)$ at that cutoff. The gray vertical segments are 95% pointwise binomial CIs.

Figure 2

Proportion of correct reclassification (PCR^c) comparing the “ER/PR, Her-2” model to the “ER/PR” model plotted for sub-groups of probands with specific combinations of ER/PR and Her-2/neu status. The figures below each point are the total number of probands reclassified $(n_R^c)$ at that cutoff. When no reclassification occurs at a given cutoff, no point is plotted.

Figure 3

Power vs. Type I error in simulations: The parameters for various scenarios are listed in Table 4. The title of each plot corresponds to the scenario for power, while the type I error is for Scenario 9 in all plots. The light gray line is at type I error of 0.05. IDI: Integrated Discrimination Improvement; PCR: Proportion of Correct Reclassification.

Figure 4

Power vs. Type I error in simulations: The parameters for various scenarios are listed in Table 4. The title of each plot corresponds to the scenario for power, while the type I error is for Scenario 10 in all plots. The light gray line is at type I error of 0.05. IDI: Integrated Discrimination Improvement; PCR: Proportion of Correct Reclassification.

Figure 5

A sample of p’s and q’s generated under scenario 5. The orange and green pairs of lines (vertical and horizontal) are to facilitate seeing the total number of observations reclassified $(n_R^c)$ at cutoffs of 0.2 and 0.7. Each pair of lines divides all points in four quadrants. The points in the second and fourth quadrant are the ones reclassified differently by the two models at that cutoff.