Incorporating prior knowledge into regularized regression

Abstract

Motivation: Associated with genomic features like gene expression, methylation and genotypes, used in statistical modeling of health outcomes, there is a rich set of meta-features like functional annotations, pathway information and knowledge from previous studies, that can be used post hoc to facilitate the interpretation of a model. However, using this meta-feature information a priori rather than post hoc can yield improved prediction performance as well as enhanced model interpretation.

Results: We propose a new penalized regression approach that allows a priori integration of external meta-features. The method extends LASSO regression by incorporating individualized penalty parameters for each regression coefficient. The penalty parameters are, in turn, modeled as a log-linear function of the meta-features and are estimated from the data using an approximate empirical Bayes approach. Optimization of the marginal likelihood on which the empirical Bayes estimation is performed using a fast and stable majorization-minimization procedure. Through simulations, we show that the proposed regression with individualized penalties can outperform the standard LASSO in terms of both parameters estimation and prediction performance when the external data is informative. We further demonstrate our approach with applications to gene expression studies of bone density and breast cancer.

Availability and implementation: The methods have been implemented in the R package xtune freely available for download from https://cran.r-project.org/web/packages/xtune/index.html.

Fig. 1.

Simulation results. Subplot (a): SNR = 1, 2, 3, with n = 200, p = 1000, q = 10, ${\alpha }_0=3,\hspace{0.17em}\delta =0.5,\hspace{0.17em}\rho =0.2$. Subplot (b): p = 500, 1000, 3000, with n = 200, SNR = 2, q = 10, ${\alpha }_0=3,\hspace{0.17em}\delta =0.5,\hspace{0.17em}\rho =0.2$. Subplot (c): q = 10, 30, 50, with n = 200, p = 1000, SNR = 2, ${\alpha }_0=3,\hspace{0.17em}\delta =0.5,\hspace{0.17em}\rho =0.2$. Subplot (d): regression coefficients sparsity δ = $0.3,0.5,0.7$, with n = 200, p = 1000, SNR = 2, q = 10, ${\alpha }_0=3,\hspace{0.17em}\rho =0.2$. Subplot (e): overall penalty magnitude α₀ = 1, 3, 5, n = 200, p = 1000, SNR = 2, q = 10, $\delta =0.5,\hspace{0.17em}\rho =0.2$. Subplot (f): $\rho =0.3,0.6,0.9$, with n = 200, p = 1000, SNR = 2, q = 10, ${\alpha }_0=3,\hspace{0.17em}\delta =0.5$

Fig. 2.

Compare test R² and number of selected covariates of adaptive LASSO, standard LASSO and xtune LASSO using bone density data. The mean test R² is 0.27 for adaptive LASSO; 0.38 for the standard LASSO, and 0.43 for xtune LASSO. The mean number of selected covariates is 10 for adaptive LASSO, 43 for standard LASSO and 16 for the xtune LASSO

Fig. 3.

ROC curves for adaptive LASSO, standard LASSO and xtune LASSO applied to the breast cancer dataset