Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR

Abstract

Variable selection can be challenging, particularly in situations with a large number of predictors with possibly high correlations, such as gene expression data. In this article, a new method called the OSCAR (octagonal shrinkage and clustering algorithm for regression) is proposed to simultaneously select variables while grouping them into predictive clusters. In addition to improving prediction accuracy and interpretation, these resulting groups can then be investigated further to discover what contributes to the group having a similar behavior. The technique is based on penalized least squares with a geometrically intuitive penalty function that shrinks some coefficients to exactly zero. Additionally, this penalty yields exact equality of some coefficients, encouraging correlated predictors that have a similar effect on the response to form predictive clusters represented by a single coefficient. The proposed procedure is shown to compare favorably to the existing shrinkage and variable selection techniques in terms of both prediction error and model complexity, while yielding the additional grouping information.

Figure 1

Graphical representation of the constraint region in the (β₁, β₂) plane for the LASSO, Elastic Net, and OSCAR. Note that all are non-differentiable at the axes. (a) Constraint region for the Lasso (solid line), along with three choices of tuning parameter for the Elastic Net. (b) Constraint region for the OSCAR for four values of c. The solid line represents c = 0, the LASSO.

Figure 2

Graphical representation in the (β₁, β₂) plane. The OSCAR solution is the first time the contours of the sum-of-squares function hits the octagonal constraint region. (a) Contours centered at OLS estimate, low correlation (ρ = .15). Solution occurs at ${\widehat{\beta }}_1=0$. (b) Contours centered at OLS estimate, high correlation (ρ = .85). Solution occurs at ${\widehat{\beta }}_1={\widehat{\beta }}_2$.

Figure 3

Graphical representation of the correlation matrix of the 15 predictors for the soil data. The magnitude of each pairwise correlation is represented by a block in the grayscale image.

Figure 4

LASSO solution paths for the soil data. Absolute value of the 15 coefficients as a function of s, the proportion of the OLS norm, for the fixed value of c = 0, the LASSO. The vertical lines represent the best LASSO models in terms of the GCV and the 5-fold cross-validation criteria. (a) Solution paths for the 7 cation-related coefficients. (b) Solution paths for the remaining 8 coefficients.

Figure 5

OSCAR solution paths for the soil data. Absolute value of the 15 coefficients as a function of s, the proportion of the OLS norm, for the value of c = 4 as chosen by both GCV and 5-fold cross-validation. The vertical lines represent the selected models based on the two criteria. (a) Solution paths for the 7 cation-related coefficients. (b) Solution paths for the remaining 8 coefficients.