Sequential Co-Sparse Factor Regression

Abstract

In multivariate regression models, a sparse singular value decomposition of the regression component matrix is appealing for reducing dimensionality and facilitating interpretation. However, the recovery of such a decomposition remains very challenging, largely due to the simultaneous presence of orthogonality constraints and co-sparsity regularization. By delving into the underlying statistical data generation mechanism, we reformulate the problem as a supervised co-sparse factor analysis, and develop an efficient computational procedure, named sequential factor extraction via co-sparse unit-rank estimation (SeCURE), that completely bypasses the orthogonality requirements. At each step, the problem reduces to a sparse multivariate regression with a unit-rank constraint. Nicely, each sequentially extracted sparse and unit-rank coefficient matrix automatically leads to co-sparsity in its pair of singular vectors. Each latent factor is thus a sparse linear combination of the predictors and may influence only a subset of responses. The proposed algorithm is guaranteed to converge, and it ensures efficient computation even with incomplete data and/or when enforcing exact orthogonality is desired. Our estimators enjoy the oracle properties asymptotically; a non-asymptotic error bound further reveals some interesting finite-sample behaviors of the estimators. The efficacy of SeCURE is demonstrated by simulation studies and two applications in genetics.

Keywords: multivariate analysis; reduced-rank regression; regularization; singular value decomposition.

Figure 1:

Simulation: Boxplots of scaled Er(XC) (left panel) and ORT (right panel) in Model II and ρ = 0.3. SeCURE(E*) and SeCURE(E) denote SeCURE using non-adaptive elastic net penalty with and without orthogonality constraints, respectively.

Figure 2:

Yeast cell cycle data: the estimated transcriptional effects of 3 experimentally confirmed TFs identified by SeCURE.

Figure 3:

Yeast cell cycle data: the estimated loadings of the RNAs of 18 time points on the three identified latent factors from the TFs. The fitted curves using kernel smoothing are added. The two vertical lines are drawn at 15 and 75 in the first panel, at 20 and 80 in the second panel and at 30 and 90 in the third panel.

Algorithm 1

Sequential Factor Extraction via Co-Sparse Unit-Rank Estimation (SeCURE)

Algorithm 2

Co-Sparse Unit-Rank Estimation Algorithm (CURE)