Transfer Learning for High-Dimensional Linear Regression: Prediction, Estimation and Minimax Optimality

Abstract

This paper considers estimation and prediction of a high-dimensional linear regression in the setting of transfer learning where, in addition to observations from the target model, auxiliary samples from different but possibly related regression models are available. When the set of informative auxiliary studies is known, an estimator and a predictor are proposed and their optimality is established. The optimal rates of convergence for prediction and estimation are faster than the corresponding rates without using the auxiliary samples. This implies that knowledge from the informative auxiliary samples can be transferred to improve the learning performance of the target problem. When the set of informative auxiliary samples is unknown, we propose a data-driven procedure for transfer learning, called Trans-Lasso, and show its robustness to non-informative auxiliary samples and its efficiency in knowledge transfer. The proposed procedures are demonstrated in numerical studies and are applied to a dataset concerning the associations among gene expressions. It is shown that Trans-Lasso leads to improved performance in gene expression prediction in a target tissue by incorporating data from multiple different tissues as auxiliary samples.

Figure 1.

Estimation errors of the Ad hoc ℓ₁-transfer, Agg-Lasso, Lasso, Oracle Trans-Lasso, and Trans-Lasso with identity covariance matrices of the predictors. The two rows correspond to configurations (i) and (ii), respectively. The y-axis corresponds to $\Vert b-\beta {\Vert }_2^2$ for some estimator b.

Figure 2.

Estimation errors of the Ad hoc ℓ₁-transfer, Agg-Lasso, Lasso, Oracle Trans-Lasso, and Trans-Lasso with homogeneous covariance matrices. The two rows correspond to configurations (i) and (ii), respectively. The y-axis corresponds to $\Vert b-\beta {\Vert }_2^2$ for some estimator b.

Figure 3.

Estimation errors of the Ad hoc ℓ₁-transfer, Agg-Lasso, Lasso, Oracle Trans-Lasso, and Trans-Lasso with heterogeneous covariance matrices. The two rows correspond to configurations (i) and (ii), respectively. The y-axis corresponds to $\Vert b-\beta {\Vert }_2^2$ for some estimator b.

Figure 4.

Prediction errors of Agg-Lasso, Naive Trans-Lasso, Trans-Lasso, and Ad hoc ℓ₁-transfer relative to the Lasso evaluated via 5-fold cross validation for gene JAM2 in multiple tissues.

Figure 5.

Prediction errors of Ad hoc ℓ₁-transfer, Agg-Lasso, Naive Trans-Lasso*, and Trans-Lasso relative to the Lasso for the 25 genes on chromosome 21 and in Module 137, in multiple target tissues. The Naive Trans-Lasso has two outliers for the tissue Cerebellum not showing in the figure with values 1.61 and 1.95.