A New Bayesian Lasso

Abstract

Park and Casella (2008) provided the Bayesian lasso for linear models by assigning scale mixture of normal (SMN) priors on the parameters and independent exponential priors on their variances. In this paper, we propose an alternative Bayesian analysis of the lasso problem. A different hierarchical formulation of Bayesian lasso is introduced by utilizing the scale mixture of uniform (SMU) representation of the Laplace density. We consider a fully Bayesian treatment that leads to a new Gibbs sampler with tractable full conditional posterior distributions. Empirical results and real data analyses show that the new algorithm has good mixing property and performs comparably to the existing Bayesian method in terms of both prediction accuracy and variable selection. An ECM algorithm is provided to compute the MAP estimates of the parameters. Easy extension to general models is also briefly discussed.

Keywords: Bayesian Lasso; Gibbs Sampler; Lasso; MCMC; Scale Mixture of Uniform.

Figure 1

Posterior mean Bayesian lasso estimates (computed over a grid of λ values, using 10,000 samples after burn-in) and corresponding 95% credible intervals (equal-tailed) of Diabetes data (n = 442) covariates. The hyperprior parameters were chosen as a = 1, b = 0.1. OLS estimates with corresponding 95% confidence intervals are also reported. For the lasso estimates, the tuning parameter was chosen by 10-fold CV of the LARS algorithm.

Figure 2

ACF plots and histograms based on posterior samples of Diabetes data covariates.

Figure 3

Trace plots of Diabetes data covariates.

Figure 4

Posterior mean Bayesian lasso estimates (computed over a grid of λ values, using 10,000 samples after burn-in) and corresponding 95% credible intervals (equal-tailed) of Prostate data (n = 67) covariates. The hyperprior parameters were chosen as a = 1, b = 0.1. OLS estimates with corresponding 95% confidence intervals are also reported. For the lasso estimates, the tuning parameter was chosen by 10-fold CV of the LARS algorithm.

Figure 5

Trace plots of Prostate data covariates.

Figure 6

ACF plots and histograms based on posterior samples of Prostate data covariates.