Fast sampling with Gaussian scale-mixture priors in high-dimensional regression

Abstract

We propose an efficient way to sample from a class of structured multivariate Gaussian distributions. The proposed algorithm only requires matrix multiplications and linear system solutions. Its computational complexity grows linearly with the dimension, unlike existing algorithms that rely on Cholesky factorizations with cubic complexity. The algorithm is broadly applicable in settings where Gaussian scale mixture priors are used on high-dimensional parameters. Its effectiveness is illustrated through a high-dimensional regression problem with a horseshoe prior on the regression coefficients. Other potential applications are outlined.

Keywords: Confidence interval; Gaussian scale mixture; Global-local prior; Shrinkage; Sparsity.

Fig. 1

Boxplots of ℓ₁, ℓ₂ and prediction error across 100 simulation replicates. HSme and HSm respectively denote posterior point wise median and mean for the horeshoe prior. True β₀ is 5-sparse with nonzero entries ±{1.5, 1.75, 2, 2.25, 2.5}. Top row: Σ = I_p (independent). Bottom row: Σ_jj = 1, Σ_jj′ = 0.5, j ≠ j′ (compound symmetry).

Fig. 2

Same setting as in Fig 1. True β₀ is 5-sparse with non-zero entries ±{0.75, 1, 1.25, 1.5, 1.75}.