Default Priors for the Intercept Parameter in Logistic Regressions

Abstract

In logistic regression, separation occurs when a linear combination of predictors perfectly discriminates the binary outcome. Because finite-valued maximum likelihood parameter estimates do not exist under separation, Bayesian regressions with informative shrinkage of the regression coefficients offer a suitable alternative. Classical studies of separation imply that efficiency in estimating regression coefficients may also depend upon the choice of intercept prior, yet relatively little focus has been given on whether and how to shrink the intercept parameter. Alternative prior distributions for the intercept are proposed that downweight implausibly extreme regions of the parameter space, rendering regression estimates that are less sensitive to separation. Through simulation and the analysis of exemplar datasets, differences across priors stratified by established statistics measuring the degree of separation are quantified. Relative to diffuse priors, these proposed priors generally yield more efficient estimation of the regression coefficients themselves when the data are nearly separated. They are equally efficient in non-separated datasets, making them suitable for default use. Modest differences were observed with respect to out-of-sample discrimination. These numerical studies also highlight the interplay between priors for the intercept and the regression coefficients: findings are more sensitive to the choice of intercept prior when using a weakly informative prior on the regression coefficients than an informative shrinkage prior.

Keywords: Bayesian Methods; Exponential-Power Distribution; Pivotal Separation; Quasi-Complete Separation; Rare Events.

Figure 1:

Densities of six prior distributions on α, arranged by the height of the density at α = 0. The scales of the Logis(1.17), EP₂(2.41), and EP₁₀(4.19) priors were set according to Algorithm 1 with n = 250, using q = 0.01, which ensures that only 1% of the prior mass falls outside of the interval [−6.21, 6.21], denoted by the vertical lines. EP₁₀(6.22) is included as a non-adaptive, diffuse EP-type prior, the scale of which comes from using s_n ≡ 10⁻⁴ in Algorithm 1.

Figure 2:

Comparison of t₃(10) prior on α against five alternative priors on α (columns) under two different priors on β (rows). Each point represents an individual simulated dataset. The y-axis gives root mean-squared error (RMSE) ratios on the log₂ scale when estimating β with its posterior mean, and the x-axis classifies datasets into five categories based upon degree of separation. The second group, n_comp/n_piv = 0, indicates datasets that are completely but not pivotally separated. Positive values on the y-axis indicate that the given prior on α yielded more efficient estimation of β than a t₃(10) prior on α. Different plot characters are used to indicate p, the number of predictors. In total, each panel contains 6600 points (33 unique scenario-sample size configuration times 200 simulated datasets per configuration).