Ubiquitous bias and false discovery due to model misspecification in analysis of statistical interactions: The role of the outcome's distribution and metric properties

Abstract

Studies of interaction effects are of great interest because they identify crucial interplay between predictors in explaining outcomes. Previous work has considered several potential sources of statistical bias and substantive misinterpretation in the study of interactions, but less attention has been devoted to the role of the outcome variable in such research. Here, we consider bias and false discovery associated with estimates of interaction parameters as a function of the distributional and metric properties of the outcome variable. We begin by illustrating that, for a variety of noncontinuously distributed outcomes (i.e., binary and count outcomes), attempts to use the linear model for recovery leads to catastrophic levels of bias and false discovery. Next, focusing on transformations of normally distributed variables (i.e., censoring and noninterval scaling), we show that linear models again produce spurious interaction effects. We provide explanations offering geometric and algebraic intuition as to why interactions are a challenge for these incorrectly specified models. In light of these findings, we make two specific recommendations. First, a careful consideration of the outcome's distributional properties should be a standard component of interaction studies. Second, researchers should approach research focusing on interactions with heightened levels of scrutiny. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Figure 1:

Scatterplot of $\mathbb{E}(y|x,z)$ and $xz$ for different values of ${\beta }_0\left({\beta }_1={\beta }_2=1,{\beta }_3=0,N=1000\right)$ when $y$ is a binary outcome.

Figure 2:

Illustration of the geometry driving false discovery due to variation in ${\beta }_0$ when the linear model is used $\left({\beta }_1={\beta }_2=1,{\beta }_3=0,N=1000\right)$ for analysis of binary outcomes. Blue and red dots represent those data points within half a unit of their respective $z$ values (i.e., $z$ such that $|z-1|<0.5$ are in blue and $z$ such that $|z+1|<0.5$ are in red. Fitted lines for ±1 are similarly shaded). Solid lines are fits from logistic regression model while dashed lines are fits from linear model.

Figure 3:

Variation in bias (left) and Type 1 error rate (right) associated with estimates of ${\beta }_3$ when ${\beta }_3=0$ for binary outcomes (for ${\beta }_0=0,{\beta }_1={\beta }_2=1$). Left: Estimates $\widehat{{\beta }_3^{\mathrm{L}\mathrm{M}}}$ based on the LM for two sample sizes as a function of ${\beta }_0$. Shaded regions capture span of estimates between 10th and 90th percentile while the solid line shows median estimates. Right: Levels of Type 1 error as a function of sample size for two values of ${\beta }_0$.

Figure 4:

Type 1 error rate as a function of sample size and prevalence when using LM to analyze binary outcomes. In all cases, ${\beta }_3=0$ while ${\beta }_0$ varies (such that stated prevalence is equivalent to $\frac{e^{{\beta }_0}}{1+e^{{\beta }_0}}$). The ${\beta }_1$ and ${\beta }_2$ coefficients are as shown for each panel. Blue regions indicate areas where Type 1 error is appropriate whereas red indicates regions wherein bias is leading to elevated levels of Type 1 error.

Figure 5:

Scatterplot of $\mathbb{E}(y|x,z)$ and $xz$ for different values of ${\beta }_2\left({\beta }_0=1,{\beta }_1=0.2,N=1000\right)$ when $y_i$ is a count outcome.

Figure 6:

Illustration of the geometry driving false discovery due to variation in ${\beta }_2$ when the linear model is used $\left({\beta }_0=1,{\beta }_1=0.2,N=1000\right)$ for analysis of count outcomes. Blue and red dots represent those data points within half a unit of their respective $z$ values (i.e., $z$ such that $|z-1|<0.5$ are in blue and $z$ such that $|z+1|<0.5$ are in red. Fitted lines for ±1 are similarly shaded). Solid lines are fits from Poisson regression model while dashed lines are fits from linear model.

Figure 7:

Variation in bias (left) and Type 1 error rate (right) associated with $\widehat{{\beta }_3^{\mathrm{L}\mathrm{M}}}$ when ${\beta }_3=0$ for count outcomes (for ${\beta }_0=0,{\beta }_1=0.5$). Left: Estimates of ${\beta }_3$ based on the LM for two sample sizes as a function of ${\beta }_2$. Shaded regions capture span of estimates between 10th and 90th percentile while the solid line shows median estimates. Right: Levels of Type 1 error as a function of sample size for two values of ${\beta }_2$.

Figure 8:

Scatterplot of $y$ or $y^{\star }$ and $xz$ for different values of $c\left({\beta }_0=0,{\beta }_1={\beta }_2=1,{\sigma }_y^2=0.25,N=1000\right)$ when $y_i$ is a censored outcome.

Figure 9:

Illustration of the geometry driving false discovery due to variation in $c$ when the linear model is used ${\beta }_0=0,{\beta }_1={\beta }_2=1,{\sigma }_y^2=0.25,N=1000$) for analysis of censored outcomes. Blue and red dots represent those data points within half a unit of their respective $z$ values (i.e., $z$ such that $|z-1|<0.5$ are in blue and $z$ such that $|z+1|<0.5$ are in red. Fitted lines for ±1 are similarly shaded). Solid lines are fits from Tobit regression model while dashed lines are fits from linear model.

Figure 10:

Variation in bias (left) and Type 1 error rate (right) associated with $\widehat{{\beta }_3^{\mathrm{L}\mathrm{M}}}$ when ${\beta }_3=0$ for outcomes with a floor $\left({\beta }_0=0,{\beta }_1={\beta }_2=1,{\sigma }_y^2=1\right)$. Left: Estimates of ${\beta }_3$ based on the LM for two sample sizes as a function of $c$. Shaded regions capture span of estimates between 10th and 90th percentile while the solid line shows median estimates. Right: Levels of Type 1 error as a function of sample size for two values of $c$.

Figure 11:

Effect of transformation when $\alpha =0$ for various values of $\lambda$.

Figure 12:

Variation in bias (left) and Type 1 error rate (right) associated with $\widehat{{\beta }_3^{\mathrm{L}\mathrm{M}}}$ when ${\beta }_3=0$ for outcomes observed following Lord’s transformation $\left({\beta }_0=0,{\beta }_1={\beta }_2=1,{\sigma }_y^2=1,\alpha =0\right)$. Left: Estimates of ${\beta }_3$ based on the LM for two sample sizes as a function of $\lambda$. Shaded regions capture span of estimates between 10th and 90th percentile while the solid line shows median estimates. Right: Levels of Type 1 error as a function of sample size for two values of $\lambda$.