Evidence of misuse of nonparametric tests in the presence of heteroscedasticity within obesity research

Abstract

Background: Classic nonparametric tests (cNPTs), like Kruskal-Wallis or Mann-Whitney U, are sometimes used to detect differences in central tendency ( i.e., means or medians). However, when the tests' assumptions are violated, such as in the presence of unequal variance and other forms of heteroscedasticity, they are no longer valid for testing differences in central tendency. Yet, sometimes researchers erroneously use cNPTs to account for heteroscedasticity.

Objective: To document the appropriateness of cNPT use in obesity literature, characterize studies that use cNPTs, and evaluate the citation and public sharing patterns of these articles.

Methods: We reviewed obesity studies published in 2017 to determine whether the authors used cNPTs: (1) to correct for heteroscedasticity (invalid); (2) when heteroscedasticity was clearly not present (correct); or (3) when it was unclear whether heteroscedasticity was present (unclear). Open science R packages were used to transparently search literature and extract data on how often papers with errors have been cited in academic literature, read in Mendeley, and disseminated in the media.

Results: We identified nine studies that used a cNPT in the presence of heteroscedasticity (some because of the mistaken rationale that the test corrected for heteroscedasticity), 25 articles that did not explicitly state whether heteroscedasticity was present when a cNPT was used, and only four articles that appropriately reported that heteroscedasticity was not present when a cNPT was used. Errors were found in observational and interventional studies, in human and rodent studies, and only when studies were unregistered. Studies with errors have been cited 113 times, read in Mendeley 123 times, and disseminated in the media 41 times, by the public, scientists, science communicators, and doctors.

Conclusions: Examples of inappropriate use of cNPTs exist in the obesity literature, and those articles perpetuate the errors via various audiences and dissemination platforms.

Keywords: Nonparametric tests; heteroscedasticity; nutrition; obesity; open science; research rigor; statistical methods.

Figure 1.. Flowchart of inclusion of studies for review of nonparametric test use.

Figure 2.. Appropriateness of nonparametric test use in the included studies.

Error, used a nonparametric test and explicitly stated that the test was used in the presence of heteroscedasticity or to correct for heteroscedasticity; No Link, used at least one nonparametric test but made no explicit statement of whether heteroscedasticity was tested for or present; Ambiguous, used a nonparametric test and stated that variance assumptions were tested for, but did not explicitly state whether heteroscedasticity was present; Correct, used a nonparametric test and explicitly stated that heteroscedasticity was not present.

Figure 3.. Appropriateness of nonparametric test use in nutrition and obesity studies published in 2017 according to study characteristics.

*Nonparametric Tests: Description of the test used was only “non-parametric tests.” Observed statistical significance: Number of articles that report results obtained from a nonparametric test as statistically significant (significant), mixed (mixed significance), or not significant. Correct, used a nonparametric test and explicitly stated that heteroscedasticity was not present; Ambiguous, used a nonparametric test and stated that variance assumptions were tested for but did not explicitly state whether heteroscedasticity was present; No Link, used at least one nonparametric test, but made no explicit statement of whether heteroscedasticity was tested for or present; Error, used a nonparametric test and explicitly stated that the test was used in the presence of heteroscedasticity or to correct for heteroscedasticity.

Figure 4.. Appropriateness of nonparametric test use in nutrition and obesity studies published in 2017 according to reported responsible research practices.

Correct, used a nonparametric test and explicitly stated that heteroscedasticity was not present; Ambiguous, used a nonparametric test and stated that variance assumptions were tested for but did not explicitly state whether heteroscedasticity was present; No Link, used at least one nonparametric test, but made no explicit statement of whether heteroscedasticity was tested for or present; Error, used a nonparametric test and explicitly stated that the test was used in the presence of heteroscedasticity or to correct for heteroscedasticity. For Raw Data Available, “No” means the authors stated the data are not available, the paper had no data availability statement, or the data availability statement was left blank.

Figure 5.. Dissemination of nutrition and obesity studies published in 2017 after an average of 1268 days since publication. A.

Total number of journal citations, reads on Mendeley reference library (Mendeley Ltd), internet posts, and total Altmetric Attention Score (Altmetric LLP) by appropriateness of nonparametric test use. Sample size of included articles with dissemination data: Error: n = 4; No Link: n = 11; Ambiguous: n = 9; Correct: n = 4. Citation counts were available for all included studies. B. Total number of internet posts (e.g., news, blogs, Twitter, Wikipedia, LinkedIn, Facebook) by cohort and appropriateness of nonparametric test use. Sample size of included articles with dissemination data: Error: n = 4; No Link: n = 11; Ambiguous: n = 8; Correct: n = 4. Correct, used a nonparametric test and explicitly stated that heteroscedasticity was not present; Ambiguous, used a nonparametric test and stated that variance assumptions were tested for but did not explicitly state whether heteroscedasticity was present; No Link, used at least one nonparametric test, but made no explicit statement of whether heteroscedasticity was tested for or present; Error, used a nonparametric test and explicitly stated that the test was used in the presence of heteroscedasticity or to correct for heteroscedasticity.