Air quality environmental epidemiology studies are unreliable

Abstract

Ever since the London Great Smog of 1952 is estimated to have killed over 4000 people, scientists have studied the relationship between air quality and acute mortality. There are many hundreds of papers examining the question. There is a serious statistical problem with most of these papers. If there are many questions under consideration, and there is no adjustment for multiple testing or multiple modeling, then unadjusted p-values are totally unreliable making claims unreliable. Our idea is to determine the statistical reliability of eight papers published in Environmental Health Perspectives that were used in meta-analysis papers appearing in Lancet and JAMA. We counted the number of outcomes, air quality predictors, time lags and covariates examined in each paper. We estimate the multiplicity of questions that could be asked and the number of models that could be constructed. The results were that the median numbers of comparisons possible for multiplicity, models and search space were 135, 128, and 9568 respectively. Given the large search spaces, finding a small number of nominally significant results is not unusual at all. The claims in these eight papers are not statistically supported so these papers are unreliable as are the meta-analysis papers that use them.

Keywords: Air quality; Analysis search space; Environmental epidemiology; Mortality; Multiple modeling; Multiple testing; Observational studies.