Semi-automated sensitivity analysis to assess systematic errors in observational data

Abstract

Background: Published epidemiologic research usually provides a quantitative assessment of random error for effect estimates, but no quantitative assessment of systematic error. Sensitivity analysis can provide such an assessment.

Methods: We describe a method to reconstruct epidemiologic data, accounting for biases, and to display the results of repeated reconstructions as an assessment of error. We illustrate with a study of the effect of less-than-definitive therapy on breast cancer mortality.

Results: We developed SAS code to reconstruct the data that would have been observed had a set of systematic errors been absent, and to convey the results. After 4,000 reconstructions of the example data, we obtained a median estimate of relative hazard equal to 1.5 with a 95% simulation interval of 0.8-2.8. The relative hazard obtained by conventional analysis equaled 2.0, with a 95% confidence interval of 1.2-3.4.

Conclusions: Our method of sensitivity analysis can be used to quantify the systematic error for an estimate of effect and to describe that error in figures, tables, or text. In the example, the sources of error biased the conventional relative hazard away from the null, and that error was not accurately communicated by the conventional confidence interval.