Using simulation studies to evaluate statistical methods

Abstract

Simulation studies are computer experiments that involve creating data by pseudo-random sampling. A key strength of simulation studies is the ability to understand the behavior of statistical methods because some "truth" (usually some parameter/s of interest) is known from the process of generating the data. This allows us to consider properties of methods, such as bias. While widely used, simulation studies are often poorly designed, analyzed, and reported. This tutorial outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting, and presentation. In particular, this tutorial provides a structured approach for planning and reporting simulation studies, which involves defining aims, data-generating mechanisms, estimands, methods, and performance measures ("ADEMP"); coherent terminology for simulation studies; guidance on coding simulation studies; a critical discussion of key performance measures and their estimation; guidance on structuring tabular and graphical presentation of results; and new graphical presentations. With a view to describing recent practice, we review 100 articles taken from Volume 34 of Statistics in Medicine, which included at least one simulation study and identify areas for improvement.

Keywords: Monte Carlo; graphics for simulation; simulation design; simulation reporting; simulation studies.

Figure 1

The impacts of bias and empirical SE on root MSE and coverage of nominal 95% confidence intervals, compared for three methods: Method A is unbiased but imprecise; Method B is biased (independent of n _obs) and more precise; Method C is biased (with bias $\propto \sqrt{1/n_{\text{obs}}}$) and the same precision as method B. The comparison of root MSE and coverage depends on the choice of n _obs; the constant bias of method B dominates its increasingly poor MSE and coverage as n _obs increases [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 2

Visualisation of the true hazard rate over follow‐up time in the two data‐generating mechanisms. Black (flat) lines are for the first data‐generating mechanism, where γ = 1; Red curves are for the second, where γ = 1.5 [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 3

Plot of the 1600 ${\widehat{\theta }}_i$ (left panels) and $\widehat{\text{SE}}{(\widehat{\theta })}_i$ (right panels) by data‐generating mechanisms, for the three analysis methods. The vertical axis is repetition number, to provide some separation between points. The yellow pipes are sample means [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 4

Comparison of estimates for methods when γ = 1.5, where each point represents one repetition. A, Upper triangle displays ${\widehat{\theta }}_i$; lower triangle displays $\widehat{\text{SE}}({\widehat{\theta }}_i)$; B, Plot of difference vs mean for ${\widehat{\theta }}_i$ and $\widehat{\text{SE}}({\widehat{\theta }}_i)$, with Weibull as the comparator

Figure 5

“Zip plot” of the 1600 confidence intervals for each data‐generating mechanism and analysis method. The vertical axis is the fractional centile of |z| with $z=({\widehat{\theta }}_i-\theta )/\text{ModSE}$ associated with the confidence interval [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 6

Lollipop plot of performance for measures of interest (Monte Carlo 95% confidence intervals in parentheses). Concerning features need not be highlighted since they are readily visible. See, also, Table 8

Figure A1

Reviewer agreement on key variables for Statistics in Medicine Volume 34 review. Frequency of agreement of TPM with IRW (marker W) and MJC (marker C). For the same frequency, C is nudged left and W right to avoid visual clash [Colour figure can be viewed at wileyonlinelibrary.com]

Figure A2

Results of Statistics in Medicine Volume 34 review for data‐generating mechanisms. Values are both frequency and %

Figure A3

Results of Statistics in Medicine Volume 34 review for estimands (A) and methods (B) evaluated