An automated sampling importance resampling procedure for estimating parameter uncertainty

Abstract

Quantifying the uncertainty around endpoints used for decision-making in drug development is essential. In nonlinear mixed-effects models (NLMEM) analysis, this uncertainty is derived from the uncertainty around model parameters. Different methods to assess parameter uncertainty exist, but scrutiny towards their adequacy is low. In a previous publication, sampling importance resampling (SIR) was proposed as a fast and assumption-light method for the estimation of parameter uncertainty. A non-iterative implementation of SIR proved adequate for a set of simple NLMEM, but the choice of SIR settings remained an issue. This issue was alleviated in the present work through the development of an automated, iterative SIR procedure. The new procedure was tested on 25 real data examples covering a wide range of pharmacokinetic and pharmacodynamic NLMEM featuring continuous and categorical endpoints, with up to 39 estimated parameters and varying data richness. SIR led to appropriate results after 3 iterations on average. SIR was also compared with the covariance matrix, bootstrap and stochastic simulations and estimations (SSE). SIR was about 10 times faster than the bootstrap. SIR led to relative standard errors similar to the covariance matrix and SSE. SIR parameter 95% confidence intervals also displayed similar asymmetry to SSE. In conclusion, the automated SIR procedure was successfully applied over a large variety of cases, and its user-friendly implementation in the PsN program enables an efficient estimation of parameter uncertainty in NLMEM.

Keywords: Asymptotic covariance matrix; Bootstrap; Confidence intervals; Nonlinear mixed-effects models; Parameter uncertainty; Sampling importance resampling.

Fig. 1

SIR diagnostic plot showing SIR convergence for one of the investigated examples. Lines represent the value of the dOFV for each percentile of the proposal distributions (dotted colored lines) and the resamples distributions (solid colored lines) at each iteration. The shaded area represents the resampling noise around the last resamples dOFV distribution. The dOFV resamples distributions of the last two iterations need to be within sampling noise for SIR results to be considered final. The reference Chi square distribution with degrees of freedom equal to the number of estimated parameters is the grey solid line and the estimated degrees of freedom for each distribution are displayed in the bottom right corner

Fig. 2

Proposed SIR workflow. To obtain SIR parameter uncertainty for a given model and data, SIR is started using, in order of preference: the covariance matrix, a limited bootstrap (e.g. 200 samples or less). or a generic covariance matrix (e.g. 50% RSE on all parameters) as first proposal distribution. Then, SIR iterations are automatically performed using the resamples of one iteration as proposal distribution of the next, until the dOFV distributions of the resamples of the last 2 iterations are overlaid in the dOFV plot, in which case SIR results are considered final

Fig. 3

Convergence of the informed SIR over the 25 investigated models as represented by the estimated degree of freedom of the SIR resamples distribution at each iteration, normalized by the total number of estimated parameters of each model. The normalized degree of freedom at the 0th iteration is the degree of freedom of the informed proposal distribution (covariance matrix or limited bootstrap). Boxplots represent the median, first and third quartiles of the degree of freedom during the proposed iterative procedure until the 5th iteration, when most of the models had converged

Fig. 4

Normalized degree of freedom of the SIR resamples distribution at stabilization for the generic SIR (y-axis) and for the informed SIR (x-axis) for the 17 models for which results from both SIR were available. The full black line is the identity line and the dashed lines represent deviations of 20% from the identity line

Fig. 5

Distribution of the median (over all parameters) RSE, 95% CI width (WIDTH95) and asymmetry (ASYM95) for all models by uncertainty method: SIR, covariance matrix (cov), bootstrap (boot) and SSE

Fig. 6

Normalized degree of freedom by uncertainty method: SIR, covariance matrix (cov), bootstrap (boot) and SSE. The dashed horizontal line corresponds to a degree of freedom equal to the total number of estimated parameters