Prediction discrepancies for the evaluation of nonlinear mixed-effects models

Abstract

Reliable estimation methods for non-linear mixed-effects models are now available and, although these models are increasingly used, only a limited number of statistical developments for their evaluation have been reported. We develop a criterion and a test to evaluate nonlinear mixed-effects models based on the whole predictive distribution. For each observation, we define the prediction discrepancy (pd) as the percentile of the observation in the whole marginal predictive distribution under H(0). We propose to compute prediction discrepancies using Monte Carlo integration which does not require model approximation. If the model is valid, these pd should be uniformly distributed over (0, 1) which can be tested by a Kolmogorov-Smirnov test. In a simulation study based on a standard population pharmacokinetic model, we compare and show the interest of this criterion with respect to the one most frequently used to evaluate nonlinear mixed-effects models: standardized prediction errors (spe) which are evaluated using a first order approximation of the model. Trends in pd can also be evaluated via several plots to check for specific departures from the model.

Fig. 1

Typical data set containing 300 simulated pseudo-observed concentrations and mean-predicted concentrations for the basic model, in case IIa (N=100, n=3).

Fig. 2

Goodness-of-fit plots under H₀, for standardized prediction errors (top) or for prediction discrepancies (bottom), for one simulation in case IIa (N=100, n=3). Left: quantile-quantile plot for a uniform distribution; middle: histograms of errors; right: box plots of the 50 errors at each sampling time versus time.

Fig. 3

Goodness-of-fit plots under the alternative assumption that the variability of Cl is multiplied by two, for standardized prediction errors (top) or prediction discrepancies (bottom), for one simulation in case IIa (N=100, n=3) (see legend of Fig. 2 for more details).

Fig. 4

Goodness-of-fit plots under the alternative assumption that the pharmacokinetic model is a two-compartment model, for standardized prediction errors (top) or prediction discrepancies (bottom), for one simulation in case IIa (N=100, n=3) (see legend of Fig. 2 for more details).