Sloppy models, parameter uncertainty, and the role of experimental design

Abstract

Computational models are increasingly used to understand and predict complex biological phenomena. These models contain many unknown parameters, at least some of which are difficult to measure directly, and instead are estimated by fitting to time-course data. Previous work has suggested that even with precise data sets, many parameters are unknowable by trajectory measurements. We examined this question in the context of a pathway model of epidermal growth factor (EGF) and neuronal growth factor (NGF) signaling. Computationally, we examined a palette of experimental perturbations that included different doses of EGF and NGF as well as single and multiple gene knockdowns and overexpressions. While no single experiment could accurately estimate all of the parameters, experimental design methodology identified a set of five complementary experiments that could. These results suggest optimism for the prospects for calibrating even large models, that the success of parameter estimation is intimately linked to the experimental perturbations used, and that experimental design methodology is important for parameter fitting of biological models and likely for the accuracy that can be expected from them.

Figure 1

(A) Uncertainty ellipse for a simple two-parameter system. The parameters inside the ellipse are feasible. The major and minor axes of the ellipse are proportional to ${\lambda }_1^{-1∕2}$ and ${\lambda }_2^{-1∕2}$, respectively. The grey contours are lines of equal objective function value. The bounding box (dashed magenta lines) shows the single parameter errors. The length and width of the bounding box is the range of values for the individual parameters. (B) Two non-complementary experiments. (C) Two complementary experiments. When two experiments (blue and red ellipses) are combined the resulting parameter estimate (green ellipses) can be improved as in the case of panel C or not as in the case of panel B depending on the complementarity of the component experiments.

Figure 2

(A) Search result for sets of experiments that maximize the number of parameters estimated. Each spectrum is the eigenspectrum of estimation problem. The goal of the design is to maximize the number of parameters with errors less than 10% (dashed line). (B) Design based on selecting the best single experiments. (C) Design based on selecting random experiments. (D) The number of parameters estimated by each search method. By the fifth experiment the greedy algorithm is able to estimate all 48 parameters to the desired accuracy. The error bars show the standard deviation for 10000 random searches.

Figure 3

The subset of parameters that can be estimated with different levels of experiment. (A,B) Doses of EGF and NGF alone. (C,D) Doses and single knockouts/overexpressions. (E,F) Doses and double knockouts/overexpressions. (G,H) Doses and triple knockouts/overexpressions. The network diagrams show the individual parameter errors. Each arrow represents a reaction (black for activating, red for inhibitory). Each reaction is parameterized by two parameters. If both parameters are estimated to 10% the line is thick, if only one of the two parameters is estimated the line is medium, and if neither parameter is estimated then the line is thin. The eigenvector matrices pictures on the right show the vector perspective. The eigenvalues decrease from left to right. The green lines indicates the cutoff for 10% relative error. The yellow and red lines indicate directions with 100% and 1000% error.

Figure 4

Experimental perturbations push enzymes from operating in pure $\frac{k_{\mathrm{cat}}}{K_M}$ conditions to facilitate estimation of k_cat and K_M individually. Experiment #4 calls for the overexpression of the three enzymes P90Rsk, Akt, and PI3K, which are substrates for the enzymes (A,B) ERK, (C,D) PI3K, and (E,F) EGFR, respectively. The top row shows the rate of reaction for each enzyme as a function of its substrate concentration (solid black line), the linearized rate law at low substrate concentration $\frac{k_{\mathrm{cat}}}{K_M}$ (dashed black line), and a histogram of concentration in experiment #1 (blue shading) and in experiment #4 (red shading). The bottom row shows the corresponding parameter uncertainty, with K_M on the abscissa and k_cat on the ordinate. Uncertainty is shown for experiment #1 alone (blue ellipse), experiment #4 alone (red ellipse), and the combination of both experiments (green ellipse). In experiment #1 only the linear region of the enzyme rate curve is explored (blue shading in top row). The linear regime specifies $\frac{k_{\mathrm{cat}}}{K_M}$ (represented as log(k_cat)–log(K_M) in log parameter space and corresponds to the blue ellipses in the bottom row of the Figure). The blue ellipses are extended along the y = x direction, indicating very little uncertainty in log(k_cat)–log(K_M) but great uncertainty in log(k_cat)+log(K_M). In experiment #4 all three substrates are overexpressed and the saturating portion of the rate curve is populated, where r≈e₀k_cat, which specifies the log(k_cat) but not log(K_M) direction, as indicated by the red ellipses being aligned nearly along the abscissa. Combining both experiments specifies both rate parameters, as indicated by the small extent of the green ellipses.

Figure 5

Experimental design with different relative parameter errors. The violet line is the original design with 10% uncertainty. It is interesting to note that with an objective of 37% uncertainty (gold line) that all but two of the parameters can be estimated with two experiments, and all parameters can be estimated with four experiments.