Universally sloppy parameter sensitivities in systems biology models

Abstract

Quantitative computational models play an increasingly important role in modern biology. Such models typically involve many free parameters, and assigning their values is often a substantial obstacle to model development. Directly measuring in vivo biochemical parameters is difficult, and collectively fitting them to other experimental data often yields large parameter uncertainties. Nevertheless, in earlier work we showed in a growth-factor-signaling model that collective fitting could yield well-constrained predictions, even when it left individual parameters very poorly constrained. We also showed that the model had a "sloppy" spectrum of parameter sensitivities, with eigenvalues roughly evenly distributed over many decades. Here we use a collection of models from the literature to test whether such sloppy spectra are common in systems biology. Strikingly, we find that every model we examine has a sloppy spectrum of sensitivities. We also test several consequences of this sloppiness for building predictive models. In particular, sloppiness suggests that collective fits to even large amounts of ideal time-series data will often leave many parameters poorly constrained. Tests over our model collection are consistent with this suggestion. This difficulty with collective fits may seem to argue for direct parameter measurements, but sloppiness also implies that such measurements must be formidably precise and complete to usefully constrain many model predictions. We confirm this implication in our growth-factor-signaling model. Our results suggest that sloppy sensitivity spectra are universal in systems biology models. The prevalence of sloppiness highlights the power of collective fits and suggests that modelers should focus on predictions rather than on parameters.

Figure 1. Parameter Sensitivity Spectra

The quantities we calculate from H ^χ2 are illustrated in (A), while (B) and (C) show that all the models we examined have sloppy sensitivity spectra. (A) Analyzing H ^χ2 corresponds to approximating the surfaces of constant model behavior change (constant χ ²) as ellipsoids. The width of each principal axis is proportional to one over the square root of the corresponding eigenvalue. The inner ellipsoid's projection onto and intersection with the θ ₁ axis are denoted by P ₁ and I₁, respectively. (B) Plotted are the eigenvalue spectra of H ^χ2 for our collection of systems biology models. The many decades generally spanned indicate that the ellipses have a very large aspect ratio. (The spectra have each been normalized by their largest eigenvalue. Not all values are visible for all models.) (C) Plotted is the spectrum of I / P for each parameter in each model in our collection. Generally very few parameters have I / P ≈ 1, suggesting that the ellipses are skewed from the bare parameter axes. (Not all values are visible for all models.) The models are plotted in order of increasing number of free parameters and are: (a) eukaryotic cell cycle [28], (b) Xenopus egg cell cycle [29], (c) eukaryotic mitosis [30], (d) generic circadian rhythm [31], (e) nicotinic acetylcholine intra-receptor dynamics [32], (f) generic kinase cascade [33], (g) Xenopus Wnt signaling [34], (h) Drosophila circadian rhythm [35], (i) rat growth-factor signaling [21], (j) Drosophila segment polarity [36], (k) Drosophila circadian rhythm [37], (l) Arabidopsis circadian rhythm [2], (m) in silico regulatory network [25], (n) human purine metabolism [38], (o) Escherichia coli carbon metabolism [39], (p) budding yeast cell cycle [40], (q) rat growth-factor signaling [41].

Figure 2. Sloppiness and Uncertainties

As in Figure 1A, the contours represent surfaces of constant model behavior deviation. The clouds of points represent parameter set ensembles. (A) Collective fitting of model parameters naturally constrains the parameter set ensemble along stiff directions and allows it to expand along sloppy directions. The resulting ensemble may be very large, yet encompass little variation in model behavior, yielding small prediction uncertainties despite large parameter uncertainties. (Σ₁ denotes the 95% confidence for the value of θ ₁.) (B) If all parameters are directly measured to the same precision, the parameter set ensemble is spherical. The measurement precision required for well-constrained predictions is set by the stiffest direction. (C) If one parameter (here θ ₂) is known less precisely than the rest, the cloud is ellipsoidal. If not aligned with a sloppy direction, the cloud will admit many model behaviors and yield large prediction uncertainties. (Note that the aspect ratio of the real contours can be greater than 1,000.)

Figure 3. Fitting Parameters to Idealized Data

Shown are histograms of the relative confidence interval size Σ for each parameter in each model of our collection, after fitting 100 times as many time-series data points (each with 10% uncertainty) as parameters. In most cases, a large number of parameters are left with greater than 100% uncertainty. (A parameter constrained with 95% probability to lie between 1 and 100 would have Σ ≈ 100.) Labels are as in Figure 1.

Figure 4. Parameter and Prediction Uncertainties

(A) Our example prediction is for ERK activity upon EGF stimulation given PI3K inhibition in this 48-parameter model of growth-factor signaling in PC12 cells [21]. (B) Shaded regions are 95% confidence intervals calculated via exhaustive Monte Carlo for our example prediction given various scenarios for constraining parameter values. (C) Plotted is the relative size Σ of the 95% confidence interval for each parameter. The scenarios represented are: (red, squares) all model parameters individually measured to high precision, (blue, triangles) all parameters precisely measured, except one estimated to low precision, (yellow, circles) all parameters collectively fit to 14 real cell-biology experiments. Precisely measured individual parameter values enable a tight prediction, (B) middle red band; but even one poorly known parameter can destroy predictive power, (B) wide blue band. In contrast, the collective fit yields a tight prediction, (B) tightest yellow band; but only very loose parameter constraints, (C) circles. The large parameter uncertainties from the collective fit, (C) circles, calculated here by Monte Carlo are qualitatively similar to those seen in the linearized fit to idealized data (Figure 3, model (i)). (For clarity, the dashed red lines trace the boundary of the red confidence interval.)