Model validation and selection in metabolic flux analysis and flux balance analysis

Abstract

13C-Metabolic Flux Analysis (13C-MFA) and Flux Balance Analysis (FBA) are widely used to investigate the operation of biochemical networks in both biological and biotechnological research. Both methods use metabolic reaction network models of metabolism operating at steady state so that reaction rates (fluxes) and the levels of metabolic intermediates are constrained to be invariant. They provide estimated (MFA) or predicted (FBA) values of the fluxes through the network in vivo, which cannot be measured directly. These fluxes can shed light on basic biology and have been successfully used to inform metabolic engineering strategies. Several approaches have been taken to test the reliability of estimates and predictions from constraint-based methods and to compare alternative model architectures. Despite advances in other areas of the statistical evaluation of metabolic models, such as the quantification of flux estimate uncertainty, validation and model selection methods have been underappreciated and underexplored. We review the history and state-of-the-art in constraint-based metabolic model validation and model selection. Applications and limitations of the χ² -test of goodness-of-fit, the most widely used quantitative validation and selection approach in 13C-MFA, are discussed, and complementary and alternative forms of validation and selection are proposed. A combined model validation and selection framework for 13C-MFA incorporating metabolite pool size information that leverages new developments in the field is presented and advocated for. Finally, we discuss how adopting robust validation and selection procedures can enhance confidence in constraint-based modeling as a whole and ultimately facilitate more widespread use of FBA in biotechnology.

Keywords: constraint-based modeling; flux balance analysis; metabolic flux analysis; metabolic modeling; model selection; model validation.

Figure 1:

Graphical summary of validation strategies in (A) FBA and (B) 13C-MFA. Dotted lines connect inputs with the associated validation technique(s). (A) FBA predictions can be validated by comparing growth rate or growth/no-growth phenotypes across different substrates, growth conditions, or sets of gene knockouts in silico and in vivo. Values can be calculated from flux maps and compared with experimental measurements. FBA internal flux predictions can be compared with 13C-MFA fluxes. (B) Values can be calculated from 13C-MFA flux maps and compared with an independent experimental measurement from the in vivo system. Goodness-of-fit can be assessed between simulated and measured MIDs, and simulated and measured metabolite pool sizes in INST-MFA. Flux maps can be compared with the results of independent modeling exercises. Molecules are schematically shown as connected circles of atomic positions: open circles are unlabeled, and filled circles are isotopically labeled. Abbreviations: Mn - metabolites in the metabolic network; S_n – exogenous substrates; V_i – Fluxes; [M_n] – metabolite concentrations.

Figure 2:

Approaches to model selection for 13C-MFA. Metabolic network models 1–3 having increasing complexity are compared. Model 2 in this example is the correct description of the network. (A) Labeling data (MID1 & MID2) are gathered and, for each model, agreement between model output and these data is optimized. The χ²-test of goodness-of-fit is used to assess each model fit and these model fits are ranked 1^st, 2^nd, or 3^rd, with the 1^st passing the test by the widest margin and being selected as the most statistically well-supported model. (B) Labeling data are split into “training” and “testing” subsets and agreement between model output and the “training” data is optimized. The Sum-of-Squared Residuals (SSR) is then calculated for each model from the deviation between its output and the “testing” data. The model fits are then ranked 1^st, 2^nd, and 3^rd, with the 1^st having the lowest SSR and being selected. (C) Labeling data and metabolite pool data (C1 and C2) are gathered and split into “training” and “testing” subsets. For each model, agreement between model output and these data is optimized. The Sum-of-Squared Residuals (SSR) is then calculated for each model from the deviation between its output and the “testing” data. The model fits are then ranked 1^st, 2^nd, and 3^rd, with the 1^st having the lowest SSR and being selected. The inclusion of metabolite pool size data into both the “fitting” and “testing” datasets provides more data to go off when evaluating goodness-of-fit, potentially increasing the likelihood of identifying the correct model from a set of alternatives.