Optimal control of agent-based models via surrogate modeling

Abstract

This paper describes and validates an algorithm to solve optimal control problems for agent-based models (ABMs). For a given ABM and a given optimal control problem, the algorithm derives a surrogate model, typically lower-dimensional, in the form of a system of ordinary differential equations (ODEs), solves the control problem for the surrogate model, and then transfers the solution back to the original ABM. It applies to quite general ABMs and offers several options for the ODE structure, depending on what information about the ABM is to be used. There is a broad range of applications for such an algorithm, since ABMs are used widely in the life sciences, such as ecology, epidemiology, and biomedicine and healthcare, areas where optimal control is an important purpose for modeling, such as for medical digital twin technology.

Fig 1. Summary of the key steps involved in using ODE surrogate models for control.

For an ABM control problem, we first create an ODE surrogate (Step 1). Next, we apply control techniques to the ODE system (Step 2) and then transfer the control solution back to the original ABM (Step 3). The snapshots in the ABM panel depict simulations performed in NetLogo [6] of tumor [7], slime mold [7], and wolf-sheep predation models [8].

Fig 2. Correspondence between ABM and ODE model components.

The aggregation of agents by type or attributes defines the state variables X_i ≡ X_i(t) (i ∈ {1,2,…,n}) of an ODE approximation. Similarly, all interactions, events, and rules in an ABM define the processes of an ODE model. Depending on an ABM’s structure, its environment can either be transformed into state variables or contributes to the processes of an ODE approximation.

Fig 3. Model reduction from the sheep-wolves-grass ABM to a mechanistic ODE surrogate model.

In the sheep-wolves-grass ABM, the energy of sheep (E_S) and wolf (E_W) agents prevents (inhibits) their death. When an agent’s energy reaches zero, it dies. Offspring generation occurs at each time step and depends only on the probability of reproduction. After reproduction, energy is split between the parent and offspring. To create a mechanistic mass-action ODE model, the inhibitory effect of energy on agent death was reassigned to a positive effect on population growth. In the ABM, W, S, and G represent wolf, sheep, and grass agents, while Z, Y, and X represent their respective populations in the ODE model.

Fig 4. Comparison of the effectiveness of different ODE surrogate models for solving the sheep-wolves-grass ABM control problem.

The black cross marks the near-optimal solution (κ₂ = 0.83% and κ₃ = 0.45% per time step) for the sheep-wolves-grass ABM control problem as determined by a grid search (with a step of 0.0001 in both dimensions). Orange dots indicate suboptimal control solutions within one standard deviation from the target (a steady state with 50% fewer wolves and 10% more sheep compared to the original steady state). Blue and red dots show the control parameter values associated with the ODE surrogate models that have been calibrated against datasets I and II and datasets I-V, respectively. The best solutions were obtained for surrogate models parameterized with datasets containing control information (III-V). However, all four of these surrogate models (red dots) identified control solutions approximately equidistant from the optimal one.

Fig 5. Metabolic network ABM representations.

(A) The macroscopic representation of the ABM is a simplification of (B) the microscopic representation. In the ABM, all reactions are modeled at the microscopic or elementary level, as depicted in (B). (C) The macroscopic representation when the model is used in continuous mode, where a constant inflow of agents S occurs while all metabolites are removed at a constant rate. In (B), all pairwise interactions and complex decompositions are modeled with different probabilities. Two agents can only interact when present at the same grid point.

Fig 6. Comparison of the effectiveness of different ODE surrogate models for solving the metabolic pathway ABM control problem.

The red square shows the optimal inflow point and the corresponding mean loss function value as determined for the ABM by a grid search of inflows between 0 and 1 with a step size of 0.1, where in each step 100 simulations runs were averaged. The red line highlights the mean of each of the 100 simulation runs of the ABM, and the area between the orange lines is the 68% confidence interval. Circles denote the predicted optimal inflow and corresponding loss function value for each ODE surrogate. ODE models that did not exhibit a minimum within the [0,1] domain have their domain of integrability shown with a line. The line depicts the range of loss function values predicted by the approximation. Panel A shows a zoomed-in version of panel B, focusing on the best-performing ODE surrogate models (GMA and mechanistic approximations). Panel B shows the results for all surrogate models. The S-system I performed worst, as it could only be integrated between 0.8 and 1.0, and in that range predicted loss function values between 8 and 9. While S-system C, Quad I, and Linear I, all resulted in models with a larger domain over which they could be integrated, neither had a minimum within their respective domains. GMA I was the ODE surrogate that predicted an optimal inflow of substrate closest to the ABM optimum, and Mech. I best predicted the loss function value of the ABM at the optimal inflow point.