Federated Learning Adaptive Dynamic Programming for Massive Multiagent Mean-Field Games-Based Optimal Consensus

Abstract

Massive multiagent systems typically involve a very large number of interactions and conflicts of interest among agents, which presents a significant challenge for achieving stable and efficient adaptive optimal consensus control in real-time. To fill this gap, this article develops a novel federated learning adaptive dynamic programming (FL-ADP) control scheme to solve the massive multiagent mean-field games (MFGs)-based optimal consensus problem. First, the complex interactions of each individual agent with all other agents can be approximated by an average or collective influence in the context of MFGs. Then, a novel undiscounted performance index function involving the mean-field coupling term, the tracking errors and their derivatives is proposed to circumvent the potential impact of the improper discount factor selection and achieve better control performance. By designing the critic-mass neural network structure, the coupled Hamilton-Jacobi-Bellman and Fokker-Planck-Kolmogorov equations are solved to derive the approximate optimal control policy and quantify the probability density function of the collective behavior simultaneously. Additionally, to comply with the required convergence condition of the MFG, a novel event-triggered federated learning mechanism is formulated, which achieves a balance between communication resource consumption and the guarantee of algorithm convergence. On the basis of Lyapunov's direct method, the tracking errors and the weight estimation errors of all agents are guaranteed to be uniformly ultimately bounded. Simulation results of massive multi-uncrewed aerial vehicle systems affirm the rationality and effectiveness of the proposed method.