Multi-agent reinforcement learning with approximate model learning for competitive games

Abstract

We propose a method for learning multi-agent policies to compete against multiple opponents. The method consists of recurrent neural network-based actor-critic networks and deterministic policy gradients that promote cooperation between agents by communication. The learning process does not require access to opponents' parameters or observations because the agents are trained separately from the opponents. The actor networks enable the agents to communicate using forward and backward paths while the critic network helps to train the actors by delivering them gradient signals based on their contribution to the global reward. Moreover, to address nonstationarity due to the evolving of other agents, we propose approximate model learning using auxiliary prediction networks for modeling the state transitions, reward function, and opponent behavior. In the test phase, we use competitive multi-agent environments to demonstrate by comparison the usefulness and superiority of the proposed method in terms of learning efficiency and goal achievements. The comparison results show that the proposed method outperforms the alternatives.

Fig 1. Overview of the proposed CTRL when two adversarial teams exist.

Each team is trained independently.

Fig 2. Architectures of the CTRL with an AMLAPN.

In the training phase, the observations are sequentially processed by the actor and critic along the arrows. The gradient signals are propagated in the reverse direction of the arrows. The shaded regions represent the auxiliary prediction networks for the approximate model learning. The number of units are shown within parentheses.

Fig 3

Illustrations of the experimental environment for four scenarios: physical deception (top left), keep-away (top right), predator-prey (bottom left), and complex predator-prey (bottom right).

Fig 4

Learning curves for the four competitive scenarios: episode rewards in physical deception (top left), episode rewards in keep away (top right), episode rewards in predator-prey (bottom left), and episode rewards in complex predator-prey (bottom right) scenarios. Each bar cluster represents the converged episode reward at the end of training. The shading region is a 95% confidence interval across the different random seeds.

Fig 5

Relative performances in round-robin tournament evaluations: the performances of team A trained by four methods (a), and the performances of team B trained by four methods (b). Each bar cluster shows the score for a set of competing policies; a higher score is better for the agent.

Fig 6

Learning curves in the partial observation environments: episode rewards in predator-prey (left) and episode rewards in complex predator-prey (right) scenarios. Each bar cluster represents the converged episode reward at the end of training. The shading region is a 95% confidence interval across the different random seeds.

Fig 7

Average relative performances with partial observation in round-robin tournament evaluations: performances of team A trained by four methods (a) and performances of team B trained by four methods (b). Each bar cluster shows the score for a set of competing policies; a higher score is better for the agent.