Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Sep 14;18(9):e0291545.
doi: 10.1371/journal.pone.0291545. eCollection 2023.

Sample-efficient multi-agent reinforcement learning with masked reconstruction

Jung In Kim  1 , Young Jae Lee  1 , Jongkook Heo  1 , Jinhyeok Park  1 , Jaehoon Kim  1 , Sae Rin Lim  1 , Jinyong Jeong  1 , Seoung Bum Kim  1
Affiliations

Sample-efficient multi-agent reinforcement learning with masked reconstruction

Jung In Kim et al. PLoS One. .

Abstract

Deep reinforcement learning (DRL) is a powerful approach that combines reinforcement learning (RL) and deep learning to address complex decision-making problems in high-dimensional environments. Although DRL has been remarkably successful, its low sample efficiency necessitates extensive training times and large amounts of data to learn optimal policies. These limitations are more pronounced in the context of multi-agent reinforcement learning (MARL). To address these limitations, various studies have been conducted to improve DRL. In this study, we propose an approach that combines a masked reconstruction task with QMIX (M-QMIX). By introducing a masked reconstruction task as an auxiliary task, we aim to achieve enhanced sample efficiency-a fundamental limitation of RL in multi-agent systems. Experiments were conducted using the StarCraft II micromanagement benchmark to validate the effectiveness of the proposed method. We used 11 scenarios comprising five easy, three hard, and three very hard scenarios. We particularly focused on using a limited number of time steps for each scenario to demonstrate the improved sample efficiency. Compared to QMIX, the proposed method is superior in eight of the 11 scenarios. These results provide strong evidence that the proposed method is more sample-efficient than QMIX, demonstrating that it effectively addresses the limitations of DRL in multi-agent systems.

PubMed Disclaimer

Conflict of interest statement

The authors have declared that no competing interests exist.

Figures

Fig 1
Fig 1. Overall architecture of the proposed method, which combines QMIX with a masked reconstruction task.
Masked reconstruction task consists of a target and online network. The gray boxes represent the three recurrent networks.
Fig 2
Fig 2
(a) Overall framework of QMIX. The output values obtained from each agent network are monotonically mixed to generate a joint action value function. (b) Agent network architecture. The network takes the current observation and the last action of an individual agent as inputs and outputs the corresponding Q-value for each agent.
Fig 3
Fig 3. Comparison between M-QMIX and QMIX on all super hard maps.
Fig 4
Fig 4. Comparison between M-QMIX and QMIX on all hard maps.
Fig 5
Fig 5. Comparison between M-QMIX and QMIX on all easy maps.
Fig 6
Fig 6. Performance of M-QMIX under different masking ratios.
Fig 7
Fig 7. Performance of M-QMIX under different momentum values.
See this image and copyright information in PMC

References

    1. CHüttenrauch, M., Šošić, A., Neumann, G. Guided deep reinforcement learning for swarm systems. arXiv preprint arXiv:170906011. 2017;. 10.48550/arXiv.1709.06011 - DOI
    1. Cao Yongcan and Yu Wenwu and Ren Wei and Chen Guanrong. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics. 2012;9(1):427–438. https://ieeexplore.ieee.org/abstract/document/6303906 doi: 10.1109/TII.2012.2219061 - DOI
    1. Lipowska D, Lipowski A. Emergence of linguistic conventions in multi-agent reinforcement learning. PLoS One. 2018;13(11):e0208095.438. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0208095 - PMC - PubMed
    1. Tampuu A, Matiisen T, Kodelja D, Kuzovkin I, Korjus K, Aru J, et al.. Multiagent cooperation and competition with deep reinforcement learning. PloS one. 2017;12(4):e0172395. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0172395 - PMC - PubMed
    1. Park YJ, Cho YS, Kim SB. Multiagent cooperation and competition with deep reinforcement learning. PloS one. 2019;14(9):e0222215. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0222215 - PMC - PubMed

Publication types