Vector-based navigation using grid-like representations in artificial agents

Abstract

Deep neural networks have achieved impressive successes in fields ranging from object recognition to complex games such as Go^1,2. Navigation, however, remains a substantial challenge for artificial agents, with deep neural networks trained by reinforcement learning^3-5 failing to rival the proficiency of mammalian spatial behaviour, which is underpinned by grid cells in the entorhinal cortex ⁶ . Grid cells are thought to provide a multi-scale periodic representation that functions as a metric for coding space^7,8 and is critical for integrating self-motion (path integration)^6,7,9 and planning direct trajectories to goals (vector-based navigation)^7,10,11. Here we set out to leverage the computational functions of grid cells to develop a deep reinforcement learning agent with mammal-like navigational abilities. We first trained a recurrent network to perform path integration, leading to the emergence of representations resembling grid cells, as well as other entorhinal cell types ¹² . We then showed that this representation provided an effective basis for an agent to locate goals in challenging, unfamiliar, and changeable environments-optimizing the primary objective of navigation through deep reinforcement learning. The performance of agents endowed with grid-like representations surpassed that of an expert human and comparison agents, with the metric quantities necessary for vector-based navigation derived from grid-like units within the network. Furthermore, grid-like representations enabled agents to conduct shortcut behaviours reminiscent of those performed by mammals. Our findings show that emergent grid-like representations furnish agents with a Euclidean spatial metric and associated vector operations, providing a foundation for proficient navigation. As such, our results support neuroscientific theories that see grid cells as critical for vector-based navigation^7,10,11, demonstrating that the latter can be combined with path-based strategies to support navigation in challenging environments.