Chemotactic navigation in robotic swimmers via reset-free hierarchical reinforcement learning

Abstract

Microorganisms have evolved diverse strategies to propel themselves in viscous fluids, navigate complex environments, and exhibit taxis in response to stimuli. This has inspired the development of miniature robots, where artificial intelligence (AI) is playing an increasingly important role. Can AI endow these synthetic systems with intelligence akin to that honed through natural evolution? Here, we demonstrate, in silico, chemotactic navigation in a multi-link robotic model using two-level hierarchical reinforcement learning (RL). The lower-level RL allows the model-configured as a chain or ring topology-to acquire topology-adapted swimming gaits: wave propagation characteristic of flagella or body oscillation akin to an amoebae. Such chain and ring swimmers, further enabled by the higher-level RL, accomplish chemotactic navigation in prototypical biologically relevant scenarios that feature conflicting chemoattractants, pursuing a swimming bacterial mimic, steering in vortical flows, and squeezing through tight constrictions. Additionally, we achieve reset-free RL under partial observability, where simulated robots rely solely on local scalar observations rather than global or vectorial data. This advancement illuminates potential solutions for overcoming persistent challenges of manual resets and partial observability in real-world microrobotic RL.

Fig. 1. Two-level hierarchical RL framework for in silico chemotactic navigation of multi-link robotic models in viscous fluids.

The model adopts either a chain (a) or ring (b) topology. c Workflow of the lower-level RL: training simulated robots to acquire primitive swimming skills—propulsion and reorientation. d Workflow of the higher-level RL: training robots to steer up chemoattractant gradients.

Fig. 2. Learning primitive swimming skills: propulsion and reorientation.

a A chain swimmer achieves propulsion through symmetric beating, with vectors indicating the flow field. The inset shows the horizontal (x) and vertical (y) displacements of the swimmer’s tip, resembling a traveling transverse wave. b The chain swimmer reorients through asymmetric actuation. c A ring swimmer propels by periodically contracting and expanding its body. The inset displays the x and y displacements of the swimmer’s front, typical of a longitudinal wave propagating horizontally. d Evolution of front-rear pressure differences during the policy iteration process: training the chain swimmer for propulsion (purple solid line) and reorientation (purple dashed), and the ring model for propulsion (green solid in the inset). This quantity is time-averaged over one policy. e, f Similar to d, but for the rotational rate and centroid translational velocity of the swimmers, respectively. The latter reaches a plateau value (dashed line), $\mathcal{U}/\left(\widehat{\mathrm{\Omega }}L\right)$, representing the swimming speed of simulated agents once training has converged. Source data are provided as a Source Data file.

Fig. 3. Chemotaxis towards a stationary chemical source with and without chemical disturbances.

a A chain swimmer initially at (0, 0) swims toward the target source at (4, 0). b Similar to a, but with a disturbing chemical source (star symbol) at (0, 4) featuring strengths of Q_d = 0.2Q (left panel) and 0.6Q (right panel), respectively. Here, ΔC signifies the difference in the chemoattractant concentration produced by the target and disturbance sources. c Akin to a, but for a ring model. d Analogous to b, but for a ring swimmer, with the disturbing source strengths of Q_d = 0.6Q (left panel) and 0.8Q (right panel), respectively. Source data is provided as a Source Data file.

Fig. 4. Simulated robotic swimmers pursue a dynamic chemical source of strength Q.

a–c A chain swimmer chasing (a), encountering (b), and following (c) a chemical source (symbolized by a star) moving along a circular orbit. d–f Similar to a–c, but for a ring swimmer and a source executing a figure-eight-shaped path. g Trajectories of chain (left column) and ring (right column) swimmers pursuing a source at varying speeds. Source data are provided as a Source Data file.

Fig. 5. Chemotactic navigation in periodic cellular vortices with a vortical velocity U_v.

Swimmers start from (0, 0) with the goal of reaching a chemical source positioned at (4, 4). a, b A chain model swims in vortices with $U_{\mathrm{v}}=0.005\widehat{\mathrm{\Omega }}L$ and $U_{\mathrm{v}}=0.03\widehat{\mathrm{\Omega }}L$, respectively. The background color denotes the dimensionless flow vorticity. The inset of A depicts the typical streamlines. c Statistics of successful chemotaxis versus the vortical strength U_v, where crosses and circles signify failures and successes, respectively. d, e similar to a and b, but for a ring swimmer, with $U_{\mathrm{v}}=0.0005\widehat{\mathrm{\Omega }}L$ and $U_{\mathrm{v}}=0.005\widehat{\mathrm{\Omega }}L$, respectively. f Similar to c, while for a ring swimmer. Source data is provided as a Source Data file.

Fig. 6. Chemotactic navigation through a narrow constriction.

a The swimmer navigates a constriction of height d to reach the chemical source located on the opposite side. The source and locomotion are within the same xy-plane. A chain swimmer successfully traverses a constriction of d = 0.3L (b), but fails to pass through a narrower one of d = 0.2L (c). d Similar to b, but for a ring swimmer, which squeezes through the narrow gap of d = 0.5L/π, where L/π denotes the effective diameter of ring models. e analogous to d, but with a tighter constriction of d = 0.4L/π, which obstructs the swimmer. Source data is provided as a Source Data file.