Reinforcement learning-based dynamic field exploration and reconstruction using multi-robot systems for environmental monitoring

Abstract

In the realm of real-time environmental monitoring and hazard detection, multi-robot systems present a promising solution for exploring and mapping dynamic fields, particularly in scenarios where human intervention poses safety risks. This research introduces a strategy for path planning and control of a group of mobile sensing robots to efficiently explore and reconstruct a dynamic field consisting of multiple non-overlapping diffusion sources. Our approach integrates a reinforcement learning-based path planning algorithm to guide the multi-robot formation in identifying diffusion sources, with a clustering-based method for destination selection once a new source is detected, to enhance coverage and accelerate exploration in unknown environments. Simulation results and real-world laboratory experiments demonstrate the effectiveness of our approach in exploring and reconstructing dynamic fields. This study advances the field of multi-robot systems in environmental monitoring and has practical implications for rescue missions and field explorations.

Keywords: dynamic field reconstruction; environmental monitoring; mobile sensor networks; multi-robot systems; reinforcement learning; source seeking.

FIGURE 1

A symmetric formation composed of four mobile robots $r_i^k,i=\mathrm{0,1,2,3}$ shown in blue. The formation center $r_c^k$ is shown in red. The distance between each robot and the formation center is $\mathrm{\Delta }r$ . The shaded region is the time-varying view-scope $\mathrm{\Gamma }(k)$ .

FIGURE 2

Flow chart showing the key components of our algorithm and two operation modes of the robot formation.

FIGURE 3

The PPO neural network architecture used in the source mapping mode.

FIGURE 4

A $3\times 3$ section of the discretized advection-diffusion field.

FIGURE 5

A set of 15 simulation environments, each consisting of one to four diffusion fields centered at various locations with distinct diffusion and advection coefficients.

FIGURE 6

Sample of a diffusion field environment over different time steps.

FIGURE 7

(A) Stage 1: Pre-training PPO on static fields. (B) Stage 2: Training PPO on dynamic fields. (C) Samples of training environments and generated trajectories post-training.

FIGURE 8

Field Re-partitioning Behavior of the Exploration Module using K-Mean Clustering. The red dot is the starting position of the formation center. Partitioning happens at the beginning of an episode and whenever a new source is detected. (A) The trajectory of the formation center. (B) The initial partition of the field. (C) The updated partition after the first source on the upper right corner is detected. (D) The updated partition after the second source on the lower left corner is detected.

FIGURE 9

Generated trajectories obtained from simulations across different spatial-diffusion environments. The robot formation center is initially located at $r_c=(\mathrm{90,10})$ - shown as the red marker, and the red square shows the position of the formation the end of the episode.

FIGURE 10

Analysis of the generated trajectory and mapping errors of our solution in comparison with the Lawn Mowing and Random Walking approaches.

FIGURE 11

Mobile robot formation setup for real-world testing and evaluation.

FIGURE 12

(a) Lab experiment setup with a projected dynamic field and four mobile robots. (b) Snapshots of the trajectories of the robot formation at three different time steps in an experiment. The red dashed lines represent the trajectories.

FIGURE 13

The field exploration and reconstruction results in two experiments with two and three diffusion sources. “Environment Field End State” figures illustrate the end states of the two experiments with corresponding trajectories of the robot formation. The red dots indicate the starting locations of the formation center and the red squares are the ending locations of the formation. “Agent Field End State” figures show the end states of the reconstructed fields in the two experiments. “Concentration” figures illustrate the estimated field concentration along the trajectories of the formation center, and “Mapping Error” figures show the mapping errors while reconstructing the fields.

FIGURE 14

Screenshot captured from the experiment on the first environment with two diffusion fields.

FIGURE 15

Screenshot captured from the experiment on the second environment with three diffusion fields.