Exploring Transfers between Earth-Moon Halo Orbits via Multi-Objective Reinforcement Learning

Abstract

Multi-Reward Proximal Policy Optimization, a multi-objective deep reinforcement learning algorithm, is used to examine the design space of low-thrust trajectories for a SmallSat transferring between two libration point orbits in the Earth-Moon system. Using Multi-Reward Proximal Policy Optimization, multiple policies are simultaneously and efficiently trained on three distinct trajectory design scenarios. Each policy is trained to create a unique control scheme based on the trajectory design scenario and assigned reward function. Each reward function is defined using a set of objectives that are scaled via a unique combination of weights to balance guiding the spacecraft to the target mission orbit, incentivizing faster flight times, and penalizing propellant mass usage. Then, the policies are evaluated on the same set of perturbed initial conditions in each scenario to generate the propellant mass usage, flight time, and state discontinuities from a reference trajectory for each control scheme. The resulting low-thrust trajectories are used to examine a subset of the multi-objective trade space for the SmallSat trajectory design scenario. By autonomously constructing the solution space, insights into the required propellant mass, flight time, and transfer geometry are rapidly achieved.

Figure 1:

Members of the Earth-Moon L₁ Lyapunov and northern halo orbit families depicted using dashed and solid arcs, respectively, and shaded by Jacobi constant.

Figure 2:

Members of the Earth-Moon L₂ Lyapunov and southern halo orbit families depicted using dashed and solid arcs, respectively, and shaded by Jacobi constant.

Figure 3:

A natural, heteroclinic transfer from an L₁ Lyapunov orbit to an L₂ Lyapunov orbit.

Figure 4:

MRPPO training N policies with k_i agents and reward functions $r_{i,t}\left({\overline{s}}_t,{\overline{u}}_t\right)$ within a shared dynamical model.

Figure 5:

1,000 evaluation trajectories departing an L₁ Lyapunov orbit to an L₂ Lyapunov orbit with an equal Jacobi constant in Scenario 1 controlled using: (a) Policy 1 and (b) Policy 8.

Figure 6:

Relationship between the propellant mass usage and the position and velocity differences for the L₁ Lyapunov to L₂ Lyapunov transfer in Scenario 1.

Figure 7:

Trajectory from a highly perturbed initial condition departing an L₁ Lyapunov orbit to an L₂ Lyapunov orbit with an equal Jacobi constant in Scenario 1 controlled using: (a) Policy 1 and (b) Policy 8.

Figure 8:

Thrust magnitude along an evaluation trajectory in Scenario 1 commanded using: (a) Policy 1 and (b) Policy 8.

Figure 9:

Position and velocity differences measured with respect to the closest state on the final orbit for a single trajectory in Scenario 1 commanded by Policy 1.

Figure 10:

Position and velocity differences measured with respect to the closest state on the final orbit for a single trajectory in Scenario 1 commanded by Policy 8.

Figure 11:

1,000 evaluation trajectories departing an L₁ northern halo orbit to an L₂ southern halo orbit with an equal Jacobi constant in the Earth-Moon CR3BP in Scenario 2 controlled using: (a) Policy 1 and (b) Policy 8.

Figure 12:

Relationship between the state discontinuities and propellant mass usage for the L₁ northern halo to L₂ southern transfer in Scenario 2.

Figure 13:

Trajectory from a highly perturbed initial condition departing an L₁ northern halo orbit to an L₂ southern halo orbit with an equal Jacobi constant in Scenario 2 controlled using: (a) Policy 1 and (b) Policy 8.

Figure 14:

Thrust magnitude along an evaluation trajectory in Scenario 2 commanded using: (a) Policy 1 and (b) Policy 8.

Figure 15:

Jacobi constant along an evaluation trajectory in Scenario 2 commanded using: (a) Policy 1 and (b) Policy 8.

Figure 16:

Position and velocity differences measured with respect to the closest state on the final orbit for a single trajectory in Scenario 2 commanded by Policy 1.

Figure 17:

Position and velocity differences measured with respect to the closest state on the final orbit for a single trajectory in Scenario 2 commanded by Policy 8.

Figure 18:

1,000 evaluation trajectories departing an L₁ northern halo orbit to an L₂ southern halo orbit with a distinct Jacobi constant in the Earth-Moon CR3BP in Scenario 3 controlled using: (a) Policy 1 and (b) Policy 8.

Figure 19:

Relationship between the state discontinuities and propellant mass usage for the L₁ northern halo to L₂ southern halo transfer with distinct Jacobi constants in Scenario 3.

Figure 20:

Trajectory from a highly perturbed initial condition departing an L₁ northern halo orbit to an L₂ southern halo orbit with a distinct Jacobi constant in Scenario 3 controlled using: (a) Policy 1 and (b) Policy 8.

Figure 21:

Jacobi constant along an evaluation trajectory in Scenario 3 commanded using: (a) Policy 1 and (b) Policy 8.

Figure 22:

Thrust magnitude along an evaluation trajectory in Scenario 3 commanded using: (a) Policy 1 and (b) Policy 8.

Figure 23:

Position and velocity differences measured with respect to the closest state on the final orbit for a single trajectory in Scenario 3 commanded by Policy 1.

Figure 24:

Position and velocity differences measured with respect to the closest state on the final orbit for a single trajectory in Scenario 3 commanded by Policy 8.