Reinforcement Learning for Radiotherapy Dose Fractioning Automation

Abstract

External beam radiotherapy cancer treatment aims to deliver dose fractions to slowly destroy a tumor while avoiding severe side effects in surrounding healthy tissues. To automate the dose fraction schedules, this paper investigates how deep reinforcement learning approaches (based on deep Q network and deep deterministic policy gradient) can learn from a model of a mixture of tumor and healthy cells. A 2D tumor growth simulation is used to simulate radiation effects on tissues and thus training an agent to automatically optimize dose fractionation. Results show that initiating treatment with large dose per fraction, and then gradually reducing it, is preferred to the standard approach of using a constant dose per fraction.

Keywords: automatic treatment planning; cellular simulation; reinforcement learning.

Figure 1

Agent–environment interaction where the environment is an in silico simulation of a tumor within its surrounding tissues. At each time step t, the agent takes an action $a_t$, which represents the dose and the environment transitions from $s_t$ to $s_{t+1}$, while providing a reward $r_t$ as well as an observation ${\omega }_{t+1}$.

Figure 2

Observation of the 2D model where green pixels are healthy cells, red pixels are tumor cells, and black pixels are empty volumes. The intensities are proportional to the density of cells.

Figure 3

Shape of treatments found by the agent for the collapsed toy environment (in gray) compared with the baseline treatment (in black). Each point represents a dose delivered to the patients.

Figure 4

Shape of treatments found by the agents for the 2D environment (in gray) using the DQN algorithm (top) and the DDPG algorithm (bottom). The baseline is shown (in black) for comparison. Each point represents a dose delivered to the patients.

Figure 5

Effects of the treatments prescribed by the DQN agents on the 2D simulation using the two studied reward functions (K and KD). Green pixels are healthy cells, red pixels are tumor cells, and black pixels are empty volumes.

Figure 6

Effects of the treatments prescribed by the DDPG agents on the 2D simulation using the two studied reward functions (K and KD). Green pixels are healthy cells, red pixels are tumor cells, and black pixels are empty volumes.