Explaining deep reinforcement learning decisions in complex multiagent settings: towards enabling automation in air traffic flow management

Abstract

With the objective to enhance human performance and maximize engagement during the performance of tasks, we aim to advance automation for decision making in complex and large-scale multi-agent settings. Towards these goals, this paper presents a deep multi agent reinforcement learning method for resolving demand - capacity imbalances in real-world Air Traffic Management settings with thousands of agents. Agents comprising the system are able to jointly decide on the measures to be applied to resolve imbalances, while they provide explanations on their decisions: This information is rendered and explored via appropriate visual analytics tools. The paper presents how major challenges of scalability and complexity are addressed, and provides results from evaluation tests that show the abilities of models to provide high-quality solutions and high-fidelity explanations.

Keywords: Air traffic management; Explainability; Interpretability; Multi-agent deep reinforcement learning; Stochastic decision trees; Visualization.

Fig. 1

Evolution of daily demand in sector LECPDEO in a singe day

Fig. 2

Payoff matrices for 2X2 games

Fig. 3

DQN architecture

Fig. 4

Simulation rounds for delay regulations

Fig. 5

Selecting alternative flight plans due to level capping measures

Fig. 6

Combining level capping and ground delays measures

Fig. 7

The mimicking approach for interpretability

Fig. 8

An overview of a solution: a table view with columns corresponding to sectors and rows to time intervals

Fig. 9

An overview of a solution involving flight delays. Segmented bars of different shades of grey indicate proportions of delayed flights with different delay durations. Light grey corresponds to non-delayed flights; the darkness increases proportionally to the delay duration

Fig. 10

Comparison of three solutions with a focus on a selected sector and the sectors connected to it

Fig. 11

Components and contents of Sector Explorer in the mode of showing a single solution

Fig. 12

Hourly demands for sectors are represented by time-based histograms with overlapping bars

Fig. 13

List of flights with accumulated delays. The dynamics of delays accumulations is shown if columns “Cumulative delays” and “Added delays”, representing total and momentary delays, accordingly

Fig. 14

Decisions applied to a single flight over multiple model steps are explained by a series of sets of arguments. Arguments, intervals of feature values, and actual values for each step are shown in table rows

Fig. 15

Box plots for delay regulations

Fig. 16

Box plots for delay regulations and level capping measures

Fig. 17

Box plots for M20190622-0708 results in scenario 20190714

Fig. 18

Box plots for M20190622-0708 results in scenario 20190705

Fig. 19

Histogram of M20190622-0708 distribution of delays in 20190714

Fig. 20

Histogram for M20190622-0708 distribution of unresolved hotspots in 20190714