Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Jun 12;14(1):13579.
doi: 10.1038/s41598-024-63731-3.

Virtual/augmented reality-based human-machine interface and interaction modes in airport control towers

Sara Bagassi  1 , Marzia Corsi  2 , Francesca De Crescenzio  2 , Ramona Santarelli  3 , Aurora Simonetti #  4 , Laura Moens #  4 , Michela Terenzi #  4
Affiliations

Virtual/augmented reality-based human-machine interface and interaction modes in airport control towers

Sara Bagassi et al. Sci Rep. .

Abstract

The concept of an innovative human-machine interface and interaction modes based on virtual and augmented reality technologies for airport control towers has been developed with the aim of increasing the human performances and situational awareness of air traffic control operators. By presenting digital information through see-through head-mounted displays superimposed over the out-of-the-tower view, the proposed interface should stimulate controllers to operate in a head-up position and, therefore, reduce the number of switches between a head-up and a head-down position even in low visibility conditions. This paper introduces the developed interface and describes the exercises conducted to validate the technical solutions developed, focusing on the simulation platform and exploited technologies, to demonstrate how virtual and augmented reality, along with additional features such as adaptive human-machine interface, multimodal interaction and attention guidance, enable a more natural and effective interaction in the control tower. The results of the human-in-the-loop real-time validation exercises show that the prototype concept is feasible from both an operational and technical perspective, the solution proves to support the air traffic controllers in working in a head-up position more than head-down even with low-visibility operational scenarios, and to lower the time to react in critical or alerting situations with a positive impact on the human performances of the user. While showcasing promising results, this study also identifies certain limitations and opportunities for refinement, aimed at further optimising the efficacy and usability of the proposed interface.

Keywords: Air traffic control; Airport control tower; Augmented reality; Human machine interface; Multimodal interaction; Safety nets.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

Figure 1
Figure 1
RETINA concept—an operator with the HMD and the displayed information seen superimposed through the window.
Figure 2
Figure 2
Validation platform implemented in the CAVE-like virtual environment of the University of Bologna.
Figure 3
Figure 3
Meteorological aerodrome report (METAR) interface.
Figure 4
Figure 4
Aircraft tracking labels, permanent information is reported on the first line of the label, adaptive information on the second one.
Figure 5
Figure 5
Aircraft tracking labels and airport layout overlays in low visibility conditions (CONDI VIS 3), the colour of the runway follows the same coding of the aircraft TL.
Figure 6
Figure 6
ATCOs can simultaneously see both the out of the tower view and the AR overlays through the head mounted smart glasses (Microsoft HoloLens2). (a) Personal view of the GND controller during the Air Gestures solution, the blue buttons allow the ATCO to issue Push back and Start-up clearances to the pseudo-pilot. (b) Personal view of the RWY controller during the Safety Net solution exercise.
Figure 7
Figure 7
The validation platform consists of two air traffic controller posts that communicate to a pseudo-pilot post. The platform can simulate any airport environment in different visibility conditions by means of a full 4D model system exchanging data with five subsystems: out of the tower view generator (OOT), ground augmented reality overlay application (GND App), runway augmented reality overlay application (RWY App), head down equipment (HDE) and pseudo-pilot application (PP App).
Figure 8
Figure 8
Average years of experience of the ATCOs involved in the campaign.
Figure 9
Figure 9
Average age distribution of the ATCOs involved in the campaign.
Figure 10
Figure 10
Share of time spent head-down/head-up by the user in Reference and Solution scenario—Tracking Labels exercises. Average values.
Figure 11
Figure 11
Share of time spent head-down/head-up by the user in Reference and Solution scenario—Tracking Labels exercises in good visibility condition (CONDI VIS 1). Average values.
Figure 12
Figure 12
Share of time spent head-down/head-up by the user in Reference and Solution scenario—Air Gestures exercises. Average values.
Figure 13
Figure 13
Share of time spent head-down/head-up by the user in Reference and Solution scenario—Safety Nets exercises. Average values.
Figure 14
Figure 14
Number of vocal communications in Reference and Solution scenario. Average values.
Figure 15
Figure 15
Time to react to a safety event in Reference and Solution scenario. Average values.
Figure 16
Figure 16
Bedford scale—average workload in Reference and Solution scenario—Tracking Labels exercise.
Figure 17
Figure 17
Bedford scale—average workload in Reference and Solution scenario—Safety Nets exercise.
Figure 18
Figure 18
Average physical workload in Reference and Solution scenario—Tracking Labels exercise.
Figure 19
Figure 19
Average physical workload in Reference and Solution scenario—Safety Nets exercise.
Figure 20
Figure 20
China Lake scale—average situation awareness in Reference and Solution scenario—Tracking Labels exercise.
Figure 21
Figure 21
China Lake scale—average situation awareness in Reference and Solution scenario—Safety Nets exercise.
Figure 22
Figure 22
Team situation awareness in solution scenario—Tracking Labels and Safety Nets exercises.
Figure 23
Figure 23
CARS scale—average acceptance level in Solution scenario—Tracking Labels and Safety Nets exercises.
See this image and copyright information in PMC

Similar articles

See all similar articles

References

    1. Pinska, E. An investigation of the head-up time at tower and ground control positions. In Proceeding of the 5th Eurocontrol Innovative Research Workshop. 81–86 (2006).
    1. Pinska, E. & Tijus, C. Augmented reality technology for control tower analysis of applicability based on the field study. In Proceeding of 1 st CEAS European Air and Space Conference. 573–580 (Citeseer, 2007).
    1. Papenfuss A, et al. Assessing operational validity of remote tower control in high-fidelity tower simulation. IFAC Proc. Vol. 2010;43:117–122. doi: 10.3182/20100831-4-FR-2021.00022. - DOI
    1. Papenfuss, A. & Friedrich, M. Head up only-a design concept to enable multiple remote tower operations. In 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC). 1–10 (IEEE, 2016).
    1. Hilburn, B. Head-Down Time in Aerodrome Operations: A Scope Study. (Center for Human Performance Research, 2004).