Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Mar 26;22(7):2554.
doi: 10.3390/s22072554.

Challenges of Large-Scale Multi-Camera Datasets for Driver Monitoring Systems

Juan Diego Ortega  1   2 , Paola Natalia Cañas  1 , Marcos Nieto  1 , Oihana Otaegui  1 , Luis Salgado  3
Affiliations

Challenges of Large-Scale Multi-Camera Datasets for Driver Monitoring Systems

Juan Diego Ortega et al. Sensors (Basel). .

Abstract

Tremendous advances in advanced driver assistance systems (ADAS) have been possible thanks to the emergence of deep neural networks (DNN) and Big Data (BD) technologies. Huge volumes of data can be managed and consumed as training material to create DNN models which feed functions such as lane keeping systems (LKS), automated emergency braking (AEB), lane change assistance (LCA), etc. In the ADAS/AD domain, these advances are only possible thanks to the creation and publication of large and complex datasets, which can be used by the scientific community to benchmark and leverage research and development activities. In particular, multi-modal datasets have the potential to feed DNN that fuse information from different sensors or input modalities, producing optimised models that exploit modality redundancy, correlation, complementariness and association. Creating such datasets pose a scientific and engineering challenge. The BD dimensions to cover are volume (large datasets), variety (wide range of scenarios and context), veracity (data labels are verified), visualization (data can be interpreted) and value (data is useful). In this paper, we explore the requirements and technical approach to build a multi-sensor, multi-modal dataset for video-based applications in the ADAS/AD domain. The Driver Monitoring Dataset (DMD) was created and partially released to foster research and development on driver monitoring systems (DMS), as it is a particular sub-case which receives less attention than exterior perception. Details on the preparation, construction, post-processing, labelling and publication of the dataset are presented in this paper, along with the announcement of a subsequent release of DMD material publicly available for the community.

Keywords: ADAS; automotive; datasets; driver monitoring; multi-camera.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
Examples of activities performed in the DMD.
Figure 2
Figure 2
General overview of the DMD creation process.
Figure 3
Figure 3
Metadata label taxonomy for the DMD.
Figure 4
Figure 4
DMD camera setup and recording environments.
Figure 5
Figure 5
Participants information.
Figure 6
Figure 6
DMD material weight in raw and video count.
Figure 7
Figure 7
Process for multi-sensor stream synchronization: (a) Region of interest and signal extraction, (b) Processing of temporal signal and correlation calculation. Blue signal is computed from face and hands ROI in face and body camera, respectively. Red signal is computed from face and hands ROI in body and hands camera, respectively.
Figure 8
Figure 8
DMD distribution for: (a) Driver_actions (b) Hands_using_wheel (c) Talking (d) Hand_on_gear (e) Objects_in_scene (f) Gaze_on_road.
Figure 9
Figure 9
Procedure to distribute the DMD.
Figure 10
Figure 10
DMD file structure.
See this image and copyright information in PMC

References

    1. Wang W., Liu C., Zhao D. How much data are enough? A statistical approach with case study on longitudinal driving behavior. IEEE Trans. Intell. Veh. 2017;2:85–98. doi: 10.1109/TIV.2017.2720459. - DOI
    1. Terzi R., Sagiroglu S., Demirezen M.U. Big Data Perspective for Driver/Driving Behavior. IEEE Intell. Transp. Syst. Mag. 2020;12:20–35. doi: 10.1109/MITS.2018.2879220. - DOI
    1. Saab . Saab Driver Attention Warning System. The Saab Network; Brussels, Belgium: 2007.
    1. Toyota Motor Corporation . Toyota Enhances Pre-Crash Safety System with Eye Monitor. Toyota Motor Corporation; Aichi, Japan: 2008.
    1. Volvo Car Group . Volvo Cars Conducts Research into Driver Sensors in Order to Create Cars That Get to Know Their Drivers. Volvo Car Group; Gothenburg, Sweden: 2014.