DMD – Driving Monitoring Dataset

A multi-modal dataset for different

driver monitoring scenarios

Today’s driving accidents are mainly due to human error. To make roads safer, we present: DMD, a multi-modal dataset that can contribute to the development of Driver Monitoring Systems; this way, we can rely on technology and put a brake on these undesirable statistics.

23

Times

Is the increase in accident risk for Distracted Driving.

-WHO-

ABOUT

Driver Monitoring Dataset

The DMD was devised to address different scenarios in which driver monitoring must be essential in the context of automated vehicles SAE L2-3

Lights, Camera, Action!

  • Variation in the material was one of our priorities:
  • 27% were women and 73% were men.
  • The average age of the participants is 30 years (all over 18 years old).
  • 10 of the participants wore glasses, some were recorded with and without wearing them.
  • The recordings present variation in lightning angle conditions according to the time the session was recorded (morning or afternoon)
  • Some activities were recorded with the car moving, the car stopped and in a second simulation environment.

A lot of data

The DMD dataset was designed specifically to cover the lack of data for a fully operational DMS

 

Material X-Rays

  • These are a few facts about our dataset:
  • There is about 25TB of raw data!
  • All recordings were grouped by sessions, each one determined by a protocol of activities, a participant, an environment and a lighting condition.
  • In each session there were 3 cameras recording simultaneously: Body, Face and Hands camera.
  • Each camera captures 3 channels of information: RBG, Infrared and Depth.
  • All material comes with VCD annotations of different modalities.

FEATURES

Activities Included

DMD Annotations are in VCD (Video Content Description). It supports the description of scenes with spatio-temporal object annotations, and semantics with actions, events and relations between them.

Here there are a few of those annotations available in the DMD.

Distraction-Related:

  • Talking on the phone (L/R)
  • Texting (L/R)
  • Drinking
  • Talking to passenger
  • Reaching behind/side
  • Operating the radio
  • Hair and MakeUp

Wheel-Related:

  • Hands on the wheel (quiet/moving)
  • Hands of the wheel
  • Give control to the vehicle

Gaze-Related:

  • Stare at defined region inside the car
  • Front & left front
  • Left/right Window
  • Rear-view mirrow
  • Left-right side-view mirror
  • Radio zone
  • Lap zone

Fatigue-Related:

  • Yawning with/out hand
  • Microsleep
  • Sleepy driving

FEATURES

Activities Included

DMD Annotations are in VCD (Video Content Description). It supports the description of scenes with spatio-temporal object annotations, and semantics with actions, events and relations between them.

Here there are a few of those annotations available in the DMD.

Spatial annotations:

  • Body landmarks
  • Face landmarks
  • Eyes landmarks
  • Objects Location
  • Hands Location

Temporal annotations:

  • Driver actions
  • Camera Occlusions
  • Talking
  • Hands using wheel
  • Objects in scene

FEATURES

Channels & Scenarios

We recognize the need to include an infrared channel to the dataset for circumstances in which the lighting is not in favor. Also, our data set also contains depth information of the scene, opening possibilities towards a new analysis approach. We bring to you the data you needed in 3 different forms.

RGB and car    RGB and car

                                                            CAR                                                          SIMULATOR

 

 

 

                                                            CAR                                                          SIMULATOR

 

 

                                                            CAR                                                          SIMULATOR

SETUP

Recording Set-Ups

For the recordings, 3 of Intel® RealSense™ Depth Camera D400-Series were installed in the car and simulator, and placed correctly to capture images of the driver’s face, body and hands as shown in the figure.

1. Microphone

A microphone was included in the recordings, so audio file can be found in some sessions too.

2 Scenarios

The recordings took place in Vicomtech’s car. The activities that are considered as illegal/dangerous to perform while real driving, were made with the car stopped. Also a driving simulation set-up was prepared recreating as much as possible the conditions of the car; this way, the participant could perform these activities having normal reactions and behaviours as he/she was indeed driving.

3 Cameras

For the face and the hands, 2x D415 cameras were used respectively, and 1x D435 for the body since it has a wider field of view. All cameras captured the 3 channels in high resolution and with an established fps without sacrificing the exposure level.

APPLICATIONS

Potential Uses of DMD

The DMD Annotations allow using the material for different research purposes and fields.

TOOLS.

TaTo and DEx

Our team had developed a couple of open-source tools dedicated to DMD annotation (TaTo) and exploration (DEx). With TaTo, you can edit, adapt o create new temporal annotations of the DMD. And, if you want to access the annotations and prepare material for training ML algorithms, DEx will do it for you.

«We believe this collaboration will play a crucial role in making our roads safer for everyone around the world.»

Annotations are being made in collaboration with Deepen AI, a data labeling company specialized in computer vision solutions.

REFERENCES

Cite us!

We have presented DMD on this paper. It would be awesome if you reference this dataset on your work.

The DMD Team thank you.

APA:

Ortega, J. D., Kose, N., Cañas, P., Chao, M.-A., Unnervik, A., Nieto, M., Otaegui, O., Salgado, L. (2020). DMD: A Large-Scale Multi-modal Driver Monitoring Dataset for Attention and Alertness Analysis. In: A. Bartoli & A. Fusiello (eds), Computer Vision — ECCV 2020 Workshops (pg. 387–405). Springer International Publishing.

Other publications regarding the DMD:

  • Cañas P.N., García M., Aranjuelo N., Nieto M., Iglesias A. and Rodríguez I. (2023) Dynamic Risk Assessment Methodology with an LDM-Based System for Parking Scenarios. IEEE 26thInternational Conference on Intelligent Transportation Systems (ITSC), Bilbao, Spain, 2023,pp. 5034-5039, doi: 10.1109/ITSC57777.2023.10422385
  • Urselmann, T., Cañas, P.N., Ortega, J.D., & Nieto, M. (2022). Semi-automatic Pipeline for Large-Scale Dataset Annotation Task: A DMD Application. ECCV Workshops.
  • Ortega J., Cañas P., Nieto M. and Otaegui O, and Salgado L. (2022). Challenges of Large-Scale Multi-Camera Datasets for Driver Monitoring Systems. Sensors 22, no. 7: 2554. https://doi.org/10.3390/s22072554
  • Cañas P., Ortega J., Nieto M. and Otaegui O. (2021). Detection of Distraction-related Actions on DMD: An Image and a Video-based Approach Comparison.In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications – Volume 4: VISAPP, ISBN 978-989-758-488-6, pages 458-465. DOI: 10.5220/0010244504580465

DOWNLOAD

I Need the DMD!

Download a Mini Sample

To help researchers use our dataset faster, we now offer a mini sample. This sample provides a convenient way to explore the data format, content, annotations, and available tools before committing to a full download.

We strongly recommend starting with this sample to familiarize yourself with the dataset structure and ensure it meets your research needs. Once you’re confident about what you require, you can proceed to download the specific data segments you need (distraction, drowsiness, gaze).

Because each part of the dataset is very large, multiple or unnecessary download requests can slow down access for everyone. By exploring the sample first, you help us maintain a smoother and more efficient experience for all DMD users. Thank you for your cooperation!

Download Sample

Download Complete Dataset

You must be 18 or older to download. This dataset can only be used for academic purposes. Any other use beyond that is prohibited. To ensure the continuous development of this project, we demand a responsible use of the data you are about to acquire.

This dataset is published under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

If you have any questions contact us at info-dmd@vicomtech.org.

For easier access, the data has been divided and organized by its primary applications, offered through three different downloading services:

Gaze & Hands
Drowsiness   
Distraction (RGB-IR)
Distraction (Depth)

TEAM

Nice to meet you!

We are the creators of DMD!

Paola Natalia Cañas

Marcos Nieto

Oihana Otaegui

Juan Diego Ortega

Legal

All texts, pictures and other works published on the website are – unless stated otherwise – subject to the copyright of Vicometch. Any duplication, distribution, storage, transmission, transmission and reproduction or transfer of the contents is expressly prohibited without the written permission of Vicomtech.

Vicomtech assumes no responsibility for the correctness and completeness of the information on these pages and dataset.

Licence

All datasets on this page are copyright by Vicomtech and published under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. This means that you must attribute the work in the manner specified by the authors, you may not use this work for commercial purposes and if you remix, transform, or build upon the material, you may not distribute the modified material.

Privacy

The protection of your personal data is very important to us. For this reason, we deal responsibly with your data processing and take into account the statutory requirements of data protection, in particular the provisions of the EU General Data Protection Regulation.

Found yourself in the dataset material? If you want more information on how your data is being stored and/or processed, or you want it to be removed from the dataset, please write to the contact email. (Article 15 & 17 GDPR)


The VI-DAS Project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 690772.


The DMD has been improving during the Aware2All Project This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement number 101076868.


The DMD was created within the framework of the VI-DAS Project led by Vicomtech. Intel® was part of the consortium and is also a contributor to the DMD.

Legal notice

 

Vicomtech

San Sebastian, E20009, Spain
E-mail: info-dmd@vicomtech.org