DMD - Driving Monitoring Dataset

A multi-modal dataset for different

driver monitoring scenarios.

Find out more

Today's driving accidents are mainly due to human error. To make roads safer, we present: DMD, a multi-modal dataset that can contribute to the development of  Driver Monitoring Systems; this way, we can rely on technology and put a brake on these undesirable statistics.
23
Times
Is the increase in accident risk for Distracted Driving.
-WHO-
About.

Driver Monitoring Dataset

The DMD was devised to address different scenarios in which driver monitoring must be essential in the context of automated vehicles SAE L2-3

A lot of data

The DMD dataset was designed specifically to cover the lack of data for a fully operational DMS.
37
Participants
20+
Labels

Lights, Camera, Action!

Variation in the material was one of our priorities:
  • 27% were women and 73% were men.
  • The average age of the participants is 30 years (all over 18 years old).
  • 10 of the participants wore glasses, some were recorded with and without wearing them.
  • The recordings present variation in lightning angle conditions according to the time the session was recorded (morning or afternoon)
  • Some activities were recorded with the car moving, the car stopped and in a second simulation environment.

Material X-Rays

These are a few facts about our dataset:
  • There is about 25TB of raw data!
  • All recordings were grouped by sessions, each one determined by a protocol of activities, a participant, an environment and a lighting condition.
  • In each session there were 3 cameras recording simultaneously: Body, Face and Hands camera.
  • Each camera captures 3 channels of information: RBG, Infrared and Depth.
  • All material comes with VCD annotations of different modalities.
Features.

Activities Included

The dataset consists of videos of drivers performing actions related to different driving scenarios in which it is intended to add monitoring systems, so driver state can be identified and later be able to estimate its risk on the road
Phone ringing
Distraction-Related:
  • Talking on the phone (L/R)
  • Texting (L/R)
  • Drinking
  • Talking to passenger
  • Reaching behind/side
  • Operating the radio
  • Hair and MakeUp
Eye estimation
Gaze-Related:
Stare at a defined region inside the car
  • Front & left front
  • Left/right window
  • Rear-view mirrow
  • Left/rigth side-view mirror
  • Radio zone
  • Lap zone
Wheel
Wheel-Related:
  • Hands on the wheel (quiet/moving)
  • Hands off the wheel
  • Give control to the vehicle
Sleep
Fatigue-Related:
  • Yawning with/out hand
  • Microsleep
  • Sleepy driving
Features.

Annotations

DMD Annotations are in VCD (Video Content Description). It supports the description of scenes with spatio-temporal object annotations, and semantics with actions, events and relations between them.
Temporal annotations were made with TaTo.
VCD Logo
Get to know VCD!
Spatial annotations:
  • Body landmarks
  • Face landmarks
  • Eyes landmarks
  • Objects Location
  • Hands Location
Temporal annotations:
  • Driver Actions
  • Camera Occlusions
  • Talking
  • Hands using wheel
  • Objects in scene
Features.

Channels & Scenarios

We recognize the need to include an infrared channel to the dataset for circumstances in which the lighting is not in favor. Also, our data set also contains depth information of the scene, opening possibilities towards a new analysis approach.We bring to you the data you needed in 3 different forms!
RGB and car
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Setup.

Recordings Set-Ups

For the recordings, 3 of Intel® RealSense™ Depth Camera D400-Series were installed in the car and simulator, and placed correctly to capture images of the driver’s face, body and hands as shown in the figure.

1 Microphone

A microphone was included in the recordings, so audio file can be found in some sessions too.
Camera position diagram
Real car

3 Cameras

For the face and the hands, 2x D415 cameras were used respectively, and 1x D435 for the body since it has a wider field of view. All cameras captured the 3 channels in high resolution and with an established fps without sacrificing the exposure level.

2 Scenarios

The recordings took place in Vicomtech's car. The activities that are considered as illegal/dangerous to perform while real driving, were made with the car stopped. Also a driving simulation set-up was prepared recreating as much as possible the conditions of the car; this way, the participant could perform these activities having normal reactions and behaviours as he/she was indeed driving.
Driving simulator
Applications.

Potential Uses of DMD

The DMD Annotations allow using the material for different research purposes and fields.
Body pose estimation

Body Pose Estimation

This dataset can be used by algorithms for body pose estimation. This is possible with the annotations of body landmarks available. Also, depth information analysis could bring interesting results on this matter.
Action recognition

Action Recognition

With all images/videos, you can easily train an algorithm that classifies the action the driver is performing. Timestamps of time intervals of each action are included.
Hands detection

Hands-on-wheel Detection

You can create your own hands-on-wheel detector with our dataset. Body camera and hands camera material can be useful for this task. Using the timestamps of time intervals of the different hands position classification you can train an algorithm, also you could use de body landmarks to have a different approach.
Gaze estimation

Gaze Direction Estimation

With the timestamps of time intervarls in which the driver is looking at a defined region, you could be able to develop a gaze direction estimator. Also from the face camera, the face landmarks might be an important input to your algorithm.

DMD Use Cases

dBehaviourMD by Intel®
The dBehaviourMD is a contribution of annotations from Intel®. This sub-dataset was built to perform driver behaviour recognition tasks. It contains temporal annotations of activities related to distraction. You can find more details on dBehaviourMD in our ECCV Workshop 2020 paper.
Tools.

TaTo and DEx

Our team had developed a couple of open-source tools dedicated to DMD annotation (TaTo) and exploration (DEx).
With TaTo, you can edit, adapt o create new temporal annotations of the DMD. And, if you want to access the annotations and prepare material for training ML algorithms, DEx will do it for you.
Github logo
Find out more!
Deepen logo

"We believe this collaboration will play a crucial role in making our roads safer for everyone around the world."

Annotations are being made in collaboration with Deepen AI, a data labeling company specialized in computer vision solutions.

Reference.

Cite us!

We have presented DMD on this paper.
It would be awesome if you reference this dataset on your work.
The DMD Team thanks you.
Citation:


Ortega, J. D., Kose, N., Cañas, P., Chao, M.-A., Unnervik, A., Nieto, M., Otaegui, O., Salgado, L. (2020). DMD: A Large-Scale Multi-modal Driver Monitoring Dataset for Attention and Alertness Analysis. In: A. Bartoli & A. Fusiello (eds), Computer Vision -- ECCV 2020 Workshops (pg. 387–405). Springer International Publishing.

Other publications regarding the DMD:

Urselmann, T., Cañas, P.N., Ortega, J.D., & Nieto, M. (2022). Semi-automatic Pipeline for Large-Scale Dataset Annotation Task: A DMD Application. ECCV Workshops.

Ortega J., Cañas P., Nieto M. and Otaegui O, and Salgado L. (2022). Challenges of Large-Scale Multi-Camera Datasets for Driver Monitoring Systems. Sensors 22, no. 7: 2554. https://doi.org/10.3390/s22072554

Cañas P., Ortega J., Nieto M. and Otaegui O. (2021). Detection of Distraction-related Actions on DMD: An Image and a Video-based Approach Comparison.In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, ISBN 978-989-758-488-6, pages 458-465. DOI: 10.5220/0010244504580465

Download.

I Need the DMD!

If your research needs this juicy dataset, feel free to get in touch with us to download it or to get more info.
Box logo
To download our dataset you must have a Box Account. Create one before filling out this form.
Thank you for Downloading!

You should get a Box notification in your email, in which you are invited to collaborate with a folder that contains the dataset.
Please, make sure you already have a Box Account with the email you've introduced in this form and you are logged in before accepting the invitation.
Please, check spam in case you can't see our emails in your inbox.

If you have any problem, please contact [email protected]
Oops! Something went wrong while submitting the form. Try again, please!
Only a lite version of the dataset is available for you to download! More data is coming soon...
Team.

Nice to meet you!

We are the creators of the DMD :)
Juan Diego Ortega
Linked in logoResearch Gate logoORCID logo
Oihana Otaegui
Linked in logoResearch Gate logoORCID logo
Marcos Nieto
Linked in logoResearch Gate logoORCID logo
Paola Natalia Cañas
Linked in logoResearch Gate logoORCID logo