DMD - Driving Monitoring Dataset

A multi-modal dataset for different

driver monitoring scenarios.

Find out more

Today's driving accidents are mainly due to human error. To make roads safer, we present: DMD, a multi-modal dataset that can contribute to the development of  Driver Monitoring Systems; this way, we can rely on technology and put a brake on these undesirable statistics.
Is the increase in accident risk for Distracted Driving.

Driver Monitoring Dataset

The DMD was devised to address different scenarios in which driver monitoring must be essential in the context of automated vehicles SAE L2-3

A lot of data

The DMD dataset was designed specifically to cover the lack of data for a fully operational DMS.
Hours of video

Lights, Camera, Action!

Variation in the material was one of our priorities:
  • 27% were women and 73% were men.
  • The average age of the participants is 30 years (all over 18 years old).
  • 10 of the participants wore glasses, some were recorded with and without wearing them.
  • The recordings present variation in lightning angle conditions according to the time the session was recorded (morning or afternoon)
  • Some activities were recorded with the car moving, the car stopped and in a second simulation environment.

Material X-Rays

These are a few facts about our dataset:
  • There is about 25TB of raw data!
  • All recordings were grouped by sessions, each one determined by a protocol of activities, a participant, an environment and a lighting condition.
  • In each session there were 3 cameras recording simultaneously: Body, Face and Hands camera.
  • Each camera captures 3 channels of information: RBG, Infrared and Depth.
  • All material comes with VCD annotations of different modalities.

Activities Included

The dataset consists of videos of drivers performing actions related to different driving scenarios in which it is intended to add monitoring systems, so driver state can be identified and later be able to estimate its risk on the road
  • Talking on the phone (L/R)
  • Texting (L/R)
  • Drinking
  • Talking to passenger
  • Reaching behind/side
  • Operating the radio
  • Hair and MakeUp
Stare at a defined region inside the car
  • Front & left front
  • Left/right window
  • Rear-view mirrow
  • Left/rigth side-view mirror
  • Radio zone
  • Lap zone
  • Hands on the wheel (quiet/moving)
  • Hands off the wheel
  • Give control to the vehicle
  • Yawning with/out hand
  • Microsleep
  • Sleepy driving


DMD Annotations are in VCD (Video Content Description).
It supports the description of scenes with spatio-temporal object annotations, and semantics with actions, events and relations between them.
Temporal annotations were made with TaTo.

We are looking for annotators for the DMD!!
Get to know VCD!
Spatial annotations:
  • Body landmarks
  • Face landmarks
  • Eyes landmarks
  • Objects Location
  • Hands Location
Temporal annotations:
  • Driver Actions
  • Camera Occlusions
  • Talking
  • Hands using wheel
  • Objects in scene

Channels & Scenarios

We recognize the need to include an infrared channel to the dataset for circumstances in which the lighting is not in favor. Also, our data set also contains depth information of the scene, opening possibilities towards a new analysis approach.We bring to you the data you needed in 3 different forms!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Recordings Set-Ups

For the recordings, 3 of Intel® RealSense™ Depth Camera D400-Series were installed in the car and simulator, and placed correctly to capture images of the driver’s face, body and hands as shown in the figure.

1 Microphone

A microphone was included in the recordings, so audio file can be found in some sessions too.

3 Cameras

For the face and the hands, 2x D415 cameras were used respectively, and 1x D435 for the body since it has a wider field of view. All cameras captured the 3 channels in high resolution and with an established fps without sacrificing the exposure level.

2 Scenarios

The recordings took place in Vicomtech's car. The activities that are considered as illegal/dangerous to perform while real driving, were made with the car stopped. Also a driving simulation set-up was prepared recreating as much as possible the conditions of the car; this way, the participant could perform these activities having normal reactions and behaviours as he/she was indeed driving.

Potential Uses of DMD

The DMD Annotations allow using the material for different research purposes and fields.

Body Pose Estimation

This dataset can be used by algorithms for body pose estimation. This is possible with the annotations of body landmarks available. Also, depth information analysis could bring interesting results on this matter.

Action Recognition

With all images/videos, you can easily train an algorithm that classifies the action the driver is performing. Timestamps of time intervals of each action are included.

Hands-on-wheel Detection

You can create your own hands-on-wheel detector with our dataset. Body camera and hands camera material can be useful for this task. Using the timestamps of time intervals of the different hands position classification you can train an algorithm, also you could use de body landmarks to have a different approach.

Gaze Direction Estimation

With the timestamps of time intervarls in which the driver is looking at a defined region, you could be able to develop a gaze direction estimator. Also from the face camera, the face landmarks might be an important input to your algorithm.

TaTo and DEx

Our team had developed a couple of open-source tools dedicated to DMD annotation (TaTo) and exploration (DEx).
With TaTo, you can edit, adapt o create new temporal annotations of the DMD. And, if you want to access the annotations and prepare material for training ML algorithms, DEx will do it for you.
Find out more!

Cite us!

We have presented DMD on this paper.
It would be awesome if you reference this dataset on your work.
The DMD Team thank you.

Ortega, J., Kose, N., Cañas, P., Chao, M.a., Unnervik, A., Nieto, M., Otaegui, O., & Salgado, L. (2020). DMD: A Large-Scale Multi-Modal Driver Monitoring Dataset for Attention and Alertness Analysis. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops (Accepted).


author = {Ortega, Juan Diego and Kose, Neslihan and Cañas, Paola and Chao, Min-an and Unnervik, Alexander and Nieto, Marcos and Otaegui, Oihana and Salgado, Luis},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV) Workshops (Accepted)},
title = {{DMD : A Large-Scale Multi-Modal Driver Monitoring Dataset for Attention and Alertness Analysis}},
year = {2020}

Other publications regarding the DMD:

Cañas P., Ortega J., Nieto M. and Otaegui O. (2021). Detection of Distraction-related Actions on DMD: An Image and a Video-based Approach Comparison.In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, ISBN 978-989-758-488-6, pages 458-465. DOI: 10.5220/0010244504580465


I Need the DMD!

If your research needs this juicy dataset, feel free to get in touch with us to download it or to get more info.
To download our dataset you must have a Box Account. Create one before filling out this form.
Thank you for Downloading!

You should get a Box notification in your email, in which you are invited to collaborate with a folder that contains the dataset.
Please, make sure you already have a Box Account with the email you've introduced in this form and you are logged in before accepting the invitation.
Please, check spam in case you can't see our emails in your inbox.

If you have any problem, please contact
Oops! Something went wrong while submitting the form. Try again, please!
Only a lite version of the dataset is available for you to download! More data is coming soon...

Nice to meet you!

We are the creators of the DMD :)
Juan Diego Ortega
Oihana Otaegui
Marcos Nieto
Paola Natalia Cañas