HumanE-AI research needs data to advance. Often, researcher struggle to progress for the lack of data. At the same time, collecting a rich and accurate dataset is no easy task. Therefore, we propose to share through the AI4EU platform the datasets already collected so far by different research groups. The datasets will be curated to be ready-to-use for researchers.
Possible extension and variation of such datasets will also be generated using artificial techniques and published on the platform.
A performance baseline will be provided for each dataset, in form of publication reference, developed model or written documentation.
The relevant legal framework will be investigated with specific attention to privacy and data protection in relation to the use and extension of existing datasets as well as future data collection on the subject of multimodal data collection for perception modelling. The microproject will serve as a case study to highlight challenges and opportunities in the development of legal protection by design in data curation for machine learning.


Publication of OPPORTUNITY dataset (and other datasets if time available) on the AI4EU platform. [lead: UoS, contributor: DFKI]

Publication of baseline performance pipeline for OPPORTUNITY dataset (and other datasets if time available) on AI4EU platform. [lead: UoS, contributor: DFKI]

Investigation of data loader and pipeline integration on AI4EU experiment to load HAR dataset and pre-existent pipelines, with a focus on the opportunity dataset (and other datasets if time available) [lead: UoS, contributor: DFKI]

Generation of variation [lead: DFKI]

Survey publications describing datasets and performance baseline [lead: DFKI, contributor: UoS]


Project Partners:

  • University of Sussex (UOS), Mathias Ciliberto
  • German Research Centre for Artificial Intelligence (DFKI), Vitor Fortes Rey
  • Vrije Universiteit Brussel (VUB), Arno de Bois

Primary Contact: Mathias Ciliberto, University of Sussex

Main results of micro project:

Collection, curation and publication of 4 datasets for Multi Modal Perception and Modeling (WP2):
– activity of daily living
– sensor rich
– New additional anonymised, annotated video with OpenPose tracks
– Capacitive Gym:
– 7 popular gym workouts
– 11 subjects, each with separate 5 days
– Capacitive sensors in 3 position
– New dataset
– HCI FreeHand dataset:
– Freehand synthetic gestures
– Multiple 3D accelerometers
– SkodaMini dataset:
– Car manufacturing gestures
– Multiple 3D accelerometer and gyroscope
– Beach volleyball (

Contribution to the objectives of HumaneAI-net WPs

Multi-modal perception and modeling needs data to progress, but recording a new rich and accurate dataset allowing for comparative evaluations by the scientific community is no easy task. Therefore, we gathered rich datasets for multimodal perception and modelling of human activities and gestures. We curated the dataset in order to make them easy to use for research thanks to clear documentation and file formats.
The highlight of this microproject is the OPPORTUNITY++ dataset of activities of daily living, a multi-modal extension of the well-established OPPORTUNITY dataset. We enhanced this dataset which contains wearable sensor data, with previously unreleased data, including video and motion tracking data, which make OPPORTUNITY++ a truly multi-modal dataset with wider appeal, such as to the computer vision community.
In addition, we released other well established activity datasets (HCI FreeHand and SkodaMini dataset) as well as datasets involving novel sensor modalities (CapacitiveGym) and skill-assessment dataset (Wearlab BeachVolleyball)

Tangible outputs