Contact person: Samuel Kaski (samuel.kaski@aalto.fi)

Internal Partners:

1. Aalto University
2. Delft University of Technology, Frans Oliehoek

In human-AI collaboration, one of the key difficulties is establishing a common ground for the interaction, especially in terms of goals and beliefs. In practice, the AI might not have access to this necessary information directly and must infer it during the interaction with the human. However, training a model to support this kind of inference would require massive collections of interaction data and is not feasible in most applications.

Modern cognitive models, on the other hand, can equip AI tools with the necessary prior knowledge to readily support inference, and hence, to quickly establish a common ground for collaboration with humans. However, utilizing these models in realistic applications is currently impractical due to their computational complexity and non-differentiable structure.

Contact person: Catholijn Jonker, Maria Tsfasman (c.r.m.m.oertel@tudelft.nl; m.tsfasman@tudelft.nl)

Internal Partners:

1. Technical University Delft, Catharine Oertel, c.r.m.m.oertel@tudelft.nl
2. Eotvos Lorand University, Andras Lorincz, lorincz@inf.elte.hu

In this micro-project, we propose investigating human recollection of team meetings and how conversational AI could use this information to create better team cohesion in virtual settings. Specifically, we would like to investigate how a person’s emotion, personality, relationship to fellow teammates, goal and position in the meeting influences how they remember the meeting. We want to use this information to create memory aware conversational AI that could leverage such data to increase team cohesion in future meetings. To achieve this goal, we first record a multi-modal dataset of team meetings in a virtual setting. Second, administrate questionnaires to participants in different time intervals succeeding a session. Third, annotate the corpus. Fourth, carry out an initial corpus analysis to inform the design of memory-aware conversational AI. This micro-project will contribute to a longer-term effort in building a computational memory model for human-agent interaction.

Results Summary

The MEMO corpus was collected, which contains 45 group discussions around the topic of COVID-19. A total of 15 groups were formed, consisting of 3 to 6 participants who took part in 3 group discussions, with a 3-4 day gap between sessions. A total of 59 individuals with diverse backgrounds took part in the study. Before and after each session participants completed a series of questionnaires to determine which moments they recalled from their conversations, along with their personality traits, values and perceptions.

To capture conversational memory, we collected first-party free-recall reports of the most memorable moments from the discussion immediately after the interaction and again 3-4 days later. For the shorter-term memories, participants also mapped the moments to a particular interval in the video of their discussion, which were used for the ground-truth conversational memory annotations.

For each participant, personality and value profiles were recorded in the pre-screening survey, along with demographic information to identify their social group affected by COVID-19. Pre-session questionnaires also assessed participants’ mood before each session. Post-session questionnaire included questions about mutual understanding, personal attitude and perceived social distance. The perception of the discussion and the group as a whole was also monitored in the post-session questionnaire with variables such as Task and Group Cohesion, Entitativity, Perceived Interdependence, Perceived Situation Characteristics, Syncness, and Rapport.

The following automatic annotations were extracted on the corpus:

* Transcripts – Transcripts were generated with automatic speech recognition methods and were manually reviewed and corrected where needed. Transcript timestamps are available at the utterance level as well as word-level text grid files for each recording. Speaker diarization is also available.

* Eye gaze and head pose – automatically annotated with EyeWare software, the annotation itself will be provided, but the code uses proprietary API. This includes gaze targets collected through screenshots of participants’ screen views.

* Prosody – eGeMAPS feature set was extracted from the default eGeMAPS configuration in OpenSmile

* Body pose – Body pose (upper body only) and hand pose when visible were estimated with the models available in the MediaPipe software

* Facial action units – Facial action units were estimated for participants using the OpenFace Software

A Paper describing the corpus and the annotations in more detail is in preparation. Additionally, the collected annotations are to be packaged in an appropriate manner for ease of use for future researchers.

Tangible Outcomes

1. Tsfasman, M., Fenech, K., Tarvirdians, M., Lorincz, A., Jonker, C., & Oertel, C. (2022). Towards creating a conversational memory for long-term meeting support: predicting memorable moments in multi-party conversations through eye-gaze. In ICMI 2022 – Proceedings of the 2022 International Conference on Multimodal Interaction (pp. 94-104). (ACM International Conference Proceeding Series). Association for Computing Machinery (ACM). https://doi.org/10.1145/3536221.3556613 
2. summary
   

Contact person: Eric Blaudez, (eric.blaudez@thalesgroup.com)

Internal Partners:

1. Thales, Eric Blaudez, eric.blaudez@thalesgroup.com
2. Unibo, Paolo Torrini, p.torroni@unibo.it
3. CNRS

External Partners:

1. LISN, Christophe Servan c.servan@qwant.com

The micro-project provides a demonstration of the hierarchical framework for collaboration described in the Humane-AI Net revised strategic work plan, by constructing a multimodal and multilingual conversational agents focused on search. The framework is based on hierarchical levels of abilities:

• Reactive (sensori-motor) Interaction: Interaction is tightly-coupled perception-action where actions of one agent are immediately sensed and interpreted as actions of the other. Examples include greetings, polite conversation and emotional mirroring
• Situated (Spatio-temporal) Interaction Interactions are mediated by a shared model of objects and relations (states) and shared models for roles and interaction protocols.

In this micro-project, we focused on the 2 first levels (Reactive and Situational) and designed the global framework architecture to show a Proof of Concept (PoC).

Results Summary

We show that the proposed approach provides high-quality semantic segmentation from the robot’s perspective, with accuracy comparable to the original one. In addition, we exploited the gained information and improved the recognition performance of the deep network for the lower viewpoints and showed that the small robot alone is capable of generating high-quality semantic maps for the human partner. The computations are close to real time, so the approach enables interactive applications.

Tangible Outcomes

1. T-KEIR: https://github.com/ThalesGroup/t-keir 
2. erc-unibo-module: https://github.com/helemanc/erc-unibo-module 

Contact person: Jan Hajic, Charles Univ, (jan.hajic@mff.cuni.cz)

Internal Partners:

1. Charles Univ, Jan Hajic
2. DFKI, Thierry deClerck

Many industrial NLP applications emphasize the processing and detection of nouns, especially proper nouns (Named Entity Recognition, NER). However, processing of verbs has been neglected in recent years, even though it is crucial for the development of full NLU systems, e.g., for the detection of intents in spoken language utterances or events in written language news articles. The META-O-NLU microproject focuses on proving the feasibility of a multilingual event-type ontology based on classes of synonymous verb senses, complemented with semantic roles and links to existing semantic lexicons. Such an ontology shall be usable for content- and knowledge-based annotation, which in turn shall allow for developing NLU parsers/analyzers. The concrete goal is to extend the existing Czech- English SynSemClass lexicon (which displays all the necessary features, but only for two languages) by German and Polish, as a first step to show it can be extended to other languages as well.

Results Summary

Extended version of SynSemClass (entried in additional languages)

Tangible Outcomes

1. SynSemClass 3.5 dataset, (dataset),
   URL: http://hdl.handle.net/11234/1-3750

1. SynSemClass 3.5 browser, (other),
   URL: https://lindat.cz/services/SynSemClass35/

Contact person: James Crowley (James@crowley-coutaz.fr)

Internal Partners:

1. Eotvos Lorand University – ELTE, Andras Lorincz
2. Univ Grenoble Alpes, Dominique Vaufreydaz, Fabien Ringeval
3. Uni Paris Saclay, Camille Guinaudeau, Marc Evrard
4. Jozef Stefan Institut-JSI, Marko Grobelnik
5. Charles University, Pavel Pecina

Transformers and self-attention (Vaswani et al., 2017), have become the dominant approach for natural language processing (NLP) with systems such as BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020) rapidly displacing more established RNN and CNN structures with an architecture composed of stacked encoder-decoder modules using self- attention. This micro-project provide tools and data sets for experiments and a first initial demonstration of the potential of transformers for multimodal perception and multimodal interactions. We define research challenges, benchmark data sets and performance metrics for multimodal perception and interaction tasks such as (1) audio-visual narration of scenes, cooking actions and activities, (2) audio-video recordings of lectures and TV programs (3) audio-visual deictic (pointing) gestures, and (4) perception and evocation of engagement, attention, and emotion.

1) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. and Polosukhin, I. (2017). Attention is all you need. arXiv preprint arXiv:1706.03762

2) Devlin, J., Chang, M. W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

3) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., and Amodei, D. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

Results Summary

In this project, we explore the potential of Transformer-based models in two significant domains: unsupervised object discovery and multimodal emotion recognition using physiological signals. First, we demonstrate a novel approach for unsupervised object discovery by leveraging self-supervised learning with self-distillation loss (DINO). Our method utilizes visual tokens as nodes in a weighted graph, where edges reflect connectivity scores based on token similarity. By applying a normalized graph-cut and solving it through spectral clustering with generalized eigen-decomposition, we isolate foreground objects. This approach effectively segments self-similar regions, with the second smallest eigenvector of the decomposition providing the cutting solution that indicates token association with foreground objects. This technique not only simplifies the object discovery process but also achieves substantial performance improvements over current state-of-the-art methods such as LOST, outperforming it by 6.9%, 8.1%, and 8.1% on the VOC07, VOC12, and COCO20K benchmarks, respectively. Furthermore, integrating a second-stage class-agnostic detector (CAD) enhances these results, and our method’s adaptability is demonstrated in its application to unsupervised saliency detection and weakly supervised object detection, achieving notable IoU improvements on the ECSSD, DUTS, and DUT-OMRON datasets.

In parallel, we address the challenge of multimodal emotion recognition from physiological signals using Transformer-based models. Recognizing the advantages of attention mechanisms in Transformers for creating contextualized representations, we propose a model for processing electrocardiogram (ECG) data to predict emotions. This model highlights significant segments of the signal, ensuring that relevant information is given priority. Due to the limited size of datasets with emotional labels, we adopt a self-supervised learning approach. We pre-train our model using unlabelled ECG datasets to build robust representations and then fine-tune it on the AMIGOS dataset for emotion recognition. Our findings confirm that this approach achieves state-of-the-art results in emotion recognition tasks involving ECG signals. Additionally, the success of this strategy underscores the broader potential of Transformers and pre-training techniques for analyzing time-series data in emotion recognition tasks.

Overall, the outcomes of our project demonstrate that Transformer-based models, coupled with self-supervised learning, can significantly enhance the performance of both unsupervised object discovery and emotion recognition from physiological signals. These methods provide robust solutions for complex visual and temporal signal analysis tasks, marking a substantial step forward in computer vision and affective computing.

Tangible Outcomes

1. Y. Wang, X. Shen, S. Hu, Y. Yuan, J. L. Crowley, D. Vaufreydaz, Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut. IEEE International Conference on Computer Vision and Pattern Recognition, CVPR 2022, pp14543-14553, New Orleans, Jun 2022.
   https://arxiv.org/abs/2202.11539 
2. J. Vazquez-Rodriguez, G. Lefebvre, J. Cumin and J. L Crowley, “Emotion Recognition with PreTrained Transformers Using Multimodal Signals”, 10th International Conference on Affective Computing and Intelligent Interaction (ACII), Oct 2022
   https://ieeexplore.ieee.org/document/9953852  .
3. J. Vazquez-Rodriguez, G. Lefebvre, J. Cumin, J. L. Crowley. Transformer-Based SelfSupervised Learning for Emotion Recognition. 26th International Conference on Pattern Recognition (ICPR 2022), Aug 2022, Montreal, Canada.
   https://arxiv.org/abs/2204.05103 
4. A survey of tools and datasets for a multimodal perception with transformers (http://crowley-coutaz.fr/jlc/HumanE-AI-Net/TransfomerMicroProject/TransformerTools.pdf )
5. A tutorial on the use of transformers for multimodal perception. (http://crowley-coutaz.fr/jlc/Courses/ACAI2021/Multimodal-Transformer-Tutorial.html )
6. Report on challenges for the use of transformers for multimodal perception and interaction. (http://crowley-coutaz.fr/jlc/HumanE-AI-Net/TransfomerMicroProject/ReseachChallengesDataSets.pdf )

Contact person: Florian Müller (florian.mueller@um.ifi.lmu.de )

Internal Partners:

1. LMU Munich, Florian Müller, florian.mueller@um.ifi.lmu.de  

External Partners:

1. University of Bari, Giuseppe Desolda, giuseppe.desolda@uniba.it 

Manufacturing tools like 3D printers have become accessible to the wider society, making the promise of digital fabrication for everyone seemingly reachable. While the actual manufacturing process is largely automated today, users still require knowledge of complex design applications to not only produce ready-designed objects, but also adapt them to their needs or design new objects from scratch. To lower the barrier for the design and customization of personalized 3D models, we imagine an AI-powered system that assists users in creating 3D objects for digital fabrication. Reaching this vision requires a common understanding – a common ground – between the users and the AI system. As a first step, in this micro project, we explored novices’ mental models in voice-based 3D design by conducting a high-fidelity Wizard of Oz study with 22 participants without skills in 3D design. We asked the participants to perform 14 tasks revolving around some basic concepts of 3D design for digital modeling, like the creation of objects, the manipulation of objects (e.g., scaling, rotating, and/or moving objects), and the creation of composite objects. We performed a thematic analysis of the collected data assessing how the mental model of novices translates into voice-based 3D design.

Results Summary

We found that future AI assistants to support novice users in voice-based digital modeling must: manage the correction the users do during and after the commands to fix certain errors; deal with vague and incomplete commands by automatically completing the commands with sensible defaults or by asking the users for clarification; consider the prior novices knowledge, for example, about the use of undo and redo functions; provide only a simplified set of operations for creating simple and composite 3D objects; design a workflow similar to what novices would do if they were building real objects, for example, providing wizard procedures that guide novices in designing composite 3D models starting from the bottom; provide different commands to select 3D objects; understand and execute chained commands; understand commands that are relative to the users’ point of view; grant multiple ways to refer to the axes, for example, by using their names, colors and user direction; favor explicit trigger words to avoid unintentional activation of the voice assistant; embrace diversity in naming approaches since novices often use other words to refer to 3D objects.

Contact person: András Lőrincz (lorincz@inf.elte.hu) 

Internal Partners:

1. Eötvös Loránd University (ELTE), András Lőrincz and Daniel Sindley
2. Charles University Prague, Ondřej Dušek and Tomáš Nekvinda  

We propose research on a scalable human-machine collaboration system with the goal of executing high quality actions for rehabilitation exercises. We combine video and speech for video-grounded goal-oriented dialogue. We build on our video and text database. The database has exercises for rehabilitation following knee injuries. We evaluate high performance body pose estimation tools and compare it to a real-time body pose estimation tool to be developed for smartphones via ‘knowledge distillation’ methods. The complementing part of the project deals with the texts that we have collected for these exercises and estimates the amount of texts needed for dialogues that can lead and correct the quality of exercises. Potential topics/intents include pose relative to camera, proper light conditions, audio-visual information about pain, notes about execution errors, errors discovered by the computer evaluations, requests about additional information from the patient, and reactions to other, unrelated queries.

Results Summary

Human-machine collaboration will soon be ubiquitous, as machines can help in everyday life. However, spatial tasks are challenging because of real-time constraints. We want to optimize the interaction offline before it happens in real time to ensure high quality. We present the SPAtial TAsk (SPATA) framework. SPATA is modular, and here we address two connected components; body pose optimization and navigation. Our experiments show that 3D pose estimation using 2D cameras is accurate when the motion is captured from the right direction and distance. This limitation currently restricts us to simple forms of movement, such as those used in physical rehabilitation exercises. Accurate estimation requires (a) estimation of body size, (b) optimization of body and camera position, (c) navigation assistance to a location, and (d) activity capture and error estimation. An avatar model is used to estimate the shape and a skeleton model is used to estimate the body pose for (a). For (b), we use SLAM. For (c), we use a semantic map and optimize a minimal NLP system for human needs that we test. Finally, we estimate the accuracy of the motion and propose a visual comparison between the planned and the implemented motion pattern for (d). Our SPATA framework is useful for various tasks at home, in gyms and other spatial applications. Depending on the task, different components can be integrated. The MP targeted specific topics, including

(i) body motion and pain both

— in terms a language and potential dialogues and

— in more than 400 video samples that included 50 exercises and about 7 errors on the average to be detected alone or in combinations for each motion types

and

(ii) dialogues

— from experts and

— crowdsourcing based dialogue enhancements

Tangible Outcomes

1. DeepRehab: Real Time Pose Estimation on the Edge for Knee Injury Rehabilitation – Bruno Carlos Dos Santos Melício, Gábor Baranyi, Zsófia Gaal, Sohil Zidan,and Andras Lőrincz. https://e-nns.org/icann2021/
2. Video presentation summarizing the project

Contact person: Gilles Bailly (Gilles.Bailly@sorbonne-universite.fr) 

Internal Partners:

1. Sorbonne Université, Gilles Bailly
2. Aalto University, Kashyap Todi and Antti Oulasvirta  

External Partners:

1. University of Luxembourg, Luis Leiva  

Adapting user interfaces (UIs) requires taking into account both positive and negative effects that changes may have on the user. A carelessly picked adaptation may impose high costs — for example, due to surprise or relearning effort. It is essential to consider differences between users as the effect of an adaptation depends on the user’s strategies, e.g. how each user searches for information in a UI. This microproject extends an earlier collaboration between partners on model-based reinforcement learning for adaptive UIs by developing methods to account for individual differences. Here, we first develop computational models to explain and predict users’ visual search and pointing strategies when searching within a UI. We apply this model to infer user strategies based on interaction history, and adapt UIs accordingly.

Results Summary

This micro-project reinforces the collaborations between Sorbonne Université, Aalto University and University of Luxembourg with weekly meetings. It aims at elaborating computational models of visual search in adaptive User Interfaces. We defined different visual search strategies in adaptive menus as well as promising interactive mechanisms to revisit how to to design menus. The Elaboration of the model is in progress. Concretely, we achieved 4 things:

1. Created a model of visual search and pointing in menus. The code is available on GitHub
2. The integration of the model in our platform for adaptive UI. The code is available on GitHub
3. A demo of the system
4. A publication at the conference ACM CHI

Tangible Outcomes

1.  Adapting User Interfaces with Model-based Reinforcement Learning. Kashyap Todi, Gilles Bailly, Luis A. Leiva, Antti Oulasvirta.In CHI ’21: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems https://dl.acm.org/doi/fullHtml/10.1145/3411764.3445497
2. summary of project’s key findings https://www.kashyaptodi.com/adaptive/
3. paper presentation from CHI’21
4. Interaction preview
5. Video summarizing the project

Contact person: Frank Dignum (dignum@cs.umu.se)

Internal Partners:

1. Umeå University (UMU), Frank Dignum
2. Consiglio Nazionale delle Ricerche (CNR), Eugenia Polizzi, Giulia Andrighetto and Mario Paolucci
3. Leiden university, Mark Dechesne  

External Partners:

1. UU and National Police in Netherlands, Mijke van den Hurk 

In this project we investigate whether normative behavior can be detected in facebook groups. In a first step we hypothesize about possible norms that could lead to a group becoming more extreme on social media, or whether groups that become more extreme will develop certain norms that distinguish them from other groups and that could be detected. An example of such a norm could be that a (self-proclaimed) leader of a group is massively supported by retweets, likes or affirmative messages, along with evidence of verbal sanctioning toward counter-normative replies. Simulations and analyses of historical facebook data (using manual detection in specific case studies and more broadly through NLP) will help reveal the existence of normative behavior and its potential change over time.

Results Summary

The project delivered detailed analyses of the tweets around the USA elections and subsequent riots. Where we thought we might discover some patterns in the tweets indicating more extreme behavior, it appears that extremist expressions are quickly banned from Twitter and find a home in more niche social platforms (in this case Parler). Thus the main conclusion of this project is that we need to find the connections between users in different social media platforms in order to track any extreme behavior.

In order to see how individuals might contribute to behavior that is not in the interest of society we cannot analyze one social media platform. Especially more extremist expressions quickly disappear from main stream social media to niche platforms that can quickly change over time. Thus the connection between individual and societal goals is difficult to observe by just analyzing data from a single social media platform. On the other hand it is very difficult to link users between platforms. Our core contribution could be summarized in 2 points:

1. Identification of radical behavior in Parler groups
2. Characterizing the language use of radicalized communities detected on Parler

Tangible Outcomes

1. Video presentation summarizing the project

Contact person: Antti Oulasvirta (antti.oulasvirta@aalto.fi)

Internal Partners:

1. Aalto University, Antti Oulasvirta, antti.oulasvirta@aalto.fi
2. CNRS and UPMC, Julien Gori, gori@isir.upmc.fr

This project studied how AI assistants could better alert humans by estimating their internal states from observations. Contributed to a computational theory called POSG, a multi-agent framework for human-AI interaction developed between CNRS and Aalto University. When is an opportune moment to alert a human partner? This question is hard, because the beliefs and cognitive state of the human should be taken into account when choosing if/when to alert. Every alert is interruptive and bears a cost to the human. However, especially in safety-critical domains, the consequences of not alerting can be infinitely negative. In this work, we formulate the optimal alerting problem based on the theory of partially observable stochastic games.

Results Summary

The problem of the assistant and the problem of the user are formulated and solved as a single problem in POSG. We presented first results using a gridworld environment, comparing different types of alerting agents and a roadmap for future work using realistic driver simulators. These models can inform handover/takeover decisions in semi-automated vehicles. The results were integrated into COOPIHC, a multiagent solver for interactive AI.

Tangible Outcomes

1. Contributed to a computational theory called POSG, a multi-agent framework for human-AI interaction developed between CNRS and Aalto University: https://jgori-ouistiti.github.io/CoopIHC/ 

Contact person: Brian Ravenet (brian.ravenet@limsi.fr) 

Internal Partners:

1. CNRS, Brian Ravenet, brian.ravenet@limsi.fr
2. INESC-ID, Rui Prada, rui.prada@tecnico.ulisboa.pt

This project aims at investigating the construction of humor models to enrich conversational agents through the help of interactive reinforcement learning approaches. Our methodology consists of deploying an online platform where passersby can play a game of matching sentences with humorous comebacks against an agent. The data collected from these interactions helps to gradually build the humor models of the agent following state of the art Interactive Reinforcement Learning techniques. Our work resulted in an implementation of the platform, a first model for humor-enabled conversational agent and a publication of the obtained results and evaluations.

Results Summary

The main result of this project is the creation of an intelligent agent capable of playing a game – Cards Against Humanity- that involves matching sentences with humorous comebacks. In order to achieve this, a dataset of 1712 jokes, rated on a scale of 1 to 9 in terms of joke level, originality, positivity, entertainment, whether it makes sense and whether it is family-friendly, were collected and an online game was developed to serve as the foundation of the reinforcement mechanism.