We will develop a conceptual model of key components relating to supporting healthy behavior change. The model will provide a top-level representation of the clinical (from the psychological perspective) enablers and barriers that can be exploited for developing fine-grained models supporting the realization of behavior change paths within and across specific domains.

The resulting ontology will form the basis for generating user models (Theory of Mind), developing reasoning and decision-making strategies for managing conflicting values and motives, which can be used in collaborative and persuasive dialogues with the user. Such knowledge is also fundamental for embedding empathic behavior as well as non-verbal behaviors which can be embodied by a virtual character in the role of a coach. Learning methods can be applied to explore trajectories of behavior change. The produced ontology will represent a valuable resource for the healthcare domain thanks to the knowledge included into the provided resource.

Output

1 conference paper containing the description of the ontology and guidelines for its usage

1 ontology artifact

Presentations

Project Partners:

  • Fondazione Bruno Kessler (FBK), Mauro Dragoni
  • Centre national de la recherche scientifique (CNRS), Jean-Claude Martin
  • Umeå University (UMU), Helena Lindgren

 

Primary Contact: Mauro Dragoni, FBK

Attachments

3.11 The Knowledgeable Coach FBK CNRS UMU pitch-video_Berlin.mkv

This MP studies the problem of how to alert a human user to a potentially dangerous situation, for example for handovers in automated vehicles. The goal is to develop a trustworthy alerting technique that has high accuracy and minimum false alerts. The challenge is to decide when to interrupt, because false positives and false negatives will lower trust. However, knowing when to interrupt is hard, because you must take into account both the driving situation and the driver's ability to react given the alert, moreover this inference must be done based on impoverished sensor data. The key idea of this MP is to model this as a partially observable stochastic game (POSG), which allows approximate solutions to a problem where we have two adaptive agents (human and AI). The main outcome will be an open library called COOPIHC for Python, which allows modeling different variants of this problem.

Output

COOPIHC library (Python)

Paper (e.g,. IUI’23 or CHI’23)

Project Partners:

  • Aalto University, Antti Oulasvirta
  • Centre national de la recherche scientifique (CNRS), Julien Gori

Primary Contact: Antti Oulasvirta, Aalto University

When there is no (sufficient large) training corpus it is usually hard to apply existing text analytics methods. This is even more difficult, when dealing with narratives which are spread across different media and developing over time. Another difficulty is to deal with information on the topic available in different languages and political contexts. In this project, which we will do together with a refugee Ukranian academic, we suggest using to combine methods for semantic text similarity with expert human knowledge in a participatory way. Our training corpus includes news articles containing information on extremism and terrorism.

Output

Conference paper

annotated corpus of news articles

project proposal for the SSF call for Ukraine academic refugees in Sweden

Project Partners:

  • Umeå University (UMU), Frank Dignum

 

Primary Contact: Virginia Dignum, Umeå University

The broad availability of 3D-printing enables end-users to rapidly fabricate personalized objects. While the actual manufacturing process is largely automated, users still need knowledge of complex design applications to not only produce ready-designed objects, but also adapt them to their needs or even design new objects from scratch.

In this project, we explore an AI-powered system that assists users in creating 3D objects for digital fabrication. For this, we propose to use natural language processing (NLP) to enable users to describe objects using their natural language (e.g., “A green rectangular box.”). In this micro project, we conduct a Wizard-of-Oz study to elicit the requirements for such a system. The task of the participants is to recreate a given object using a spoken description with iterative refinements. We expect that this work will support the goal to make personal digital fabrication accessible for everyone.

Output

Requirements for voice-based 3D design

dataset

Design specification for a NLP model to support voice-based 3D design

Project Partners:

  • Ludwig-Maximilians-Universität München (LMU), Florian Müller/Albrecht Schmidt

 

Primary Contact: Florian Müller, LMU Munich

The communication between patients and healthcare institutions is increasingly moving to digital applications. Whereas information about the patient’s wellbeing is typically collected by means of a questionnaire, this is a tedious task for many patients, especially when it has to be done periodically, and may result in incomplete or imprecise input. Much can be gained by making the process of filling in such questionnaires more interactive, by deploying a conversational agent that can not only ask the questions, but also ask follow-up questions and respond to clarification questions by the user. We propose to deploy and test such a system.

Our proposed research aligns well with the WP3 focus on human-AI communication, and will lead to re-usable conversation patterns for conducting questionnaires in healthcare. The work benefit from existing experience with patient-provider communication within Philips and build on the SUPPLE framework for dialog management and sequence expansion.

Output

A dataset on conversation(s) between a patient and a conversational AI

A dialog model derived from the dataset

Scientific publication

Project Partners:

  • Philips Electronics Nederland B.V., Aart van Halteren
  • Stichting VU, Koen Hindriks

Primary Contact: Aart van Halteren, Philips Research

When we go for a walk with friends, we can observe that our movements unconsciously synchronize. This is a crucial aspect of human relations that is known to build trust, liking, and the feeling of connectedness and rapport. In this project, we explore if and how this effect can enhance the relationship between humans and AI systems by increasing the sense of connectedness in the formation of techno-social teams working together on a task.

To evaluate the feasibility of this approach, we plan to build a physical object representing an AI system that can bend in two dimensions to synchronize with the movements of humans. Then, we plan to conduct an initial evaluation in which we will track the upper body motion of the participants and use this data to compute the prototype movement using different transfer functions (e.g., varying the delay and amplitude of the movement).

Output

Physical prototype that employs bending for synchronization

Study results on the feasibility of establishing trust with embodied AI systems through motion synchronization.

Publication of the results

Primary Contact: Florian Müller, LMU Munich

We study proactive communicative behavior, where robots provide information to humans which may help them to achieve desired outcomes, or to prevent possible undesired ones. Proactive behavior in an under-addressed area in AI and robotics, and proactive human-robot communication is even more so. We will combine the past expertise of Sorbonne Univ. (intention recognition) and Orebro Univ. (proactive behavior) to define proactive behavior based on the understanding of user’s intentions, and then extend it to consider communicative actions based on second-order perspective awareness.

We propose an architecture able to (1) estimate the human’s intention of goal, (2) infer robot’s and human’s knowledge about foreseen possible upcoming outcomes of intended goal, (3) detect opportunities for desirability of intended goal to robot be proactive, (4) select action from the listed opportunities. The theoretical underpinning of this work will contribute to the study of theory of mind in HRI.

Output

Jupyter Notebook / Google Colab that presents the code of proposed architecture and is able to provide plug and play interaction.

a manuscript describing the proposed architecture and initial findings of the experiment

Presentations

Project Partners:

  • Sorbonne Université, Mohamed CHETOUANI
  • Örebro University (ORU), Alessandro Saffioti and Jasmin Grosinger

 

Primary Contact: Mohamed CHETOUANI, Sorbonne University

Main results of micro project:

The goal of this micro-project is to develop a cognitive architecture able to generate proactive communicative behaviors during human-robot interactions. The general idea is to provide information to humans which may help them to achieve desired outcomes, or to prevent possible undesired ones. Our work proposes a framework that generates and selects among opportunities for acting based on recognizing human intention, predicting environment changes, and reasoning about what is desirable in general. Our framework has two main modules to initiate proactive behavior; intention recognition and equilibrium maintenance.
The main achievements are:
– Integration of two systems: user intention recognition and equilibrium maintenance in a generic architecture
– Showing stability of the architecture to many users
– Reasoning mechanism and 2nd order perspective awareness
The next steps will aim to show knowledge repair, prevent outcomes of lack of knowledge and improve trustability, transparency and legibility (user study)

Contribution to the objectives of HumaneAI-net WPs

– Playground system that HumaneAI-net partners could define their interactive scenario to play with the robot’s proactivity.

-T3.3 -> Study about how to model human rationality to detect and use computationally defined human belief, goal and intention. Then, use that model to make robots proactive. Human in the loop system to support cooperative behavior of robots while sharing the environment by generating proactive communication.

-T3.1 -> Study relates robots that generate proactive communication, possible effects on human cognition and interaction strategies.

Tangible outputs

Many industrial NLP applications emphasise the processing and detection of nouns, especially proper nouns (Named Entity Recognition, NER). However, processing of verbs has been neglected in recent years, even though it is crucial for the development of full NLU systems, e.g., for the detection of intents in spoken language utterances or events in written language news articles. The META-O-NLU microproject focuses on proving the feasibility of a multilingual event-type ontology based on classes of synonymous verb senses, complemented with semantic roles and links to existing semantic lexicons. Such an ontology shall be usable for content- and knowledge-based annotation, which in turn shall allow for developing NLU parsers/analyzers. The concrete goal is to extend the existing Czech-English SynSemClass lexicon (which displays all the necessary features, but only for two languages) by German and Polish, as a first step to show it can be extended to other languages as well.

Output

Common paper co-authored by the proposers (possibly with et. partners)

Extended version of SynSemClass (entried in additional languages)

Presentations

Project Partners:

  • Charles University Prague, Jan Hajič
  • German Research Centre for Artificial Intelligence (DFKI), Georg Rehm

 

Primary Contact: Jan Hajič, Univerzita Karlova (Charles University, CUNI)

Main results of micro project:

The main results of the META-O-NLU microproject is the extension of the original SynSemClass dataset by German classes, or more precisely, the inclusion of German verbs and event descriptors to the existing classes in SynSemClass. Together with the individual verbs, existing German lexical resources have been linked to (GermaNet, E-VALBU and GUP). Adding a third language demonstrated that future extension to other languages is feasible, in terms of annotation rules, the dataset itself, and in creating a new web browser that can show all language entries alongside each other with all the external links. The data is freely available in the LINDAT/CLAIRAH-CZ repository (and soon also through the Euroepan Language Grid) and a web browser on the resources is now also available.

Contribution to the objectives of HumaneAI-net WPs

Task 3.6 focuses on both spoken and written language-based interactions (dialogues, chats), in particular questions of multilinguality that are essential to the European vision of human-centric AI. The results of this microproject contribute especially to the multlingual issue, and is directed to full NLU (Natural Language Understanding) by describing event types, for which no general ontology exists yet. The resulting resource will be used for both text and dialog annotation, to allow for evaluation and possibly also for training of NLU systems.

Tangible outputs

The aim of the project is to investigate both the theoretical and empirical roles of agency in successful human-computer partnerships. For human-centred AI research, the understanding of agency is a key factor in achieving effective collaboration. Although recent advances in AI have enabled systems to successfully contribute to human-computer interaction, we are interested in extending this such that the interaction acts more like a ‘partnership’. This requires building systems with collaborative agency that users can manipulate in the process. Research questions include: 1) identifying which parameters are relevant to the description of the system agency, 2) what impact these parameters have on the perceived agency and 3) how to modify them in order to achieve different roles of systems in a process.

Output

Theoretical: Literature review on agency / research paper / define parameters

Empirical: Demo (paper, video, interactive)

Project Partners:

  • Institut national de recherche en sciences et technologies du numérique (INRIA), Janin Koch
  • Ludwig-Maximilians-Universität München (LMU), Albrecht Schmidt
  • Københavns Universitet (UCPH), Kasper Hornbaek
  • Stichting VU, Koen Hindriks
  • Umeå University (UMU), Helena Lindgren

 

Primary Contact: Janin Koch, Inria

Attachments

Agency_MicroProject_Koch_Mackay_March17.mov

Exloring the Impact of Agency INRIA J Koch Agency_MP3_Berlin.mov

We propose to research how autobiographical recall can be detected in virtual reality (VR). In particular, we experimentally investigate what physiological parameters accompany interaction with autobiographical memories in VR. We consider VR as one important representation of Human-AI collaboration.

For this, we plan to (1) record an EEG data set of people’s reaction and responses when recalling an autobiographical memory, (2) label the data set, and (3) do an initial analysis of the dataset to inform the design of autobiographical VR experiences. We would try to automate data collection as much as possible to make it easy to add more data over time.

This will contribute to a longer-term effort in model and theory formation. The main Contribution is to WP3. This is set in Task 3.2: Human-AI Interaction/collaboration paradigms and aims at better understanding user emotion in VR to model self-relevance in AI collaboration Task 3.4.

Output

dataset on autobiographical recall in VR

a manuscript describing the data set and initial insights into autobiographical recall in VR

Presentations

Project Partners:

  • Ludwig-Maximilians-Universität München (LMU), Albrecht Schmidt
  • German Research Centre for Artificial Intelligence (DFKI), Paul Lukowicz and Patrick Gebhard

 

Primary Contact: Albrecht Schmidt, Ludwig-Maximilians-Universität München

Main results of micro project:

We have developed VR experiences for research on autobiographical recall in virtual reality (VR). This allows us to experimentally investigate what physiological parameters accompany self-relevant memories elicited by digital content. We have piloted the experiment and are currently recording more data on the recall of autobiographical memories. After data collection is complete, we will label the data set, and do an initial analysis of the dataset to inform the design of autobiographical VR experiences. We have also co-hosted a Workshop on AI and human memory.

Contribution to the objectives of HumaneAI-net WPs

The main Contribution is to WP3. This is set in Task 3.2: Human-AI Interaction/collaboration paradigms and aims at better understanding user emotion in VR to model self-relevance in AI collaboration Task 3.4. The VR experience is implemented in Unity and we are happy to share this in the context of a joint project.

Tangible outputs

In this micro-project, we propose investigating human recollection of team meetings and how conversational AI could use this information to create better team cohesion in virtual settings.

Specifically, we would like to investigate how a person’s emotion, personality, relationship to fellow teammates, goal and position in the meeting influences how they remember the meeting. We want to use this information to create memory aware conversational AI that could leverage such data to increase team cohesion in future meetings.

To achieve this goal, we plan first to record a multi-modal data-set of team meetings in a virtual-setting. Second, administrate questionnaires to participants in different time intervals succeeding a session. Third, annotate the corpus. Fourth, carry out an initial corpus analysis to inform the design of memory-aware conversational AI.

This micro-project will contribute to a longer-term effort in building a computational memory model for human-agent interaction.

Output

A corpus of repeated virtual team meetings (6 sessions spaced, 1 week each)

manual annotations (people’s recollection of the team meeting etc.)

automatic annotations (e.g. eye-gaze, affect, body posture etc.)

A paper describing the corpus and insights gained on the design of memory-aware agents from initial analysis

Project Partners:

  • TU Delft, Catholijn Jonker
  • Eötvös Loránd University (ELTE), Andras Lorincz

 

Primary Contact: Catharine Oertel, TU Delft

Main results of micro project:

1) A corpus of repeated virtual team meetings (4 sessions spaced, 4 days apart each).
2) Manual annotations (people's recollection of the team meeting etc.)
3) Automatic annotations (e.g. eye-gaze, affect, body posture etc.)
4)A preliminary paper describing the corpus and insights gained on the design of memory-aware agents from initial analysis

Contribution to the objectives of HumaneAI-net WPs

In this micro-project, we propose investigating human recollection of team meetings and how conversational AI could use this information to create better team cohesion in virtual settings.
Specifically, we would like to investigate how a person's emotion, personality, relationship to fellow teammates, goal and position in the meeting influences how they remember the meeting. We want to use this information to create memory aware conversational AI that could leverage such data to increase team cohesion in future meetings.
To achieve this goal, we plan first to record a multi-modal data-set of team meetings in a virtual-setting. Second, administrate questionnaires to participants in different time intervals succeeding a session. Third, annotate the corpus. Fourth, carry out an initial corpus analysis to inform the design of memory-aware conversational AI.
This micro-project will contribute to a longer-term effort in building a computational memory model for human-agent interaction.

Tangible outputs

  • Dataset: MEMO – Catharine Oertel
  • Publication: MEMO dataset paper – Catharine Oertel
  • Program/code: Memo feature extraction code – Andras Lorincx

Social dilemmas are situations in which the interests of the individuals conflict with those of the team, and in which maximum benefit can be achieved if enough individuals adopt prosocial behavior (i.e. focus on the team’s benefit at their own expense). In a human-agent team, the adoption of prosocial behavior is influenced by various features displayed by the artificial agent, such as transparency, or small talk. One feature still unstudied is expository communication, meaning communication performed with the intent of providing factual information without favoring any party.

We will implement a public goods game with information asymmetry (i.e. agents in the game do not have the same information about the environment) and perform a user-study in which we will manipulate the amount of information that the artificial agent provides to the team, and examine how varying levels of information increase or decrease human prosocial behavior.

Output

Submission to one of the following: International Journal of Social Robotics, Behaviour & Information Technology, AAMAS, or CHI. Submission to be sent by the end of August 2021.

Release of the game developed for the study on the AI4EU platform to allow other researchers to use it and extend it

Educational component on the Ethical aspect of AI, giving a concrete example on how AI can “manipulate” a human

Presentations

Project Partners:

  • Örebro University (ORU), Jennifer Renoux
  • Instituto Superior Técnico (IST), Ana Paiva

 

Primary Contact: Jennifer Renoux, Örebro University

Main results of micro project:

This micro-project has led to the design and development of an experimental platform to test how communication from an artificial agent influences a human’s pro-social behavior.
The platform comprises the following components:

– a fully configurable mixed-motive public good game, allowing a single human player to play with artificial agents, and an artificial “coach” giving feedback on the human’s action. Configuration is made through json files (number and types of agents, type of feedback, game configuration…)

– a set of questionnaires designed to evaluate the prosocial behavior of the human player during a game

Contribution to the objectives of HumaneAI-net WPs

This project contributes to WP3 and WP4.
The study carried during the micro-project will give insight on how an artificial agent may influence a human's behavior in a social-dilemna context, thus allowing for informed design and development of such artificial agent.
In addition, the platform developed will be made available publicly, allowing future researchers to experiment on other configurations and other types of feedback. By using a well-development and consistent platform, the results of different studies will be more easily comparable.

Tangible outputs

  • Program/code: The Pest Control Game experimental platform – Jennifer Renoux*, Joana Campos, Filipa Correia, Lucas Morillo, Neziha Akalin, Ana Paiva
    https://github.com/jrenoux/humane-ai-sdia.git
  • Publication: nternational Journal of Social Robotics or Behaviour & Information Technology – Jennifer Renoux*, Joana Campos, Filipa Correia, Lucas Morillo, Neziha Akalin, Ana Paiva
    In preparation