Contact person: Rui Prada (rui.prada@tecnico.ulisboa.pt)

Internal Partners:

1. IST, Rui Prada, rui.prada@tecnico.ulisboa.pt
2. INRIA, Janin Koch, Jennifer.Renoux@oru.se
3. ORU, Jennifer Renoux, janin.koch@inria.fr

What do humans think and expect of themselves, the AI partner, and the team in a human-AI collaboration setting? To address this, we conducted an in-person study using Geometry Friends, a two-player cooperative physics-based game in which subjects play with agents presenting different levels of initiative and adaptability.

The study was conducted with three distinct conditions based on two variables: the agent’s level of initiative (follower or leader) and its willingness to shift initiative. The three conditions are 1) follower without shift – the agent always follows the human player’s plan, never taking the initiative, 2) leader without shift – the agent always acts as a leader, following its own plan, 3)

adaptive behaviour – the agent starts as a leader but if the human does not follow, it gives them initiative after a while. For all conditions, the agent plays as the circle character, while the human plays as the rectangle.

We ran an in-person within-subjects study with 20 participants per condition. The levels for each condition were different but comparable in terms of task complexity, and were balanced across conditions to avoid ordering effects. Game levels were designed with more than one solution so that the leader-follower roles and behaviours can be perceived as clearly distinct by the subjects. Additionally, players were able to communicate with one another through their characters’ actions to guide their partner to a specific position (e.g., jumping up and down for the circle, changing shape for the rectangle).

Prior to the main session, where subjects play the game for each of the three conditions, participants fill a questionnaire with demographic information, including gaming habits and experience with AI. Next, they play a single-player level to practice the game controls. The main session was recorded in video and audio and we collected in-game metrics, such as score, actions, and positions in the level as well. Following each condition, participants complete a post-game questionnaire that includes questions about self and AI partner evaluation (intelligence, likeability, creativity), as well as trust, group satisfaction, and team performance.

We predict that when playing with the agent displaying more initiative, subjects feel less accountable for the team’s performance, we also anticipate that when the agent shows more willingness to shift initiative, subjects perceive it as more trustworthy, and possibly, more likable and intelligent.

Results Summary

Dynamics in Human-AI interaction should lead to more satisfying and engaging collaboration. Key open questions are how to design such interactions and the role personal goals and expectations play. We developed three AI partners of varying initiative (leader, follower, shifting) in a collaborative game called Geometry Friends. We conducted a within-subjects experiment with 60 participants to assess personal AI partner preference and performance satisfaction as well as perceived warmth and competence of AI partners. Our main results are listed below:

• RQ1: AI partner perceptions influence trust levels and are mediated by feelings of control.

We found that the perception of AI partners affects trust, influenced by the level of control the agents provide. If an agent insists on its own plan taking full control, like the leader agent, without establishing trust, it seems unfriendly. However, an agent that follows the human’s plan, like the follower agent, gains more trust as it gives control to the human. Concerning the shifting agent, this agent may shift modes between leader and follower without the human fully understanding why, making participants lose control of the game, which can lead to negative perceptions.

• RQ2: Implicit communication requires time.

The follower agent’s understanding of the human’s presence and plan likely contributed to higher perceived performance, even though its objective performance was lower. Nevertheless, the lack of an explicit human-AI communication mechanism required humans to learn how to communicate with the agent, perceiving it as slower.

• RQ3: Self perceptions related to sense of responsibility in task.

Participants felt like they played better than the leader agent, indicating that they may feel less responsible for task performance when interacting with an agent playing a leadership role, reducing their sense of accountability and trust towards the agent.

• RQ4: Preferences influenced by context and personal characteristics. 

We found that competitive participants who prioritized achievement and fast-paced gaming preferred the leader agent. Meanwhile, those valuing collaboration and fun preferred the follower agent. Possibly, by highlighting collaboration and team decision-making, the leader would be chosen less often. Regarding gender, female preference might lean towards patient, supportive agents, whereas some male participants prefer proactive goal-oriented agents.

Tangible Outcomes

1. Inês Lobo, Janin Koch, Jennifer Renoux, Inês Batina, and Rui Prada. 2024. When Should I Lead or Follow: Understanding Initiative Levels in Human-AI Collaborative Gameplay. In Proceedings of the 2024 ACM Designing Interactive Systems Conference (DIS ’24). Association for Computing Machinery, New York, NY, USA, 2037–2056. https://doi.org/10.1145/3643834.3661583
2. Source code: https://github.com/1000obo/geometry-friends-study 

Contact person: András Lőrincz  (lorincz@inf.elte.hu)

Internal Partners:

1. ELTI, András Lőrincz,
2. DFKI, Daniel Sonntag, daniel.sonntag@dfki.de

Allocentric semantic 3D maps are highly useful for a variety of human-machine interactions since ego-centric instructions can be derived by the machine for the human partner. Class labels, however, may differ or could be missing for the participants due to the different perspectives. In order to overcome this issue, we extend an existing real-time 3D semantic reconstruction pipeline with semantic matching across human and robot viewpoints. We use deep recognition networks, which usually perform well from higher (i.e., human) viewpoints but are inferior from lower viewpoints like that of a small robot. We propose several approaches for acquiring semantic labels for images taken from unusual perspectives. We start with a partial 3D semantic reconstruction from the human perspective that we transfer and adapt to the robot’s perspective using superpixel segmentation and the geometry of the surroundings. The quality of the reconstruction is evaluated in the Habitat simulator and in a real environment using a robot car with an RGBD camera. 

Results Summary

We show that the proposed approach provides high-quality semantic segmentation from the robot’s perspective, with accuracy comparable to the original one. In addition, we exploited the gained information and improved the recognition performance of the deep network for the lower viewpoints and showed that the small robot alone is capable of generating high-quality semantic maps for the human partner. The computations are close to real time, so the approach enables interactive applications.

Tangible Outcomes

1. 3D Semantic Label Transfer and Matching in Human-Robot Collaboration Szilvia Szeier, Benjámin Baffy, Gábor Baranyi, Joul Skaf, László Kopácsi, Daniel Sonntag, Gábor Sörös, and András Lőrincz ECCV 2022 Workshop on Learning to Generate 3D Shapes and Scenes October 23, 2022 https://learn3dg.github.io/  . 
   1. Paper: https://drive.google.com/file/d/1GD065M4qj2BhT6ujZEv_kMTFTmBYWL6C/view  
   2. Poster: https://drive.google.com/file/d/15GsMVDqnVBQnmg-bIifgY8gENf-xtAn0/view  https://learn3dg.github.io/static/img/poster/0003-poster.png

1. Cross-Viewpoint Semantic Mapping: Integrating Human and Robot Perspectives for Improved 3D Semantic Reconstruction László Kopácsi, Benjámin Baffy, Gábor Baranyi, Joul Skaf, Gábor Sörös, Szilvia Szeier, András Lőrincz, Daniel Sonntag Special Issue Deep Learning in Visual and Wearable Sensing for Motion Analysis and Healthcare. Journal: Sensors https://www.mdpi.com/1424-8220/23/11/5126  
2. labelling tool repository: https://github.com/szilviaszeier/semantic_matching  
3. Video: https://youtu.be/iiC8nYqVHHk