[TMP-004] A Human-AI Collaboration Study using the Geometry Friends Game

Contact person: Rui Prada (rui.prada@tecnico.ulisboa.pt)

Internal Partners:

1. IST, Rui Prada, rui.prada@tecnico.ulisboa.pt
2. INRIA, Janin Koch, Jennifer.Renoux@oru.se
3. ORU, Jennifer Renoux, janin.koch@inria.fr

What do humans think and expect of themselves, the AI partner, and the team in a human-AI collaboration setting? To address this, we conducted an in-person study using Geometry Friends, a two-player cooperative physics-based game in which subjects play with agents presenting different levels of initiative and adaptability.

The study was conducted with three distinct conditions based on two variables: the agent’s level of initiative (follower or leader) and its willingness to shift initiative. The three conditions are 1) follower without shift – the agent always follows the human player’s plan, never taking the initiative, 2) leader without shift – the agent always acts as a leader, following its own plan, 3)

adaptive behaviour – the agent starts as a leader but if the human does not follow, it gives them initiative after a while. For all conditions, the agent plays as the circle character, while the human plays as the rectangle.

We ran an in-person within-subjects study with 20 participants per condition. The levels for each condition were different but comparable in terms of task complexity, and were balanced across conditions to avoid ordering effects. Game levels were designed with more than one solution so that the leader-follower roles and behaviours can be perceived as clearly distinct by the subjects. Additionally, players were able to communicate with one another through their characters’ actions to guide their partner to a specific position (e.g., jumping up and down for the circle, changing shape for the rectangle).

Prior to the main session, where subjects play the game for each of the three conditions, participants fill a questionnaire with demographic information, including gaming habits and experience with AI. Next, they play a single-player level to practice the game controls. The main session was recorded in video and audio and we collected in-game metrics, such as score, actions, and positions in the level as well. Following each condition, participants complete a post-game questionnaire that includes questions about self and AI partner evaluation (intelligence, likeability, creativity), as well as trust, group satisfaction, and team performance.

We predict that when playing with the agent displaying more initiative, subjects feel less accountable for the team’s performance, we also anticipate that when the agent shows more willingness to shift initiative, subjects perceive it as more trustworthy, and possibly, more likable and intelligent.

Results Summary

Dynamics in Human-AI interaction should lead to more satisfying and engaging collaboration. Key open questions are how to design such interactions and the role personal goals and expectations play. We developed three AI partners of varying initiative (leader, follower, shifting) in a collaborative game called Geometry Friends. We conducted a within-subjects experiment with 60 participants to assess personal AI partner preference and performance satisfaction as well as perceived warmth and competence of AI partners. Our main results are listed below:

• RQ1: AI partner perceptions influence trust levels and are mediated by feelings of control.

We found that the perception of AI partners affects trust, influenced by the level of control the agents provide. If an agent insists on its own plan taking full control, like the leader agent, without establishing trust, it seems unfriendly. However, an agent that follows the human’s plan, like the follower agent, gains more trust as it gives control to the human. Concerning the shifting agent, this agent may shift modes between leader and follower without the human fully understanding why, making participants lose control of the game, which can lead to negative perceptions.

• RQ2: Implicit communication requires time.

The follower agent’s understanding of the human’s presence and plan likely contributed to higher perceived performance, even though its objective performance was lower. Nevertheless, the lack of an explicit human-AI communication mechanism required humans to learn how to communicate with the agent, perceiving it as slower.

• RQ3: Self perceptions related to sense of responsibility in task.

Participants felt like they played better than the leader agent, indicating that they may feel less responsible for task performance when interacting with an agent playing a leadership role, reducing their sense of accountability and trust towards the agent.

• RQ4: Preferences influenced by context and personal characteristics. 

We found that competitive participants who prioritized achievement and fast-paced gaming preferred the leader agent. Meanwhile, those valuing collaboration and fun preferred the follower agent. Possibly, by highlighting collaboration and team decision-making, the leader would be chosen less often. Regarding gender, female preference might lean towards patient, supportive agents, whereas some male participants prefer proactive goal-oriented agents.

Tangible Outcomes

1. Inês Lobo, Janin Koch, Jennifer Renoux, Inês Batina, and Rui Prada. 2024. When Should I Lead or Follow: Understanding Initiative Levels in Human-AI Collaborative Gameplay. In Proceedings of the 2024 ACM Designing Interactive Systems Conference (DIS ’24). Association for Computing Machinery, New York, NY, USA, 2037–2056. https://doi.org/10.1145/3643834.3661583
2. Source code: https://github.com/1000obo/geometry-friends-study