[TMP-088] Natural Imitation of Dance Moves and Human Gestures with a Humanoid Robot

Contact person: Shivesh Kumar, (shivesh.kumar@dfki.de )

Internal Partners:

1. DFKI Bremen ,Melya Boukheddimi, Shivesh Kumar, shivesh.kumar@dfki.de
2. INRIA Paris, Justin Carpentier, justin.carpentier@inria.fr 

The objectives of this Micro-Project were to create a software toolkit that makes it possible to achieve realistic human-like motions that can lead to a feeling of trust and comfort towards a robot. The project was based on a generic formalization of robot dancing which makes it possible to use musical features for choreography generation. The DFKI team has developed and evaluated this package using the open-source software Pinocchio developed by the INRIA Paris team with its recently introduced proximal formulation of the constrained dynamics.

Results Summary

In order to address the topic, two main contributions were introduced in this project: The first one is the proposition of a generic formalization of robot dancing which allows us to use musical features for choreography generation. Optimal dance trajectories were computed using direct optimal control. From this formalization we derive three different methods of dance generation, that differ in the level of flexibility, human involvement, and automatization. The methods are: imitated, improvised, and automatic choreography 35 generations. The imitated and improvised choreographic methods are based on beat timing extraction. The automatic choreography generation method,uses the additional music features volume and vocal melody.The results are validated on 4 different music pieces in simulation using the dynamic simulator MuJoCo as well as in experiments on the real robot RH5 Manus. This work was published in the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2022, and was selected as a finalist for best entertainment and amusement paper. The second one focuses on the ability of exploiting the full capabilities of a robot through motion generation, with the aim of achieving motions that are more human-like and that can lead to a certain trust and comfort feeling of the human towards the robot acting in its environment. To this purpose, we proposed a first study on resolving all the loop-closure constraints of the series-parallel hybrid robot RH5 Manus within the trajectory optimization process. To this end, we use the open-source software Pinocchio developed by the INRIA Paris team with its recently introduced proximal formulation of the constrained dynamics. This approach allows us to converge to an optimal solution according to the least squares principle, even in the context of singularities. Among the optimization methods available in the literature, the differential dynamics programming (DDP) approach was used to generate optimal trajectories with respect to the constrained dynamics. Results are presented in simulation as well as experiments on the real robot. This work is significant for humanoid robots based on electric actuation where one must seek to push the robot to its limits to achieve human like agility. This work has been submitted to the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2023 and is under review.

Tangible Outcomes

1. Melya Boukheddimi, Daniel Harnack, Shivesh Kumar, Rohit Kumar, Shubham Vyas, Octavio Arriaga, Frank Kirchner, Robot Dance Generation with Music Based Trajectory Optimization, In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2022, (IROS-2022), IEEE, Nov/2022.
2. [under review] Melya Boukheddimi , Rohit Kumar , Shivesh Kumar , Justin Carpentier , and Frank Kirchner, Investigations into Exploiting the Full Capabilities of a Series-Parallel Hybrid Humanoid using Whole Body Trajectory Optimization, In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2023, (IROS-2022), IEEE, Nov/2023.
3. Videohttps://www.youtube.com/watch?v=aN_v39p17tg
4. Video : https://www.youtube.com/watch?v=MA42YUg3e8E