Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models

This paper presents Words2Contact, a language-guided multi-contact placement pipeline leveraging large language models and vision language models. Our method is a key component for language-assisted teleoperation and human-robot cooperation, where human operators can instruct the robots where to place their support contacts before whole-body reaching or manipulation using natural language. Words2Contact transforms the verbal instructions of a human operator into contact placement predictions; it also deals with iterative corrections, until the human is satisfied with the contact location identified in the robot's field of view. We benchmark state-of-the-art LLMs and VLMs for size and performance in contact prediction. We demonstrate the effectiveness of the iterative correction process, showing that users, even naive, quickly learn how to instruct the system to obtain accurate locations. Finally, we validate Words2Contact in real-world experiments with the Talos humanoid robot, instructed by human operators to place support contacts on different locations and surfaces to avoid falling when reaching for distant objects.

Flow Matching Imitation Learning for Multi-Support Manipulation

Humanoid robots could benefit from using their upper bodies for support contacts, enhancing their workspace, stability, and ability to perform contact-rich and pushing tasks. In this paper, we propose a unified approach that combines an optimization-based multi-contact whole-body controller with Flow Matching, a recently introduced method capable of generating multi-modal trajectory distributions for imitation learning. In simulation, we show that Flow Matching is more appropriate for robotics than Diffusion and traditional behavior cloning. On a real full-size humanoid robot (Talos), we demonstrate that our approach can learn a whole-body non-prehensile box-pushing task and that the robot can close dishwasher drawers by adding contacts with its free hand when needed for balance. We also introduce a shared autonomy mode for assisted teleoperation, providing automatic contact placement for tasks not covered in the demonstrations. Full experimental videos are available at: https://hucebot.github.io/flow_multisupport_website/

Multi-Contact Whole Body Force Control for Position-Controlled Robots

Many humanoid and multi-legged robots are controlled in positions rather than in torques, preventing direct control of contact forces, and hampering their ability to create multiple contacts to enhance their balance, such as placing a hand on a wall or a handrail. This paper introduces the SEIKO (Sequential Equilibrium Inverse Kinematic Optimization) pipeline, drawing inspiration from flexibility models used in serial elastic actuators to indirectly control contact forces on traditional position-controlled robots. SEIKO formulates whole-body retargeting from Cartesian commands and admittance control using two quadratic programs solved in real time. We validated our pipeline with experiments on the real, full-scale humanoid robot Talos in various multicontact scenarios, including pushing tasks, far-reaching tasks, stair climbing, and stepping on sloped surfaces. This work opens the possibility of stable, contact-rich behaviors while getting around many of the challenges of torque-controlled robots. Code and videos are available at https://hucebot.github.io/seiko_controller_website/ .

Feasibility Retargeting for Multi-contact Teleoperation and Physical Interaction

This short paper outlines two recent works on multi-contact teleoperation and the development of the SEIKO (Sequential Equilibrium Inverse Kinematic Optimization) framework. SEIKO adapts commands from the operator in real-time and ensures that the reference configuration sent to the underlying controller is feasible. Additionally, an admittance scheme is used to implement physical interaction, which is then combined with the operator's command and retargeted. SEIKO has been applied in simulations on various robots, including humanoid and quadruped robots designed for loco-manipulation. Furthermore, SEIKO has been tested on real hardware for bimanual heavy object carrying tasks.

Achieving Dexterous Bidirectional Interaction in Uncertain Conditions for Medical Robotics

Medical robotics can help improve and extend the reach of healthcare services. A major challenge for medical robots is the complex physical interaction between the robot and the patients which is required to be safe. This work presents the preliminary evaluation of a recently introduced control architecture based on the Fractal Impedance Control (FIC) in medical applications. The deployed FIC architecture is robust to delay between the master and the replica robots. It can switch online between an admittance and impedance behaviour, and it is robust to interaction with unstructured environments. Our experiments analyse three scenarios: teleoperated surgery, rehabilitation, and remote ultrasound scan. The experiments did not require any adjustment of the robot tuning, which is essential in medical applications where the operators do not have an engineering background required to tune the controller. Our results show that is possible to teleoperate the robot to cut using a scalpel, do an ultrasound scan, and perform remote occupational therapy. However, our experiments also highlighted the need for a better robots embodiment to precisely control the system in 3D dynamic tasks.

Multi-Contact Motion Retargeting using Whole-body Optimization of Full Kinematics and Sequential Force Equilibrium

This paper presents a multi-contact motion adaptation framework that enables teleoperation of high degree-of-freedom (DoF) robots, such as quadrupeds and humanoids, for loco-manipulation tasks in multi-contact settings. Our proposed algorithms optimize whole-body configurations and formulate the retargeting of multi-contact motions as sequential quadratic programming, which is robust and stable near the edges of feasibility constraints. Our framework allows real-time operation of the robot and reduces cognitive load for the operator because infeasible commands are automatically adapted into physically stable and viable motions on the robot. The results in simulations with full dynamics demonstrated the effectiveness of teleoperating different legged robots interactively and generating rich multi-contact movements. We evaluated the computational efficiency of the proposed algorithms, and further validated and analyzed multi-contact loco-manipulation tasks on humanoid and quadruped robots by reaching, active pushing and various traversal on uneven terrains.

Collaborative Bimanual Manipulation Using Optimal Motion Adaptation and Interaction Control

This work developed collaborative bimanual manipulation for reliable and safe human-robot collaboration, which allows remote and local human operators to work interactively for bimanual tasks. We proposed an optimal motion adaptation to retarget arbitrary commands from multiple human operators into feasible control references. The collaborative manipulation framework has three main modules: (1) contact force modulation for compliant physical interactions with objects via admittance control; (2) task-space sequential equilibrium and inverse kinematics optimization, which adapts interactive commands from multiple operators to feasible motions by satisfying the task constraints and physical limits of the robots; and (3) an interaction controller adopted from the fractal impedance control, which is robust to time delay and stable to superimpose multiple control efforts for generating desired joint torques and controlling the dual-arm robots. Extensive experiments demonstrated the capability of the collaborative bimanual framework, including (1) dual-arm teleoperation that adapts arbitrary infeasible commands that violate joint torque limits into continuous operations within safe boundaries, compared to failures without the proposed optimization; (2) robust maneuver of a stack of objects via physical interactions in presence of model inaccuracy; (3) collaborative multi-operator part assembly, and teleoperated industrial connector insertion, which validate the guaranteed stability of reliable human-robot co-manipulation.

Fine Manipulation and Dynamic Interaction in Haptic Teleoperation

Teleoperation of robots enables remote intervention in distant and dangerous tasks without putting the operator in harm's way. However, remote operation faces fundamental challenges due to limits in communication delay and bandwidth. The proposed work improves the performances of teleoperation architecture based on Fractal Impedance Controller (FIC), by integrating the most recent manipulation architecture in the haptic teleoperation pipeline. The updated controller takes advantage of the inverse kinematics optimisation in the manipulation, and hence improves dynamic interactions during fine manipulation without renouncing the robustness of the FIC controller. Additionally, the proposed method allows an online trade-off between the manipulation controller and the teleoperated behaviour, allowing a safe superimposition of these two behaviours. The validated experimental results show that the proposed method is robust to reduced communication bandwidth and delays. Moreover, we demonstrated that the remote teleoperated robot remains stable and safe to interact with, even when the communication with the master side is abruptly interrupted.

Robust Impedance Control for Dexterous Interaction Using Fractal Impedance Controller with IK-Optimisation

Robust dynamic interactions are required to move robots in daily environments alongside humans. Optimisation and learning methods have been used to mimic and reproduce human movements. However, they are often not robust and their generalisation is limited. This work proposed a hierarchical control architecture for robot manipulators and provided capabilities of reproducing human-like motions during unknown interaction dynamics. Our results show that the reproduced end-effector trajectories can preserve the main characteristics of the initial human motion recorded via a motion capture system, and are robust against external perturbations. The data indicate that some detailed movements are hard to reproduce due to the physical limits of the hardware that cannot reach the same velocity recorded in human movements. Nevertheless, these technical problems can be addressed by using better hardware and our proposed algorithms can still be applied to produce imitated motions.

Robust and Dexterous Dual-arm Tele-Cooperation using Fractal Impedance Control

Deploying robots from isolated operations to shared environments has been an increasing trend in robotics for the last decades. However, the requirement of robust interaction in highly variable environments is still beyond the capability of most robots. We proposed to achieve robustness of various interactions by using the Fractal Impedance Control (FIC) and exploiting its non-linear stiffness to adapt to multiple cooperative scenarios, which is applicable to both manipulation and teleoperation applications. The proposed method was evaluated by a wide range of experiments: drilling, moving objects with unknown dynamics, and interacting with humans. The extensive validations demonstrated that the proposed method is very robust in presence of delays and reduced bandwidth in the communication link between master and follower. The results confirmed that the proposed method can enhance the robustness significantly and allow switching tasks freely between manipulation, human-robot cooperation and teleoperation without the need of extensive re-tuning of the controllers.