Tamim Asfour
Learning Manipulation Constraints from Human Demonstration for Humanoid Robots
The era of functional humanoid robots has arrived, with AI-powered systems demonstrating remarkable locomotion and whole-body control capabilities. However, exciting systems still lack the versatile manipulation capabilities required for general-purpose tasks. This gap is critical because robotic manipulation stands as one of the grand challenges in robotics—not only a technical hurdle but a fundamental gateway to intelligence. Manipulation, the ability to change the physical world, is intrinsically intertwined with intelligence, the ability to detect change and adapt to it.
I argue that manipulation is fundamentally about understanding and leveraging constraints—geometric, kinematic, force, spatial and temporal relationships between objects and hands. Learning these constraints enables generalization across object variations, environmental changes, and task contexts. The talk addresses key questions: How do we detect, represent, and learn constraints from human demonstrations and environmental interaction? How do we leverage these constraints for task execution? I will present taxonomies as structured frameworks for understanding the space of possible constraints and approachs for keypoint-based visual imitation learning that extract keypoints, geometric constraints and movement primitives from human demonstration. Examples from our ARMAR humanoid robots will illustrate these concepts in complex bimanual manipulation tasks.
The talk concludes with perspectives on extending this constraint-based framework to the manipulation of deformable objects, a frontier challenge requiring richer constraint representations.
Renaud Detry
Efficient and Uncertainty-Aware Learning for Robot Manipulation: Insights Applicable to Deformable Objects
This presentation explores key advancements in robot manipulation that promise significant impact on the manipulation of deformable objects.
First, we discuss Diffusion Policy Learning (DDPM), a method already proven effective with deformable objects. Our work addresses the high computational costs of DDPM by presenting a novel imitation learning approach that leverages an asymmetry between robot and image diffusion. This technique reduces the training time for multi-task policies by 95% and memory usage by 93%. This efficiency is crucial, making it practical to effectively learn the high-dimensional, complex action spaces typical of deformable object manipulation.
Second, we tackle the problem of decision-making under extreme state uncertainty, a central challenge with deformable objects. We model visual uncertainty using a variational autoencoder, propagating its latent variance to dynamically modulate a maximum-entropy reinforcement learning algorithm. This method encourages uncertainty-reducing exploration, which is crucial when visual data is noisy or occluded.
Finally, we introduce methods to enhance robot manipulation with equivariance. Specifically, we present a new grasping model that is equivariant to planar rotations, directly improving the efficiency of tasks like tabletop folding or arranging pliable materials. The model uses a rotation-equivariant tri-plane representation and an equivariant generative grasp planner. This work serves as an inspiration for other equivariances vital for handling deformable objects.
Together, these contributions pave the way for a new generation of mobile manipulators capable of reasoning under uncertainty and learning effectively from less data, accelerating the development of robust solutions for deformable object manipulation.
Noémie Jaquier
A differential geometric take on deformable object modeling
Lagrangian and Hamiltonian mechanics provide a robust framework for modeling the dynamics of physical systems by leveraging fundamental energy conservation laws. Integrating such physical consistency as an inductive bias into deep neural architectures has been shown to enhance prediction accuracy, data efficiency, and generalization when modeling mechanical systems. However, scaling these architectures to deformable objects remains challenging due to their high — or even infinite — dimensionality. In this talk, I will explore recent advances that adopt a differential geometric perspective on Lagrangian and Hamiltonian mechanics to extend the scope of physics-inspired neural networks to high-dimensional mechanical systems. I will present geometric architectures that learn physically consistent, interpretable, and low-dimensional dynamics that effectively capture the complex, high-dimensional behavior of deformable objects.