How to leverage large language models for robotic grasping and code generation
2 hours ago
Grasping and manipulation remain a hard, unsolved problem in robotics. Grasping is not just about identifying points where to put your fingers on an object to create sufficient constraints. Grasping is also about applying just enough force to pick up the object with breaking it, while making sure it can be put to its intended use. At the same time, grasping provides critical sensor input to detect what an object is and what its properties are. With mobility essentially solved, grasping and manipulation remains the final frontier in unlocking truely autonomous labor replacements.
Imagine you are sending your humanoid robot companion to the supermarket and tell it âCheck the avocados for ripeness and grab an avocado for guacomole todayâ. There is a lot of stuff going on:
The quality âripenessâ is not obvious from the avocadoâs color as would be the case for a strawberry or a tomato, but requires tactile informationâGrab an avocadoâ, in particular a ripe one, implies a certain gentleness when handling it. When picking up an Avocado, you might be less careful than when picking up a raspberry, but also be cognizant of not damaging the legume.âGuacomole todayâ implies a certain ripeness level that is different than âfor Guacomole in three days from nowâ.
