Engineering household robots to have a little common sense

Although robots are good at mimicking human actions, they struggle to handle unexpected situations unless engineers program them to adapt. MIT engineers have developed a method that combines robot motion data with the “common sense knowledge” of large language models (LLMs) to give robots the ability to adjust to disruptions and continue tasks without starting over.

This new approach allows robots to break down household tasks into subtasks and adapt to failures within a subtask, improving overall task success without the need for constant manual intervention.

Yanwei Wang, a graduate student at MIT, explains, “Imitation learning is common for household robots, but errors can accumulate if a robot blindly mimics a human’s movements. Our method enables robots to correct errors and enhance task performance.”

The research team will present their findings at the International Conference on Learning Representations in May. The study’s co-authors include graduate students Tsun-Hsuan Wang and Jiayuan Mao, postdoc Michael Hagenow, and Professor Julie Shah.

The researchers demonstrate their approach using a simple task of scooping marbles. Instead of programming a robot to mimic a continuous motion, they break the task into subtasks and use LLMs to generate logical sequences of actions for the robot to follow.

By automatically connecting an LLM’s subtask labels with the robot’s physical position, the team’s algorithm enables the robot to self-correct and complete tasks even when faced with disruptions.

The team tested their approach with a robotic arm trained to scoop marbles. The robot successfully completed the task, adjusting its movements when nudged or pushed off course, without the need for additional programming or human intervention.

Wang concludes, “Our algorithm allows household robots to perform complex tasks efficiently, even in the presence of external disturbances.”