With human feedback, AI-driven robots learn tasks better and faster

At UC Berkeley, researchers in Sergey Levine’s Robotic AI and Learning Lab eyed a table where a tower of 39 Jenga blocks stood perfectly stacked. Then a white-and-black robot, its single limb doubled over like a hunched-over giraffe, zoomed toward the tower, brandishing a black leather whip. Through what might have seemed to a casual viewer like a miracle of physics, the whip struck in precisely the right spot to send a single block flying from the stack while the rest of the tower remained structurally sound.

This task, known as “Jenga whipping,” is a hobby pursued by people with the dexterity and reflexes to pull it off. Now, it’s been mastered by robots, thanks to a novel, AI-powered training method created by Levine and other members of the team.

The new system, called Human-in-the-Loop Sample Efficient Robotic Reinforcement Learning (HiL-SERL), is described in a study appearing Aug. 20 in the journal Science Robotics.

By studying demonstrations and learning from both human feedback and its own real-world attempts, this training protocol teaches robots how to perform complicated tasks like Jenga whipping with a 100% success rate. Moreover, robots are taught at an impressive speed, enabling them to learn within one to two hours how to perfectly assemble a computer motherboard, build a shelf and more.

The first time the robot conquered the Jenga whipping challenge, “that really shocked me,” said study first author Jianlan Luo, a postdoctoral researcher at UC Berkeley. “The Jenga task is very difficult for most humans. I tried it with a whip in my hand; I had a 0% success rate.”

In recent years, the robot learning field has sought to crack the challenge of how to teach machines activities that are unpredictable or complicated, as opposed to a single action, like repeatedly picking up an object from a particular place on a conveyor belt. To solve this quandary, Levine’s lab has zeroed in on what’s called “reinforcement learning.” In reinforcement learning, a robot attempts a task in the real world and, using feedback from cameras, learns from its mistakes to eventually master that skill.

The new study added human intervention to speed up this process. With a special mouse that controls the robot, a human can correct the robot’s course, and those corrections can be incorporated into the robot’s proverbial memory bank. Using reinforcement learning, the robot analyzes the sum of all its attempts—assisted and unassisted, successful and unsuccessful—to better perform its task. Luo said a human needed to intervene less and less as the robot learned from experience.

“I needed to babysit the robot for maybe the first 30% or something, and then gradually I could actually pay less attention,” he said.

The lab put its robotic system through a gauntlet of complicated tasks beyond Jenga whipping. The robot flipped an egg in a pan; passed an object from one arm to another; and assembled a motherboard, car dashboard and timing belt. The researchers selected these challenges because they were varied, and in Luo’s words, represented “all sorts of uncertainty when performing robotic tasks in the complex real world.”

The researchers also tested the robots’ adaptability by staging mishaps. They’d force a gripper to open so it dropped an object or move a motherboard as the robot tried to install a microchip, training it to react to a shifting situation it might encounter outside a lab environment.

By the end of training, the robot could execute these tasks correctly 100% of the time. The researchers compared their results to a common “copy my behavior” method known as behavioral cloning that was trained on the same amount of demonstration data; their new system made the robots faster and more accurate.

These metrics are crucial, Luo said, because the bar for robot competency is very high. Regular consumers and industrialists alike don’t want to buy an inconsistent robot. Luo emphasized that in particular, “made-to-order” manufacturing processes like those often used for electronics, automobiles and aerospace parts could benefit from robots that can reliably and adaptably learn a range of tasks.

A next step, Luo said, would be to pre-train the system with basic object manipulation capabilities, eliminating the need to learn those from scratch and instead progressing straight to acquiring more complex skills. The lab also chose to make its research open source so that other researchers could use and build on it.

“A key goal of this project is to make the technology as accessible and user-friendly as an iPhone,” Luo said. “I firmly believe that the more people who can use it, the greater impact we can make.”

Additional authors of the study include Charles Xu and Jeffrey Wu of UC Berkeley.