Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation

Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht, and Majid Khadiv

Peg-in-Hole — Easy
Peg-in-Hole — Hard
Vent Insertion
Cable Manipulation (2 cameras)
Cable Manipulation (3 cameras)

The Method Overview

Res-HIL overview: an interaction loop, where a frozen base policy and a learned residual policy act on the robot with optional human interventions, and a learning loop that updates the critics, the residual actor, and the target networks
Res-HIL runs two loops at the same time: an interaction loop that collects data on the robot, and a learning loop that trains the residual policy.

Res-HIL starts with a policy trained from demonstrations and keeps it frozen. Reinforcement learning then trains a residual policy to correct its actions. During training, a human operator can take control when the robot is likely to fail. Those interventions provide examples of the corrections to make and help identify autonomous actions that led to trouble.

  • Residual actions. The base policy produces an action from the robot's observation. The residual policy uses the observation and base action to produce a correction. The robot executes the sum of the two actions. We initialize the residual policy to output zero, so training starts with the base policy's behavior.
  • Human interventions. The operator can override the robot's action and guide it back to a recoverable state. We use the human action to teach the residual policy what correction to make. We also penalize autonomous actions immediately before an intervention.
  • Two replay buffers. The online buffer stores both autonomous and human-controlled transitions. A second buffer stores the initial demonstrations and intervention transitions. Each training batch draws equally from the two buffers.
  • Continuous learning. While the robot interacts with the task, a separate learning loop samples transitions, updates the critics and residual policy, and sends the updated weights back to the robot.
  • Learning the correction. We train the residual policy with TD3 using the robot's task reward and the intervention-shaped penalties. A behavior cloning loss teaches it to match corrections from demonstrations and interventions, while a regularization term discourages unnecessarily large corrections.