The difficulty we just identified—contact-rich manipulation with vision, partial observability, tight timing, and rare successes—sets up the central claim of HIL-SERL. The paper is not saying that dexterous manipulation suddenly becomes easy because of a new reinforcement learning loss. In fact, one of the most important messages is almost the opposite: the RL algorithm only becomes useful when the rest of the robotic learning system is engineered to make RL’s assumptions less unreasonable.

A useful way to frame the claim is to contrast two ways of using human data. In behavioral cloning or HG-DAgger-style imitation, human behavior is treated primarily as a target distribution: the robot should learn to act like the demonstrator, or like the corrected policy under human supervision. This can work when the task is mostly kinematic, when the expert demonstrations cover the relevant states, and when small action errors do not qualitatively change the future. But dexterous manipulation often violates all three assumptions. A slightly late grasp, a missed contact, or a poorly timed push can send the object into a state that was never demonstrated, and then pure imitation has no principled way to decide how to recover.

HIL-SERL instead uses human data to solve a different bottleneck: exploration. Demonstrations and interventions help the robot visit states near successful behavior and avoid catastrophic or unproductive regions of the state space. But the policy is not improved merely by copying the human. The actual improvement is driven by off-policy reinforcement learning using task reward. That distinction matters because the reward can, in principle, rank behaviors by outcome rather than by resemblance to a person.

Put differently, imitation answers the question: “What action would the human take here?” HIL-SERL’s RL component answers: “Which actions lead to higher long-term task success from here?” Those questions often agree, but not always. A robot may have different dynamics, latency, controller structure, gripper geometry, or visual processing than the human teleoperator. A behavior that looks human-like may be unnecessarily slow or brittle; a behavior that looks slightly strange may be more reliable for the robot. HIL-SERL’s thesis is that human guidance should shape the data distribution, while reward-backed dynamic programming should decide what is actually good for the robot.

This is why the system-level ingredients are not incidental details. Sparse rewards are hard to use unless the replay buffer contains meaningful transitions. Vision is hard to learn from scratch unless the representation is stabilized by pretrained visual features. Real-world trial-and-error is dangerous unless low-level controllers and human interventions keep the robot inside a recoverable operating regime. Online RL is slow unless actor and learner processes can collect, relabel, and train from experience efficiently. Each ingredient reduces a different practical failure mode:

• Pretrained visual features reduce the burden of learning perception entirely from sparse robot reward.
• Classifier rewards turn human-provided success criteria into scalable task feedback.
• Human interventions prevent long stretches of useless or unsafe exploration.
• Off-policy replay allows demonstrations, corrections, and autonomous rollouts to all contribute to the same learning process.
• Safe low-level control makes high-level exploration physically feasible on real hardware.
• Distributed actor–learner infrastructure improves wall-clock efficiency, which is crucial when robot time is the scarce resource.

The comparison with HG-DAgger or behavioral cloning is therefore not simply “RL beats imitation.” A more precise reading is: imitation with comparable human data can still inherit the fragility of supervised learning under distribution shift. HIL-SERL changes the role of that data. The human is no longer just a labeler whose actions define the desired policy; the human becomes part of a data-collection mechanism that keeps the replay buffer informative while reinforcement learning performs policy improvement.

This also explains why the reported outcomes are so striking. The paper reports that imitation-style baselines achieve roughly 49.7% average success and are about 1.8× slower in cycle time, while HIL-SERL reaches near-100% success after roughly 1–2.5 hours for most tasks, with the timing belt task taking longer. The timing-belt exception is important: it reminds us that the method is not magic. When the task demands particularly difficult predictive dynamics, precise timing, or rare successful contact sequences, even a well-designed RL system needs more experience.

The deeper takeaway is that sample-efficient real-world RL is less about a single algorithmic trick and more about aligning the entire pipeline with the structure of the learning problem. HIL-SERL makes RL practical by arranging for the robot to see useful states, receive usable reward, train from mixed sources of experience, and stay physically safe while improving. The reinforcement learner is the optimization engine, but the surrounding system determines whether that engine ever receives fuel of sufficient quality.

The visual below condenses this argument into a direct comparison. The key distinction is not merely the performance numbers, but the column labeled by what drives improvement: imitation methods improve by matching human behavior, while HIL-SERL improves by combining human-guided data collection with sparse-reward off-policy RL.

The bottom takeaway lines in the visual should be read as the paper’s central design philosophy. HIL-SERL is a recipe: pretrained vision, reward classification, RLPD-style off-policy learning, human interventions, safe controllers, and distributed replay are all arranged so that human data guides exploration, but reward-driven dynamic programming selects and reinforces successful robot behavior.

If the previous claim is that reinforcement learning becomes practical only when the whole robotic system is designed around it, the next step is to make precise what “the RL problem” actually is. In HIL-SERL, the authors are not treating dexterous manipulation as an abstract benchmark with clean simulator states and shaped rewards. They are solving a finite-horizon, vision-based, real-world robotic Markov decision process in which the robot must act from noisy observations, collect sparse success signals, and improve using data gathered on the physical system.

Formally, each manipulation task is modeled as an MDP,

$$\mathcal{M}=\{\mathcal{S},\mathcal{A},\rho,\mathcal{P},r,\gamma\}.$$

Here $\mathcal{S}$ is the observation/state space, $\mathcal{A}$ is the action space, $\rho$ is the initial-state distribution induced by resets, $\mathcal{P}$ is the unknown real-world transition dynamics, $r$ is the reward function, and $\gamma\in(0,1)$ is the discount factor. The word “unknown” matters: the robot does not get access to $\mathcal{P}$, and in the real world $\mathcal{P}$ includes contact dynamics, actuator delays, perception noise, object slippage, human reset variation, and all the other small effects that make manipulation difficult to model.

The observation $\mathbf{s}_t\in\mathcal{S}$ is not a compact simulator state such as object pose plus joint angles. It contains camera observations $\mathbf{o}^{img}$ together with proprioception $\mathbf{p}$, such as robot joint or end-effector information. This is one of the central reasons the setting is hard: the policy must infer task-relevant physical state from pixels, while also reacting quickly enough to contacts and object motion. A cup sliding, a card bending, or a toy slipping from the gripper may be obvious to a human observer, but to the policy it is just a change in high-dimensional image features.

The action $\mathbf{a}_t\in\mathcal{A}$ is typically a low-level command to the robot controller. In HIL-SERL-style systems, this is often decomposed into an end-effector command $\mathbf{a}_{eef}\in\mathcal{A}_1$, optionally combined with a gripper command $\mathbf{a}_{gripper}\in\mathcal{A}_2$. This design choice is subtle but important. The policy is not directly outputting motor torques for every actuator; it acts through a controller abstraction that makes exploration safer and more sample-efficient. The RL policy still learns meaningful manipulation behavior, but the low-level controller absorbs some of the burden of stable motion execution.

The learning objective is the standard discounted finite-horizon return:

$$\max_\theta\; \mathbb{E}_{\mathbf{s}_0\sim\rho,\;\mathbf{a}_t\sim\pi_\theta(\cdot\mid\mathbf{s}_t),\;\mathbf{s}_{t+1}\sim\mathcal{P}(\cdot\mid\mathbf{s}_t,\mathbf{a}_t)} \left[ \sum_{t=0}^{H}\gamma^t r(\mathbf{s}_t,\mathbf{a}_t) \right].$$

This equation says that we want policy parameters $\theta$ that produce actions leading to high cumulative reward over a horizon $H$. The expectation is over three sources of randomness: the reset distribution $\rho$, the policy’s own action sampling, and the environment transitions. In a real robot, all three are consequential. Resets may place objects slightly differently, stochastic policies explore different contacts, and the same command can produce different outcomes depending on friction, compliance, and object pose.

The reward in HIL-SERL is intentionally sparse. Instead of hand-engineering a dense reward based on distances, angles, or object poses, the system uses a learned visual success classifier:

$$r(\mathbf{s},\mathbf{a})=C_\omega(\mathbf{o}^{img})\in\{0,1\}.$$

The classifier $C_\omega$ looks at the image observation and predicts whether the task has succeeded. This has a major practical advantage: many real manipulation tasks have outcomes that are easier to recognize than to quantify continuously. It may be straightforward to label whether the object is inserted, opened, placed, or grasped, while being much harder to define a smooth reward that guides every intermediate contact-rich motion.

But sparse rewards also introduce a serious learning challenge. Most random behavior produces $0$ reward, especially in dexterous manipulation. From the perspective of a policy learning from scratch, the reward landscape may look almost empty until the robot accidentally succeeds. This is exactly why HIL-SERL cannot be understood as “just run RL on a robot.” The sparse classifier reward defines the goal cleanly, but the surrounding system—demonstrations, human interventions, off-policy replay, visual pretraining, and controller design—is what makes the sparse signal learnable.

The discount factor $\gamma$ adds another important bias: among policies that eventually succeed, the objective prefers policies that succeed sooner. If the same success occurs at time $t$ rather than $t+N$, its contribution is weighted by $\gamma^t$ instead of $\gamma^{t+N}$. Since $\gamma<1$,

$$\gamma^t>\gamma^{t+N}.$$

So the reward formulation does not merely encode “succeed eventually.” It also encodes a pressure toward efficient completion. In robotic manipulation this matters because slow, hesitant, or overly indirect behavior may be technically successful but practically undesirable. A policy that reacts quickly to object motion, recovers from small errors, and completes the task in fewer steps receives a larger return.

The visual summary that follows compresses this formulation into the core ingredients: the MDP definition, the observation and action structure, the discounted objective, the sparse classifier reward, and the timing preference induced by discounting. Read it as a map of the problem HIL-SERL is solving, not yet the algorithm used to solve it.

In particular, notice the separation between the unknown real-world dynamics and the learned policy objective. The robot samples trajectories from resets, its current policy, and physical transitions; the reward classifier then turns visual outcomes into binary reinforcement learning data. That distinction will become crucial in the next step, where we compare HIL-SERL against prior pipelines that either rely too heavily on imitation labels or struggle to make sparse-reward RL work from scratch.

With the robotic RL objective now explicit, the central difficulty becomes less about writing down the goal and more about constructing a learning pipeline that can actually make progress toward it on hardware. For dexterous manipulation, the objective is deceptively simple:

$$\max_\theta\;\mathbb{E}\left[\sum_{t=0}^{H}\gamma^t r(\mathbf{s}_t,\mathbf{a}_t)\right].$$

The policy $\pi_\theta$ should choose actions that accumulate reward over a finite horizon $H$, usually under sparse success signals: did the robot complete the task or not? But this objective hides three practical requirements. A successful method must reach success at least sometimes, improve the behavior beyond the data it was given, and recover from the states its own imperfect policy creates.

That last point is especially important. In real robot learning, the state distribution is not fixed. A policy trained on demonstrations will not only visit the clean, expert-like states in the dataset; it will also drift into awkward poses, partial grasps, collisions, occlusions, near-failures, and timing errors. These are precisely the states where robust manipulation is decided. A method that learns only from expert trajectories may perform well while it remains close to the expert distribution, but once it deviates, it often lacks the information needed to return to success.

This explains why behavior cloning and related imitation methods, including diffusion-policy-style action modeling, can be strong yet brittle. Their learning signal is supervised: match the demonstrated action $\mathbf{a}$ given an observed state $\mathbf{s}$. Formally, they optimize something closer to an action prediction loss than the return objective above. That makes them data-efficient and stable, but it also creates a ceiling: they are trained to reproduce the demonstrations, not to discover actions that are faster, more reliable, or better adapted to the current trial. If the human demonstrator takes a conservative path, pauses unnecessarily, or uses a strategy that works only under certain object poses, the cloned policy inherits those limitations.

There is also a distribution-shift failure mode. Suppose the policy makes a small early mistake, causing the object to rotate differently than in the demonstrations. The policy now observes a state $\mathbf{s}_t$ that was rare or absent in the dataset. The supervised target is either unavailable or extrapolated poorly. Even a highly expressive model can fail here, because the problem is not merely capacity; it is that the training data did not tell the learner what action leads back to high return from this off-distribution state.

Human-guided variants such as HG-DAgger address part of this problem by collecting corrective actions when the learned policy begins to fail. This improves coverage of the states induced by the current policy, which is a major advantage over static imitation. However, if those interventions are treated purely as supervised labels, the method still learns to imitate the human correction $\mathbf{a}_{itv}$ rather than evaluate whether that correction leads to better long-term outcomes. Human corrections can be noisy, delayed, inconsistent across operators, or locally reasonable but globally suboptimal. Supervised learning has no direct mechanism to ask: did this intervention actually increase return?

At the opposite extreme, reinforcement learning from scratch optimizes the right objective. In principle, it directly searches for actions that maximize $r(\mathbf{s}, \mathbf{a})$ over time. But sparse-reward robotic manipulation is a brutal exploration problem. The useful reward may appear only after a long sequence of precise contacts, object motions, and end-effector configurations. In a large continuous space $\mathcal{S}\times\mathcal{A}$, random exploration almost never discovers success, especially for long horizons $H$. The method is optimizing the right quantity, but it may receive too little informative signal to begin learning.

This motivates demonstration-bootstrapped off-policy RL methods such as SERL or RLPD-style training with a fixed demonstration buffer $\mathcal{D}_{demo}$. These approaches are more aligned with the return objective: demonstrations provide initial successful transitions, and off-policy RL can in principle improve beyond them by estimating values and optimizing actions. This is already a better compromise than pure imitation or pure scratch RL. But if the replay buffer contains only demonstrations plus autonomous rollouts, it may still miss the most valuable data: corrective transitions from the specific failure states produced by the current policy.

The core gap is therefore not simply “more data.” It is the absence of the right kind of data paired with the right kind of learning signal. Prior pipelines tend to lack at least one of the following:

• Reward optimization: imitation matches actions but does not directly optimize return.
• Exploration support: scratch RL optimizes return but rarely finds sparse successes.
• Corrective data at policy-induced states: demo-bootstrapped RL helps early learning but may not observe enough recoveries from the learner’s own mistakes.

This is exactly the opening that HIL-SERL exploits. Human interventions are not merely labels for supervised imitation; they become off-policy RL experience. The human helps the robot visit meaningful recovery trajectories, while the RL update still evaluates transitions through reward and bootstrapped value estimates. In other words, interventions supply exploration and corrective coverage, but the algorithm remains anchored to the objective

$$\mathbb{E}\left[\sum_{t=0}^{H}\gamma^t r(\mathbf{s}_t,\mathbf{a}_t)\right],$$

rather than collapsing into action matching.

The visual below condenses this comparison into a compact failure-mode table. The important pattern is that each prior pipeline solves one part of the problem while leaving another exposed: imitation has demonstrations but no direct reward optimization; HG-DAgger has corrective coverage but still treats corrections as labels; scratch RL has the right objective but insufficient exploration; demo-seeded RL has reward optimization but not necessarily intervention data from the current policy’s hardest states.

The bottom-line gap is the design target for the next section: demonstrations help initialize learning, but online human interventions target the current policy’s mistakes. HIL-SERL’s architecture is built around turning that insight into a practical distributed robot-learning system.

The failure modes we just discussed all point to the same engineering lesson: for real robots, the learning algorithm is only half the story. A vision-based dexterous manipulation policy must improve from scarce experience, but it also has to keep the robot moving safely, accept human corrections at the right moments, assign rewards from noisy observations, and update without interrupting control. HIL-SERL’s core architectural move is therefore to separate acting from learning: the robot runs a lightweight actor process in real time, while a separate learner process performs the expensive off-policy updates asynchronously.

At each control step, the actor observes a state consisting of image observations and proprioceptive information,

$$\mathbf{s}_t = (\mathbf{o}^{img}, \mathbf{p}),$$

where $\mathbf{o}^{img}$ is the visual input and $\mathbf{p}$ includes robot state such as end-effector pose, gripper state, or other low-dimensional proprioceptive features. The policy $\pi_\theta(\mathbf{a}\mid \mathbf{s}_t)$ proposes an action $\mathbf{a}_{RL}$, typically at roughly $10$ Hz. That rate matters: it is fast enough to support reactive manipulation, but slow enough that neural-network inference, sensing, and low-level control can be coordinated reliably on real hardware.

The human operator is not used as a source of labels in the usual behavioral-cloning sense. Instead, the human is inserted into the action execution path. When no intervention occurs, the robot executes the RL policy’s action. When the operator takes control, the executed action is replaced by the intervention action:

$$\mathbf{a}^* = \begin{cases} \mathbf{a}_{itv}, & I_t = 1,\\ \mathbf{a}_{RL}, & I_t = 0. \end{cases}$$

This distinction is subtle but crucial. The system does not merely record, “the human would have done this,” and then train the policy to imitate that action with supervised learning. Instead, it treats the executed intervention as part of the environment trajectory. The transition $(\mathbf{s}_t,\mathbf{a}^*,r_t,\mathbf{s}_{t+1})$ enters replay and is later consumed by the same off-policy RL machinery as ordinary experience. In other words, human control changes the data distribution, not the objective.

This is what makes the method different from a pure imitation-learning pipeline. A teleoperated action may be locally useful, but the learning signal still comes through Bellman backups: if the intervention leads to higher future return, the critic can assign value to nearby states and actions, and the actor can gradually move toward those high-value choices. Conversely, if an intervention is imperfect or only partially helpful, the RL update is not forced to clone it exactly. The data is useful because it reaches informative states and avoids catastrophic dead ends, not because every human action is assumed to be globally optimal.

The reward pathway is also deliberately separated from the control pathway. For sparse manipulation tasks, hand-designed reward functions are often brittle: small pose errors, occlusions, and contact-rich dynamics make geometric success tests hard to specify robustly. HIL-SERL instead uses a reward classifier $C_\omega(\mathbf{o}^{img})$ to detect task success from images, producing the sparse reward used by the learner:

$$r(\mathbf{s}_t,\mathbf{a}^*) \;\text{from}\; C_\omega(\mathbf{o}^{img}).$$

This does not make the problem dense or easy—the reward may still be sparse, delayed, and classifier-dependent—but it makes the system practical across visually defined tasks. The classifier becomes a reusable success detector, while the policy and critic learn how to reach states that trigger it.

The replay design reflects the two kinds of experience HIL-SERL wants to preserve. One buffer, $\mathcal{D}_{demo}$, is initialized with a small number of teleoperated demonstrations, often on the order of $20$--$30$ trajectories. During online training, it also stores intervention segments, because those segments are disproportionately informative: they occur precisely when the current policy is uncertain, unsafe, or about to fail. A second buffer, $\mathcal{D}_{RL}$, stores online transitions generated by the current policy, including transitions immediately before and after interventions. Those adjacent transitions are important because they capture the states where autonomy and human assistance meet—the boundary where the policy most needs improvement.

The learner samples mixed batches from these replay sources, updates the critic $Q_\phi$ and policy $\pi_\theta$, and periodically sends fresh parameters back to the actor. Because this happens asynchronously, the robot does not need to pause while gradients are computed. The actor may be running slightly stale parameters, but this is acceptable in an off-policy setting: the replay buffer already contains data from many behavior policies, including old policies, current policies, demonstrations, and human interventions. The algorithmic burden is then shifted to the off-policy update rule, which we will derive next.

There are a few assumptions hiding in this architecture. First, the low-level controller must execute actions consistently enough that replayed experience is meaningful. Second, the reward classifier must be reliable enough that success is not systematically mislabeled. Third, the intervention interface must allow the human to take over quickly, because delayed overrides can still produce bad data or unsafe contact. Finally, the learner must balance demonstration-like data and online data carefully: too much reliance on demonstrations can overfit to narrow trajectories, while too little can leave the robot exploring blindly in a sparse-reward task.

The visual below condenses this into a distributed actor-learner loop. The left side corresponds to real-time robot control: observations become state, the policy proposes $\mathbf{a}_{RL}$, the human can override with $\mathbf{a}_{itv}$, and only the selected action $\mathbf{a}^*$ is executed. The reward classifier supplies the sparse success signal, and the resulting transition is written into replay.

The right side represents the slower learning process. Demonstrations, intervention segments, and ordinary online transitions are sampled together to update $Q_\phi$ and $\pi_\theta$; updated parameters then flow back to the robot. The key takeaway is that HIL-SERL does not bolt human help onto RL as a separate supervised module. It turns human help into off-policy reinforcement learning data, allowing the same actor-critic machinery to learn from both autonomy and intervention.

With the distributed architecture in place, the next question is what the learner actually optimizes while the robot is collecting experience, humans are intervening, and demonstration data is sitting in replay. HIL-SERL does not introduce a new reinforcement learning objective for interventions. Its learning rule is much closer to a standard off-policy maximum-entropy actor-critic method, with one crucial practical choice: every minibatch is deliberately mixed between prior human data and online robot experience.

The starting point is the Bellman backup. For a transition $(\mathbf{s}, \mathbf{a}, \mathbf{s}')$, the critic should estimate the immediate reward plus the discounted value of what the current policy would do next:

$$r(\mathbf{s},\mathbf{a})+\gamma\mathbb{E}_{\mathbf{a}'\sim\pi_\theta(\cdot\mid\mathbf{s}')}\left[\bar Q_\phi(\mathbf{s}',\mathbf{a}')\right].$$

Here $\bar Q_\phi$ denotes a target critic, usually an exponential moving average of the learned critic. This target network is a stabilizing device: without it, the critic would be chasing a target that changes too quickly because the same network would be used both to predict and to define its own regression label. The expectation over $\mathbf{a}'\sim\pi_\theta(\cdot\mid\mathbf{s}')$ is also important. The backup is not asking, “what did the human or robot actually do next in the dataset?” It asks, “under the current policy, what action would we sample next, and how valuable would that be?”

That distinction is what makes the update off-policy. The dataset may contain actions from several behavior sources: scripted resets, initial demonstrations, human interventions, failed autonomous attempts, and successful autonomous recoveries. But the critic target evaluates continuation under the current learned policy $\pi_\theta$. This is exactly the property HIL-SERL needs: human intervention data can be reused as reinforcement learning experience without converting it into a supervised behavioral cloning target.

The critic is trained by regressing $Q_\phi(\mathbf{s},\mathbf{a})$ toward this one-step target:

$$\mathcal{L}_Q(\phi)=\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}')\sim\mathcal{D}_{batch}}\left[\left(Q_\phi(\mathbf{s},\mathbf{a})-\left(r(\mathbf{s},\mathbf{a})+\gamma\mathbb{E}_{\mathbf{a}'\sim\pi_\theta(\cdot\mid\mathbf{s}')}\left[\bar Q_\phi(\mathbf{s}',\mathbf{a}')\right]\right)\right)^2\right].$$

Intuitively, this loss says: given the state-action pairs that actually occurred, estimate how good they were according to reward plus future policy value. Demonstration actions and intervention actions can therefore become high-value evidence if they lead to good outcomes. But they are not treated as commands that the policy must imitate forever. This matters in dexterous manipulation, where the human may intervene only in rare recovery states, may use actions that are safe but not optimal, or may teleoperate with latency and discontinuities. A supervised objective would tend to clone these artifacts; an off-policy RL objective can instead learn their long-term consequences.

The actor update then changes the policy to choose actions that the critic currently rates highly, while preserving entropy:

$$\mathcal{L}_\pi(\theta)=-\mathbb{E}_{\mathbf{s}\sim\mathcal{D}_{batch}}\left[\mathbb{E}_{\mathbf{a}\sim\pi_\theta(\cdot\mid\mathbf{s})}\left[Q_\phi(\mathbf{s},\mathbf{a})\right]+\alpha\mathcal{H}(\pi_\theta(\cdot\mid\mathbf{s}))\right].$$

The negative sign appears because we usually write training as minimization. Minimizing $\mathcal{L}_\pi$ is equivalent to maximizing critic value plus an entropy bonus. The entropy term $\alpha\mathcal{H}(\pi_\theta(\cdot\mid\mathbf{s}))$ discourages premature collapse to a deterministic policy, which is especially useful when the robot is still discovering contact-rich strategies. In manipulation, many states are ambiguous from vision alone, and small exploratory differences in pose, timing, or force can determine whether a grasp, insertion, or reorientation succeeds.

The subtlety is that both the critic and actor expectations are taken over $\mathcal{D}_{batch}$, not over only the most recent autonomous rollouts. HIL-SERL follows the RLPD-style idea of mixing demonstration and online replay inside every update:

$$\mathcal{D}_{batch}=\lambda\mathcal{D}_{demo}+(1-\lambda)\mathcal{D}_{RL},\qquad \lambda=0.5.$$

With $\lambda=0.5$, half of each training batch comes from human-provided data and half comes from robot-collected RL data. This is a simple design choice, but it has large consequences. If the learner trains mostly on online replay early on, it may drown in failures before it has enough signal to improve. If it trains mostly on demonstrations forever, it may fail to adapt to the distribution induced by its own policy. Equal mixing keeps the learner anchored to competent behavior while still forcing it to evaluate and improve on the states it actually visits.

There are also failure modes hidden in this formulation. The critic can overestimate unseen actions, especially with high-dimensional visual observations and sparse rewards. The actor may exploit those errors by choosing actions that look valuable only because the critic is poorly calibrated. Demonstration mixing reduces, but does not eliminate, this risk by repeatedly exposing the critic to meaningful state-action regions. Target networks, entropy regularization, and continued online data collection all work together to keep the Bellman updates from drifting too far away from grounded robot experience.

The key takeaway is that HIL-SERL’s human-in-the-loop mechanism is not “human labels plus RL.” It is off-policy RL with human data in the replay distribution. Human demonstrations and interventions shape the empirical distribution of transitions used for Bellman regression; the actor is still optimized through the critic, not through direct imitation. This is what lets the method combine the reliability of human guidance with the adaptability of reinforcement learning.

The visual below condenses this derivation into a top-to-bottom flow: first define the one-step Bellman target, then regress the critic toward that target, then improve the policy under the learned critic. The replay mixture at the bottom is not an implementation detail tacked on afterward; it is the sampling distribution underlying both losses.

Reading the equations this way also clarifies why RLPD is such a natural fit for HIL-SERL. The mathematical update remains a standard actor-critic update, but the batch construction ensures that every gradient step sees both human competence and robot experience. That mixture is the bridge between safe early learning and eventual autonomous improvement.

With the Bellman target and actor objective in place, the algorithmic step in HIL-SERL is surprisingly simple: do ordinary off-policy actor-critic learning, but draw each minibatch from a mixture of replay sources. The key point is that demonstrations are not treated as a separate supervised imitation loss, and online rollouts are not treated as a separate RL phase. They are all just transitions of the form $(\mathbf{s}, \mathbf{a}, r, \mathbf{s}')$ that can participate in the same Bellman backup.

Concretely, each update constructs a minibatch by sampling some fraction from a demonstration buffer and the rest from the online RL buffer:

$$\mathcal{D}_{batch} = \lambda \mathcal{D}_{demo} + (1-\lambda)\mathcal{D}_{RL}.$$

Here $\lambda$ is not a probability distribution over individual transitions in a formal dataset-mixture sense so much as a practical batching rule: if the batch size is $B$, sample approximately $\lambda B$ transitions from $\mathcal{D}_{demo}$ and $(1-\lambda)B$ transitions from $\mathcal{D}_{RL}$. The RL buffer may contain autonomous policy data, human intervention data, and later successful trajectories produced by the improving policy. Once sampled, however, the update does not remember which transition came from which source.

That uniform treatment matters. In behavior cloning, a demonstration action is interpreted as a label: “in this state, copy this action.” In RLPD-style learning, the demonstration action is instead interpreted as an action whose long-term value should be evaluated. The critic asks: if the robot took this action in this state, received reward $r(\mathbf{s}, \mathbf{a})$, and then continued according to the current policy, what return would we estimate? The target is

$$r(\mathbf{s},\mathbf{a}) + \gamma \mathbb{E}_{\mathbf{a}' \sim \pi_\theta} \left[ \bar Q_\phi(\mathbf{s}',\mathbf{a}') \right].$$

This is the same target whether $\mathbf{a}$ came from an expert demonstration, a human correction, a partially trained robot policy, or a failed autonomous attempt. The critic loss is therefore

$$\mathcal{L}_Q(\phi) = \mathbb{E}_{\mathbf{s},\mathbf{a},\mathbf{s}'} \left[ \left( Q_\phi(\mathbf{s},\mathbf{a}) - \left( r(\mathbf{s},\mathbf{a}) + \gamma \mathbb{E}_{\mathbf{a}'\sim\pi_\theta} [ \bar Q_\phi(\mathbf{s}',\mathbf{a}') ] \right) \right)^2 \right].$$

The actor is then improved using the critic’s judgment. Rather than explicitly cloning demonstration actions, the policy is optimized to choose actions that the learned $Q$-function rates highly, while retaining entropy for exploration and robustness:

$$\mathcal{L}_\pi(\theta) = - \mathbb{E}_{\mathbf{s}} \left[ \mathbb{E}_{\mathbf{a}\sim\pi_\theta} [ Q_\phi(\mathbf{s},\mathbf{a}) ] + \alpha \mathcal{H} \bigl( \pi_\theta(\cdot \mid \mathbf{s}) \bigr) \right].$$

This distinction is subtle but important. Demonstrations shape the value function by providing high-quality state-action coverage early in training. Online data then corrects the critic where the robot’s own distribution differs from the demonstrator’s. If the policy discovers a better action than the one in the demonstration, the actor objective can prefer it. That is precisely why RLPD can move beyond pure imitation: it uses demonstrations as off-policy experience, not as immutable targets.

The target network update completes the standard off-policy actor-critic loop:

$$\bar Q_\phi \leftarrow \tau Q_\phi + (1-\tau)\bar Q_\phi.$$

This slow-moving target $\bar Q_\phi$ stabilizes bootstrapping. Without it, the critic would chase a target that changes too rapidly as $Q_\phi$ itself is updated. In real robotic manipulation, where rewards are sparse, perception is noisy, and datasets are small, such stabilization is not a minor implementation detail; it is part of what makes repeated updates from limited physical experience feasible.

There are also several assumptions hiding inside this compact update. The replayed transitions must be compatible with the current Markov state representation, the reward function must be consistently computable for both old demonstrations and new online data, and the action space must match across demonstrator, human intervention, and robot policy. If demonstrations use a different controller interface or coordinate frame than online actions, the “same replay format” assumption breaks. Similarly, if the demonstration buffer dominates too much, the policy may remain conservative; if online data dominates too early, the learner may forget rare but valuable successful behavior.

A useful way to think about the mixture ratio $\lambda$ is as a knob controlling the bias-coverage tradeoff:

• More demonstration replay gives stable access to competent behavior and successful outcomes.
• More online replay adapts the value function to the states the current policy actually visits.
• Mixing both allows the critic to compare expert-like behavior, intervention behavior, autonomous failures, and autonomous successes under one value scale.

The visual below condenses this entire update into a single algorithmic box: sample from the demonstration buffer, sample from the RL buffer, merge the transitions, compute the same Bellman target, update the critic, update the actor, and softly update the target critic. The highlighted sampling and gradient steps are meant to emphasize the central design principle: the replay source changes, but the loss does not.

This is the algorithmic bridge to the human-in-the-loop part of HIL-SERL. Once we accept that any valid transition can be replayed through the same off-policy update, a human correction no longer needs to be framed as a supervised label. It can be logged as another transition in $\mathcal{D}_{RL}$, assigned reward through the same reward mechanism, and used by the same Bellman backup. That is the conceptual move that makes interventions more than emergency controls: they become reusable reinforcement learning data.

Once we have an off-policy actor–critic update that can mix demonstrations and online experience, the next question is subtle but central: what exactly should we do with human corrections? A tempting answer is to treat the human’s action as a label and train the policy to imitate it. HIL-SERL instead makes a different choice: a human correction is treated as an executed action in the environment, producing a real next state and reward, and therefore becomes ordinary off-policy reinforcement learning data.

At every control step, the robot policy still proposes an action,

$$\mathbf{a}_{RL}\sim\pi_\theta(\cdot\mid\mathbf{s}_t).$$

This matters because the system is not switching into a separate “teleoperation dataset collection” mode. The learned policy remains in the loop, generating behavior and exposing its own weaknesses. The human only intervenes when the current behavior is likely to fail, unsafe, or inefficient. In other words, interventions are concentrated near the policy’s decision boundary: places where autonomous control is not yet reliable.

The executed action is then defined by a simple gate:

$$\mathbf{a}^*= \begin{cases} \mathbf{a}_{itv}, & I_t=1,\\ \mathbf{a}_{RL}, & I_t=0. \end{cases}$$

Here $I_t$ indicates whether the human is currently intervening. If $I_t=0$, the robot executes its own policy action. If $I_t=1$, the human action $\mathbf{a}_{itv}$ overrides the policy. In HIL-SERL, interventions are also temporally bounded: if an intervention starts at $t_i$, it lasts for at most $N$ steps, so $I_t=1$ only for $t_i\le t<t_i+N$. This prevents the human from silently taking over the whole episode and encourages the learned policy to resume control quickly.

The key design decision is what happens to the transition after the action is executed. If the human intervened, HIL-SERL stores the resulting transition in both the demonstration replay and the online replay:

$$(\mathbf{s}_t,\mathbf{a}^*,r(\mathbf{s}_t,\mathbf{a}^*),\mathbf{s}_{t+1}) \in \mathcal{D}_{demo}\cap\mathcal{D}_{RL} \quad\text{if }I_t=1.$$

Autonomous transitions before and after the correction are stored only in $\mathcal{D}_{RL}$. So the replay buffers distinguish where the data came from, but the critic still sees a transition: state, action, reward, next state. The human action is not treated as a privileged target that must be copied. It is treated as an action whose value should be inferred from its consequences.

This is the important distinction from methods such as HG-DAgger-style human-guided dataset aggregation. In a DAgger-like formulation, intervention data often becomes supervised learning data: “in state $\mathbf{s}_t$, the correct action was $\mathbf{a}_{itv}$.” That can be useful, but it also introduces a strong assumption: the human action is locally optimal, and the policy should reproduce it directly. HIL-SERL avoids adding a supervised action-label loss. Instead, it asks the critic to evaluate whether the intervened action led to better long-term outcomes.

That evaluation happens through the same Bellman backup used for ordinary off-policy RL:

$$\mathcal{L}_Q(\phi) = \mathbb{E}_{\mathcal{D}_{batch}} \left[ \left( Q_\phi(\mathbf{s},\mathbf{a}) - \left( r(\mathbf{s},\mathbf{a}) + \gamma \mathbb{E}_{\mathbf{a}'\sim\pi_\theta} \left[ \bar Q_\phi(\mathbf{s}',\mathbf{a}') \right] \right) \right)^2 \right].$$

This equation is doing more than assigning credit to the immediate human action. It propagates value backward from future outcomes. If a correction prevents a grasp from slipping and leads, several seconds later, to task success, the critic can assign higher value to the corrected transition even when the immediate sparse reward is zero. Conversely, if a human action looks reasonable locally but does not improve the long-horizon return, the RL objective is not forced to imitate it.

This gives HIL-SERL a useful asymmetry. Human corrections are high-value exploration events because they push the robot out of failure modes and into recoverable parts of the state space. But they are not treated as unquestionable ground truth. Their usefulness is mediated by reward, dynamics, and bootstrapped value estimates. In dexterous manipulation, where there may be many ways to recover an object, regrasp, or reposition a hand, this is a much better fit than rigidly cloning every human command.

There are also practical failure modes this design helps avoid:

• Over-imitation of interventions: if the policy simply clones human actions, it may learn intervention-specific behavior rather than autonomous recovery behavior.
• Distribution mismatch: human corrections occur in states induced by a partially trained policy, not necessarily in clean expert demonstrations.
• Sparse reward credit assignment: Bellman backups can connect a short correction to delayed task success.
• Noisy human input: an imperfect correction can still be evaluated by its realized consequences instead of being accepted as a perfect label.

The visual below compactly summarizes this data flow. The robot timeline alternates between autonomous regions, where $\mathbf{a}_{RL}$ is executed and stored in online replay, and intervention regions, where $\mathbf{a}_{itv}$ becomes the executed action $\mathbf{a}^*$. The intervention transitions are deliberately sent to both replay sources, making them available both as demonstration-like high-quality data and as part of the online distribution encountered during training.

The right side emphasizes the conceptual punchline: off-policy evaluation, not imitation. The case equation defines which action was actually executed, while the Bellman critic loss explains how that action is judged. The crossed-out supervised action-label loss is not just a cosmetic detail; it marks the methodological boundary between “copy the human” and “learn from what happened after the human acted.”

With interventions now understood as experience rather than labels, the full HIL-SERL loop becomes conceptually simple: the robot is always learning from the consequences of executed actions, regardless of whether those actions came from the policy or from a human takeover. The human is not training a behavior cloning model at every correction. Instead, the human temporarily changes the behavior policy that generates data, and the off-policy learner later decides how that data should shape the value function and policy.

The training loop has three coupled components. First, the system needs a way to assign rewards in the real world without hand-coding every success condition. HIL-SERL therefore trains a visual reward classifier $C_\omega(\mathbf{o}^{img})$ from labeled images. This classifier turns camera observations into sparse task rewards:

$$r(\mathbf{s}_t, \mathbf{a}^*) \text{ from } C_\omega(\mathbf{o}^{img}).$$

This is a pragmatic compromise. The reward is still sparse and imperfect, but it is much easier to obtain than a dense manually engineered reward for dexterous manipulation. The important assumption is that success is visually recognizable: for example, whether an object has been inserted, grasped, placed, opened, or otherwise brought into the desired configuration. If the classifier is poorly calibrated, the RL loop can optimize the wrong signal; if it is too conservative, learning may be unnecessarily slow.

Second, HIL-SERL seeds replay with a small number of teleoperated demonstrations, typically on the order of 20--30 episodes. These demonstrations are not treated as a final supervised dataset. They are used as high-value off-policy trajectories inside the replay mixture, so the critic can see successful outcomes early and avoid the cold-start problem of sparse-reward exploration. This is where the method inherits the spirit of RLPD: demonstration data and online robot data are sampled together for actor-critic updates, allowing the policy to improve beyond the demonstrator while still being anchored by useful successful experience.

The online phase is the key closed loop. At each time step, the robot observes a state containing both visual and proprioceptive information, such as

$$\mathbf{s}_t = (\mathbf{o}^{img}, \mathbf{p}),$$

then samples an action from the current stochastic policy:

$$\mathbf{a}_{RL} \sim \pi_\theta(\cdot \mid \mathbf{s}_t).$$

A human supervisor watches execution and sets an intervention indicator $I_t$. If $I_t = 0$, the robot executes its own action. If $I_t = 1$, the human provides an intervention action $\mathbf{a}_{itv}$. The actually executed command is the arbitration rule

$$\mathbf{a}^* = I_t\mathbf{a}_{itv} + (1-I_t)\mathbf{a}_{RL}.$$

This small equation captures a major design choice: HIL-SERL does not pause learning when the human intervenes. The world still transitions from $\mathbf{s}_t$ to $\mathbf{s}_{t+1}$, the reward classifier still assigns a reward, and the resulting transition is still valid off-policy RL data. The replay buffer therefore records what actually happened, not what the policy would have done without help.

There are two buffers with different roles. The online replay buffer $\mathcal{D}_{RL}$ stores all executed transitions, including both autonomous and intervened steps. The demonstration buffer $\mathcal{D}_{demo}$ contains the initial teleoperated demonstrations and may also receive intervention segments. This matters because interventions are often locally informative: they occur near states where the policy is brittle, unsafe, stuck, or about to fail. Adding those segments to the demonstration side of the mixture increases their training influence without converting the algorithm into pure imitation learning.

Periodically, the learner runs an RLPD-style update using mixed minibatches from $\mathcal{D}_{demo}$ and $\mathcal{D}_{RL}$. The critic $Q_\phi$ is trained off-policy using Bellman targets, the actor $\pi_\theta$ is updated to choose actions with high predicted value, and the target critic $\bar Q_\phi$ stabilizes bootstrapping. The crucial point is that human corrections affect learning through the value function: they create trajectories with better outcomes, and the critic propagates those outcomes backward through state-action space. The policy is then optimized against that critic.

A subtle failure mode appears when interventions are too long or too successful too early. Suppose the robot begins in a poor state, the human takes over for a long segment, and the episode succeeds. If the replay data do not make clear which actions were responsible for recovery, bootstrapping can overestimate the value of early autonomous actions that merely preceded human rescue. In other words, the learner may conclude, “my behavior led to success,” when success was actually caused by extensive human control. This is why intervention design, replay weighting, and careful monitoring of intervention frequency matter in practice.

The stopping condition is also behavioral rather than purely algorithmic. HIL-SERL is working when interventions become less frequent, task success stabilizes, and the learned policy begins to handle perturbations that previously required human correction. The loop is therefore not just “collect data, update model.” It is a real-time training protocol in which policy competence changes the distribution of future human effort.

The visual below compresses this into a single pseudocode-style training loop: initialize the reward classifier and demonstrations, run the robot under policy-human action arbitration, store every executed transition off-policy, and repeatedly call the RLPD update. The highlighted action arbitration line is the hinge of the method, because it is where human control and autonomous policy execution become one stream of experience.

The warning at the bottom is equally important. HIL-SERL’s strength is that humans can rescue the robot and keep learning productive, but those rescues must still be interpreted carefully by the critic. The algorithm succeeds when interventions provide targeted corrective data—not when they hide the policy’s weaknesses by always carrying the episode to success.

The training loop we just described can make HIL-SERL look deceptively simple: collect demonstrations, allow interventions, add everything to replay, and run an off-policy actor-critic update. But in real robotic manipulation, the algorithm is only half the story. The other half is the MDP interface: what the policy observes, how actions are executed, how rewards are computed, and how safety is maintained while the robot is still incompetent.

This matters because vision-based dexterous manipulation is not merely “RL with pixels.” A raw camera stream contains many nuisance variables—background, lighting, irrelevant geometry, robot self-occlusion, object pose ambiguity—and the reward signal is often sparse. If we expose the learner to the wrong state representation or an unsafe action interface, no amount of clever replay mixing will reliably produce sample-efficient learning. HIL-SERL’s practical success comes from designing the environment boundary so that sparse-reward off-policy RL becomes possible, rather than trying to solve every difficulty inside the neural network.

The perception stack is deliberately constrained. Instead of feeding arbitrary full-resolution images to the policy, HIL-SERL crops task-relevant regions from wrist and side cameras, resizes them to $128 \times 128$, and encodes them with an ImageNet-pretrained visual backbone $F_\psi(\mathbf{o}^{img})$. This is a strong inductive bias: the policy does not need to relearn generic edge, texture, and shape detectors from a few thousand robot transitions. It can spend its limited online data learning which visual features matter for control.

The visual embedding is then fused with proprioception $\mathbf{p}$, giving the actor and critic an observation $\mathbf{s}$ that combines appearance and robot state. Conceptually, the actor and critic receive

$$\mathbf{s} = \big(F_\psi(\mathbf{o}^{img}), \mathbf{p}\big),$$

and learn the usual policy and value functions,

$$\pi_\theta(\mathbf{a}\mid \mathbf{s}), \qquad Q_\phi(\mathbf{s}, \mathbf{a}).$$

This fusion is important because manipulation is neither purely visual nor purely proprioceptive. Vision tells the robot where the object, hand, and task-relevant geometry appear to be; proprioception tells it where the arm and gripper currently are, how they are moving, and how close they are to commanded references.

A subtle but important design choice is relative proprioception. HIL-SERL randomizes the robot’s initial end-effector pose and then expresses later pose features relative to that episode-start frame. This prevents the policy from overfitting to absolute coordinates that may be arbitrary or task-irrelevant. The learner is encouraged to represent behavior like “move forward relative to the starting approach pose” or “rotate relative to the initial grasp frame,” rather than memorizing a fixed global workspace location. For manipulation tasks where the setup changes slightly across trials, this can be the difference between a brittle policy and one that generalizes.

The reward design is also intentionally simple. Rather than hand-shaping dense rewards for distance, alignment, contact, force, or object progress, HIL-SERL uses a sparse visual classifier reward:

$$r(\mathbf{s}, \mathbf{a}) = C_\omega(\mathbf{o}^{img}).$$

Here $C_\omega$ predicts from the image whether the task is in a successful state. This is a pragmatic compromise. Dense hand-shaped rewards can make exploration easier, but they often encode the designer’s mistaken assumptions about the task. In dexterous manipulation, reward shaping can accidentally prefer visually plausible but physically wrong behaviors, such as touching the object without controlling it, moving toward a target while losing the grasp, or satisfying a geometric proxy without achieving the true task.

So the principle is not “make the reward informative at all costs.” Instead, HIL-SERL keeps the reward close to the real task objective and uses other mechanisms to compensate for sparsity:

• Demonstrations seed replay with successful or near-successful behavior.
• Human interventions prevent catastrophic drift and add corrective off-policy data.
• Off-policy RL propagates sparse success information backward through replay.
• Safe low-level control makes exploration physically tolerable on real hardware.

The action interface is therefore just as important as the observation and reward interface. Contact-rich manipulation benefits from impedance control and reference limiting, because the robot must be compliant enough to survive imperfect actions but precise enough to exploit contact. In more dynamic settings, the controller may include feedforward wrenches expressed in the end-effector frame. These choices shape the effective action space $\mathcal{A}$: the learned policy does not directly command arbitrary unsafe torques, but rather produces commands that pass through a stabilizing control layer.

The visual summary below consolidates these system choices as a pipeline. The key idea is that HIL-SERL is not simply “SAC plus demonstrations plus interventions.” It is an engineered loop in which perception, proprioception, reward classification, actor-critic learning, and safe execution are all chosen to support one another.

Read the pipeline from left to right: task-focused camera observations and proprioception become a compact observation $\mathbf{s}$; the actor and critic learn from that representation; the sparse classifier supplies the reward; and the low-level controller turns learned actions into safe robot behavior. The bottom callouts emphasize the larger design principle: sample efficiency comes from the interaction between relative state encoding, classifier-based sparse rewards, and human-provided off-policy experience, not from any single component in isolation.

With the main ingredients of the real-robot loop in place—visual representations, reward detection, relative actions, low-level stabilization—we can now address a small but important mismatch in the action space. The robot’s end-effector motion is naturally continuous: small Cartesian displacements and rotations are well suited to an actor-critic policy such as the RLPD-style continuous controller. But the gripper is different. In many manipulation setups, the gripper command is not a smooth real-valued control variable in the same sense; it is closer to a small menu of discrete decisions: open, close, or sometimes stay.

This matters because treating the entire robot action as one homogeneous continuous vector can blur two very different learning problems. End-effector control asks, “Which infinitesimal motion should I take next?” Gripper control often asks, “Should I commit to a contact mode change now?” That second question is more combinatorial and timing-sensitive. Closing one timestep too early may bump the object away; closing one timestep too late may miss the grasp. Opening too soon may drop the object before placement. So although the gripper has fewer possible actions, its value can be highly discontinuous in state.

HIL-SERL handles this by splitting the hybrid action into two coupled decision processes. The continuous component remains governed by the RLPD actor-critic update:

$$\mathbf{a}_{eef} \in \mathcal{A}_1,$$

while the gripper command is modeled as a discrete action:

$$\mathbf{a}_{gripper} \in \mathcal{A}_2.$$

The key modeling move is that these are not two separate environments. They share the same state distribution, transition dynamics, reward, and discount factor. They are two views of the same physical task:

$$\{\mathcal{S},\mathcal{A}_1,\rho,\mathcal{P},r,\gamma\} \quad\text{for }\mathbf{a}_{eef}, \qquad \{\mathcal{S},\mathcal{A}_2,\rho,\mathcal{P},r,\gamma\} \quad\text{for }\mathbf{a}_{gripper}.$$

This formulation is subtle. The transition $\mathcal{P}$ still depends on the joint executed robot action, including both motion and gripper state. But for learning, HIL-SERL assigns the continuous part to a continuous actor and the discrete part to a separate gripper critic. In other words, the system factorizes the policy implementation while still learning from the same off-policy replay stream.

For the discrete gripper, HIL-SERL uses a DQN-style value function $Q_{\theta_g}(\mathbf{s},\mathbf{a}_{gripper})$. Since $\mathcal{A}_2$ is small, we do not need a policy network that samples a continuous action. We can simply evaluate all candidate gripper commands and choose the one with the largest predicted value. The training objective uses a target network $Q_{\theta_g'}$, giving the Bellman-style loss

$$\mathcal{L}_g(\theta_g) = \mathbb{E}_{\mathbf{s},\mathbf{a},\mathbf{s}'} \left[ \left( r(\mathbf{s},\mathbf{a}) + \gamma Q_{\theta_g'} \left( \mathbf{s}', \arg\max_{\mathbf{a}'\in\mathcal{A}_2} Q_{\theta_g}(\mathbf{s}',\mathbf{a}') \right) - Q_{\theta_g}(\mathbf{s},\mathbf{a}_{gripper}) \right)^2 \right].$$

This is essentially a double-DQN-style target: the online gripper critic $Q_{\theta_g}$ selects the maximizing next gripper action, while the target critic $Q_{\theta_g'}$ evaluates it. That separation helps reduce overestimation bias, which can be especially harmful in sparse-reward manipulation. If the critic becomes overconfident that “close” is valuable in too many states, the robot may prematurely grasp empty space or clamp before alignment. If it overvalues “open,” it may never commit to stable contact.

At inference time, the action selection rule is cleanly factorized:

$$\mathbf{a}_{gripper} = \arg\max_{\mathbf{a}\in\mathcal{A}_2} Q_{\theta_g}(\mathbf{s},\mathbf{a}), \qquad \mathbf{a}_{eef}\sim\pi_\theta(\cdot\mid\mathbf{s}).$$

The continuous end-effector action is sampled from the learned motion policy, while the gripper command is chosen greedily with respect to the learned discrete critic. This is a practical hybrid controller: stochastic continuous control gives flexibility in motion, while discrete value maximization gives decisive gripper timing.

An important consequence is that human interventions remain reinforcement learning data, not merely supervised labels. If a human takes over and closes the gripper at a key moment, that transition enters the replay buffer with its state, action, reward, and next state. The gripper critic learns from the eventual value of that decision through Bellman backup. It is not simply trained to imitate “the human closed here”; it learns whether closing in that kind of state improves expected task return. This distinction is central to HIL-SERL’s ability to improve beyond demonstrations and interventions.

The visual below compresses this idea into the two-lane structure of the controller. A shared observation $\mathbf{s}$—vision plus proprioception—feeds both the continuous motion policy and the discrete gripper critic. The blue lane corresponds to $\mathcal{A}_1$, where RLPD-style actor-critic learning remains appropriate for $\mathbf{a}_{eef}$. The orange lane corresponds to $\mathcal{A}_2$, where the gripper critic scores a small set of commands such as open, close, and stay.

The equations in the visual summarize the coupled-MDP interpretation, the DQN-style gripper loss, and the inference-time factorization. The main takeaway is that HIL-SERL does not force dexterous manipulation into a single monolithic action model. It preserves the continuous RL machinery where it works well, and adds a small discrete value-learning problem exactly where the robot’s action space becomes categorical.

Once we separate dexterous control into a continuous motion MDP and a smaller discrete gripper MDP, the remaining question is operational: at every real robot timestep, which part of the system chooses what, what happens when a human intervenes, and how do those choices become training data? HIL-SERL’s answer is deliberately simple. The learned motion policy $\pi_\theta$ proposes continuous end-effector motion, while a separate gripper critic $Q_{\theta_g}$ chooses among a tiny set of discrete gripper commands.

This split matters because the two control problems have very different geometry. End-effector motion is naturally continuous: the robot must adjust pose, velocity, or delta commands smoothly in response to visual observations. Gripper control, however, is often sparse and mode-like: open, close, or stay. Treating those gripper decisions as continuous outputs can make learning unnecessarily brittle, especially in bimanual manipulation where opening or closing at the wrong moment can destroy the trial.

So HIL-SERL executes a hybrid action of the form

$$\mathbf{a}_t = (\mathbf{a}_{eef,t}, \mathbf{a}_{gripper,t}),$$

where the motion component is sampled from the actor,

$$\mathbf{a}_{eef,t} \sim \pi_\theta(\cdot \mid \mathbf{s}_t),$$

and the gripper component is selected greedily from the discrete critic,

$$\mathbf{a}_{gripper,t} = \arg\max_{\mathbf{a}\in\mathcal{A}_2} Q_{\theta_g}(\mathbf{s}_t,\mathbf{a}).$$

Here $\mathcal{A}_2$ is small. With one gripper, it contains three options: open, close, and stay. With two grippers, each gripper independently has three options, producing $3^2=9$ joint gripper actions. That is small enough that exact maximization is trivial; there is no need to sample, relax, or approximate the discrete argmax.

A subtle but important point is that the gripper critic is not trained as a supervised classifier of “the correct gripper action.” Human interventions do not simply become labels saying “do this gripper command in this state.” Instead, they become off-policy reinforcement learning transitions. The replay buffer records what state the system was in, what action was actually executed, what reward followed, and what next state resulted. This preserves the RL semantics: an intervention is valuable not because it is expert-labeled in isolation, but because it changes the future trajectory and therefore the return.

This is especially important for gripper timing. In manipulation, the best time to close a gripper may be defined by delayed consequences: a grasp that looks reasonable immediately may fail one second later, while a seemingly conservative “stay open” action may enable a better approach. A supervised loss would compress this into local imitation. The gripper critic instead asks: if I choose this discrete gripper action in this state, what return should I expect under the future policy?

The update is therefore a discrete-action off-policy Bellman update, analogous to double Q-learning. The online gripper critic chooses the next greedy action, while the target critic evaluates it:

$$\mathcal{L}_g(\theta_g) = \mathbb{E}_{\mathbf{s},\mathbf{a},\mathbf{s}'} \left[ \left( r + \gamma Q_{\theta_g'} \left( \mathbf{s}', \arg\max_{\mathbf{a}'\in\mathcal{A}_2} Q_{\theta_g}(\mathbf{s}',\mathbf{a}') \right) - Q_{\theta_g}(\mathbf{s},\mathbf{a}_{gripper}) \right)^2 \right].$$

The target network is updated slowly,

$$\theta_g' \leftarrow \tau \theta_g + (1-\tau)\theta_g',$$

which stabilizes learning by preventing the target from moving as quickly as the critic being optimized. This is the same basic reason target networks appear throughout deep off-policy RL: without them, the system can chase its own rapidly changing predictions, especially when training from a nonstationary mixture of demonstrations, interventions, and autonomous rollouts.

The intervention logic then sits around the action-selection procedure. The policy may propose $(\mathbf{a}_{eef}, \mathbf{a}_{gripper})$, but if the human intervention flag $I_t=1$, the robot executes the intervention action $\mathbf{a}_{itv}$ instead. Crucially, the resulting transition is still stored in replay according to the intervention-aware rules. Thus both the continuous actor-critic and the discrete gripper critic learn from the same stream of real interaction data, including states where the autonomous policy acted and states where the human took over.

This design gives HIL-SERL a practical compromise:

• Continuous control remains expressive enough for vision-based reaching, alignment, insertion, and contact-rich adjustment.
• Discrete gripper control avoids forcing sparse open/close decisions into a continuous action head.
• Human interventions are treated as RL experience, not merely imitation labels.
• Shared replay lets demonstrations, corrections, and autonomous experience all improve the same value estimates.

The visual below condenses this loop into an algorithmic view: first sample continuous end-effector motion from $\pi_\theta$, then choose the gripper command by maximizing $Q_{\theta_g}$, then allow the human override to replace the proposed action when needed. The highlighted lines emphasize the three operational ingredients that make the method work on hardware: motion sampling, discrete gripper argmax, and intervention-aware execution.

The compact gripper-cardinality note is also more than bookkeeping. It is the reason the separate gripper critic is feasible: exact maximization over $\mathcal{A}_2$ is cheap, even for two grippers. That lets HIL-SERL keep the dexterity of continuous motion while giving sparse gripper decisions their own value-based learning mechanism.

With the action-selection machinery in place—continuous end-effector control paired with a learned gripper decision—we can now ask the harder empirical question: does this actually work as a real robotic learning system, not just as an algorithmic recipe? For HIL-SERL, evaluation is especially important because the central claim is not merely higher asymptotic reward. The claim is that a robot can learn difficult, contact-rich, vision-based manipulation skills with limited real-world data, while staying safe and recoverable through human interventions.

That changes what “good performance” means. In simulation-heavy RL papers, we often care about sample efficiency in terms of environment steps and final return. In real-world dexterous manipulation, the evaluation protocol must also capture practical deployment constraints: how often the robot succeeds, how quickly it completes the task, how much wall-clock training time is required, and how much human help is still needed. A policy that eventually succeeds after many retries, frequent corrections, or slow cautious motions may be less useful than a slightly less elegant policy that performs reliably and briskly on hardware.

The experiments therefore evaluate HIL-SERL in the same kind of MDP we have been building up throughout the lecture. The state is not a clean simulator state with perfect object poses. Instead,

$$\mathbf{s}_t \in \mathcal{S} \quad \text{contains} \quad \mathbf{o}^{img} \ \text{and} \ \mathbf{p},$$

where $\mathbf{o}^{img}$ denotes visual observations from wrist and side cameras, and $\mathbf{p}$ includes proprioceptive and low-level robot information such as pose, twist, force/torque, and gripper status. This is crucial: the policy is being tested under partial observability, lighting variation, contact uncertainty, and real actuator dynamics. These are exactly the conditions under which pure behavior cloning often becomes brittle and RL from scratch often becomes prohibitively expensive.

The action space also reflects the practical decomposition used by the system. HIL-SERL separates end-effector motion from gripper decisions:

$$\mathbf{a}_{eef}\in\mathcal{A}_1, \qquad \mathbf{a}_{gripper}\in\mathcal{A}_2.$$

Here $\mathbf{a}_{eef}$ may be a 6D twist command or a feedforward wrench, depending on the task and controller interface, while $\mathbf{a}_{gripper}$ is optionally handled as a discrete gripper action. This matters experimentally because many manipulation failures are not just “bad trajectories”; they are failures of timing, contact mode, grasp closure, or recovery. A policy must decide when to move, how to comply with contact, and when to actuate the gripper.

The task suite is deliberately broad. It includes assembly-style manipulation such as RAM, SSD, and USB insertion; cable clipping; IKEA shelf assembly; car dashboard assembly; timing belt assembly; object handover; Jenga whipping; and object flipping. These are not interchangeable benchmarks. They stress different forms of dexterity:

• Precision insertion tests alignment, contact-rich correction, and force-sensitive behavior.
• Long-horizon assembly tests compounding errors and robustness over many substeps.
• Dynamic manipulation tests timing and predictive control rather than only quasi-static servoing.
• Human-facing tasks, such as handover, test reactivity and safe interaction.

This diversity is important because human-in-the-loop RL could otherwise be overfit to a narrow class of resettable insertion tasks. A convincing evaluation needs to show that the same recipe—demonstrations, interventions, off-policy updates, visual representations, and low-level controllers—extends across qualitatively different manipulation regimes.

The metrics are chosen to expose both performance and practicality. Success rate measures whether the learned behavior completes the intended task. Cycle time measures whether the policy is merely successful or also efficient. Training time captures the real-world cost of collecting interaction data and updating the model. Intervention rate measures how much human correction remains necessary after learning. That last metric is particularly revealing: a policy can have a decent success rate while still requiring frequent human rescue, which would undermine the goal of autonomous deployment.

The baselines and ablations are equally important. Comparisons to imitation-style methods such as HG-DAgger/BC and Diffusion Policy ask whether supervised learning from human behavior is enough. Comparisons to RL from scratch ask whether human data is genuinely needed for sample efficiency and safety. Methods such as Residual RL, DAPG, and IBRL probe nearby ways of combining demonstrations, interventions, and reinforcement learning. Finally, ablations of HIL-SERL without demonstrations or without interventions test whether the full loop is doing something more than simply initializing from demos or occasionally correcting mistakes.

A subtle but important assumption behind this evaluation is that real-world trials are expensive and noisy. Most results use around 100 trials per task, while very long-horizon IKEA assembly is evaluated with fewer trials because each rollout is substantially more costly. This means we should interpret results as robotic systems evidence rather than infinite-sample statistical benchmarking. The value of the suite is that it triangulates performance across many physical tasks, metrics, and ablations, making it harder for a single artifact—say, an unusually easy reward classifier or a forgiving object geometry—to explain the outcome.

The visual below condenses this protocol into a compact checklist: what tasks are used, what the robot observes, what it controls, which metrics are reported, how many trials are run, and which competing methods or ablations define the comparison set. Read it as the experimental contract for the next section: if HIL-SERL claims improved success, speed, and robustness, those claims are grounded in this real-world evaluation structure.

It also helps separate algorithmic ingredients from evaluation evidence. The previous sections explained how HIL-SERL gathers demonstrations, converts interventions into replay data, and trains off-policy actor-critic updates. This experiment suite is where those design choices are forced to prove themselves under hardware constraints: camera observations, contact-rich dynamics, discrete gripper timing, limited trials, and human effort measured directly through interventions.

With the evaluation protocol in place, the central question becomes sharper: does human-in-the-loop RL actually buy us something beyond a better way to collect demonstrations? The HIL-SERL results suggest that the answer is yes. The improvement is not merely that humans provide more data, or that the robot sees more corrective actions. The key difference is how those corrections enter the learning problem: as off-policy experience for reward-maximizing RL, rather than as supervised labels that the policy is asked to imitate directly.

Across the reported real-robot task suite, HIL-SERL reaches 100% success, while the matched HG-DAgger/behavior cloning baseline averages 49.7% success. That is a very large gap, especially because both methods have access to human guidance and are evaluated under the same task protocol. The baseline is not failing because it has no human input; it is failing because supervised correction data is still a brittle training signal for dexterous manipulation. When the robot is in a bad state, the human correction may be locally sensible, but behavior cloning has no explicit mechanism for asking whether that correction leads to task completion, faster execution, or recovery from future disturbances.

This is exactly where the off-policy RL framing matters. In HIL-SERL, an intervention transition is still an RL transition. The dataset contains states, actions, rewards, next states, and terminal outcomes, including transitions where the action came from a human rather than the learned policy. The critic can therefore evaluate those actions according to downstream return:

$$Q^\pi(s_t,a_t) \approx r_t + \gamma \, \mathbb{E}_{a_{t+1}\sim \pi(\cdot|s_{t+1})} \left[Q^\pi(s_{t+1},a_{t+1})\right].$$

The important subtlety is that the human action is not treated as a commandment to imitate everywhere. It is treated as evidence about what happened when that action was taken in that state. If the intervention helped complete the task, the return backs that up. If it merely avoided an immediate crash but led to a slow or awkward recovery, the value function can represent that too. This is a different learning signal from “copy the human action at similar observations.”

The speed result reinforces the same interpretation. HG-DAgger/BC averages 9.6 seconds per cycle, while HIL-SERL averages 5.4 seconds:

$$\frac{9.6}{5.4}\approx 1.8.$$

So the learned behavior is not just more reliable; it is roughly 1.8× faster by this cycle-time measure. That matters because dexterous manipulation is not only a binary success problem. A robot that eventually succeeds after hesitating, regrasping unnecessarily, or waiting for human rescue may look acceptable under a loose success metric, but it has not learned a strong closed-loop control strategy. HIL-SERL’s reward-driven updates can favor trajectories that complete the task efficiently, whereas imitation learning tends to reproduce the distributional quirks of the collected corrections.

The largest gaps appear on the harder manipulation problems: timing belt assembly, Jenga whipping, RAM insertion, and car dashboard assembly. These are precisely the settings where static imitation is weakest. They require contact-rich adjustment, recovery from small geometric errors, and sometimes predictive motion rather than purely reactive servoing. In such tasks, two observations can look superficially similar while requiring different actions depending on contact state, object momentum, or recent history. A supervised learner trained on interventions can average over these modes and produce indecisive behavior. An RL agent, by contrast, is pressured by task reward to discover actions that are not merely plausible, but consequential.

The learning curves tell an equally important story. As training progresses, success increases, the fraction of steps with human intervention $I_t=1$ trends toward zero, and cycle time decreases. These three trends together are stronger evidence than final success alone:

• Rising success means the robot is solving more trials.
• Falling intervention rate means it is becoming more autonomous, not merely leaning harder on the human.
• Decreasing cycle time means its solutions are becoming more direct and efficient.

This combination rules out a common misleading interpretation of human-in-the-loop systems: that the method works only because the human keeps bailing out the robot. In HIL-SERL, the interventions are increasingly absorbed into the policy through off-policy value learning, so the human becomes less necessary over time.

The ablations make the mechanism harder to dismiss. Removing both demonstrations and online interventions gives 0% success on representative tasks, which confirms that real-world sparse-reward dexterous RL from scratch is still too difficult under this sample budget. But simply adding more demonstrations is also insufficient. Demonstrations provide coverage of successful behavior, yet they do not automatically teach robust recovery from the learner’s own mistakes. Online interventions fill that gap by sampling exactly the states where the current policy is weak, and RL turns those recovery experiences into value-guided policy improvement.

Finally, the robustness videos are not just qualitative extras; they probe whether the learned policy has acquired closed-loop competence. Recovery from moving targets, forced gripper openings, poor grasps, and deformable belt perturbations indicates that the policy is not memorizing a narrow nominal trajectory. It has learned actions that remain useful under disturbances. This is the behavior we should expect if the critic has learned that certain corrective maneuvers lead to high return even when the state deviates from the demonstration manifold.

The visual below compactly organizes these empirical claims. The left side compares HIL-SERL against the matched imitation baseline on the headline metrics: success, cycle time, hard-task behavior, and ablation outcomes. The large success and speed numbers should be read together: HIL-SERL is not trading reliability for speed, but improving both.

The right side summarizes the training dynamics that explain those final numbers. Success rises, interventions decay, and cycle time falls, which is the signature of a system converting human assistance into autonomous reward-optimizing behavior. The robustness icons along the bottom then connect the quantitative story to the qualitative one: the policy is not merely completing easy trials, but recovering under the kinds of perturbations that make real manipulation hard.

The empirical results are impressive, but the deeper lesson is not simply that “more human data helps” or that “RL can fine-tune demonstrations.” The unifying mechanism is more specific: HIL-SERL makes sparse-reward robotic RL work by shaping the data distribution rather than hand-shaping the reward. Instead of designing dense heuristics for every task phase, the system arranges for the replay buffer to contain useful states, corrective actions, successful recoveries, and perceptually grounded task completions. The reward can remain relatively sparse because the agent is not forced to discover the entire behavioral scaffold from random exploration.

This distinction matters because reward shaping and data shaping fail in different ways. A shaped reward can make learning easier, but it can also encode the wrong task, create reward hacking opportunities, or overfit to one geometry of the environment. Data shaping, by contrast, biases where learning happens. Demonstrations initialize the policy near competent behavior; online rollouts expose the learner to its own mistakes; human interventions insert recovery actions exactly where the autonomous policy is unreliable. The agent still learns through off-policy RL, so the final behavior is not just behavior cloning. It is optimized under the task reward, but from a replay distribution that is far more informative than random trial-and-error.

This is why the mixed replay update is central rather than incidental:

$$\mathcal{D}_{batch} = \lambda \mathcal{D}_{demo} + (1-\lambda)\mathcal{D}_{RL}.$$

The demonstration buffer prevents a cold start, while the online RL buffer keeps the critic and actor grounded in the states actually induced by the current policy. If $\lambda$ is too large, learning can become overly conservative and imitation-like; if it is too small early on, the robot may spend most of its time in unrecoverable or irrelevant states. HIL-SERL’s practical contribution is to keep this mixture useful over time: demonstrations seed the funnel, interventions repair it, and autonomous rollouts gradually take over as the policy improves.

The Q-function analysis gives a useful lens for understanding where the learning problem is hard. In the RAM insertion task, state visitation begins as a broad cloud and gradually narrows into a funnel toward the socket. But not every point in that funnel is equally important. Some states are forgiving: many nearby actions still make progress. Others are bottlenecks, where a small action change determines whether the robot aligns, jams, recovers, or fails. One way to expose these bottlenecks is to measure the local sensitivity of the learned critic:

$$\operatorname{Var}[Q(\mathbf{s},\mathbf{a})] = \mathbb{E}_{\epsilon\sim\mathcal{U}[-c,c]} \left[ \left( Q_\phi(\mathbf{s},\mathbf{a}+\epsilon) - \mathbb{E}_{\epsilon\sim\mathcal{U}[-c,c]}Q_\phi(\mathbf{s},\mathbf{a}+\epsilon) \right)^2 \right].$$

A high value of $Q_\phi(\mathbf{s},\mathbf{a})$ says that the critic believes the state-action pair is promising. A high local variance says something different: nearby actions have meaningfully different consequences. The most interesting states are often those with both high value and high sensitivity. They are not hopeless failure states; they are critical decision states. This is exactly where human interventions are most valuable, because they add off-policy data at moments when action selection matters.

That perspective also helps reconcile the different kinds of skills learned in the experiments. Some tasks are primarily reactive. RAM insertion, dashboard manipulation, and timing belt handling require the policy to continually correct based on visual observations $\mathbf{o}^{img}$ and relative pose information $\mathbf{p}$. In these settings, robustness comes from closed-loop control: the robot observes small deviations, adjusts, and keeps the trajectory inside the success funnel. Other tasks are more predictive. Jenga whipping, for example, has a dynamic component where the decisive motion must be executed with precise timing; once the motion begins, there may be little opportunity for slow feedback correction. There, successful learning looks more like a low-variance reflex: the policy learns a reliable action pattern for the relevant pre-contact configuration.

Several engineering choices make this possible on real hardware. The reward classifier $C_\omega(\mathbf{o}^{img})$ gives a sparse but task-grounded success signal without requiring dense geometric instrumentation. The pretrained visual encoder $F_\psi(\mathbf{o}^{img})$ reduces the amount of task-specific data needed to learn visual features from scratch. Relative coordinates $\mathbf{p}$ improve transfer across object starts and robot poses. Safe low-level controllers constrain exploration so that online RL is physically tolerable. The separate gripper critic $Q_{\theta_g}(\mathbf{s},\mathbf{a}_{gripper})$ avoids forcing inherently discrete open/close/stay decisions into an awkward Gaussian continuous-action model.

Put differently, HIL-SERL works because no single component is asked to solve the whole problem. The system decomposes the burden:

• Perception supplies reusable visual structure.
• Demonstrations place the policy near useful behavior.
• Interventions convert failures into corrective RL data.
• Off-policy learning reuses all collected experience efficiently.
• Action parameterization matches the physical structure of the robot.
• Safety controllers keep exploration within a recoverable regime.

The remaining open problems are precisely the places where this recipe may not scale automatically. Longer-horizon tasks create credit assignment problems that sparse success classifiers may not resolve. Stronger generalization will require policies that transfer not only across initial states, but across object instances, backgrounds, tools, and contact dynamics. Pretrained visual backbones help, but pretrained value functions or reusable robotic priors may be needed to reduce the amount of online hardware interaction further. Finally, if many HIL-SERL-trained skills are collected across tasks, an important next step is distillation: turning isolated task policies into broader robot foundation models that retain closed-loop reliability.

The visual below condenses this final interpretation. The funnel sketch captures the idea that learning is not merely increasing success probability; it is reshaping the robot’s state visitation into a narrower, more recoverable region near the goal. The highlighted high-$Q$, high-variance states mark the bottlenecks where action choice is most consequential and where interventions or high-quality replay data are especially valuable.

The component table then ties the system pieces back to their practical roles: reward classifiers avoid brittle dense shaping, pretrained perception improves sample efficiency, mixed replay prevents cold starts, interventions stop compounding errors, relative coordinates and safe controllers support transfer and robustness, and discrete gripper reasoning fixes an otherwise subtle action-space mismatch. As a closing summary, HIL-SERL is best understood as a complete closed-loop learning system: not imitation learning alone, not RL from scratch, and not reward engineering, but a carefully structured way to make sparse-reward real-world manipulation learnable.

The difficulty we just identified—contact-rich manipulation with vision, partial observability, tight timing, and rare successes—sets up the central claim of HIL-SERL. The paper is not saying that dexterous manipulation suddenly becomes easy because of a new reinforcement learning loss. In fact, one of the most important messages is almost the opposite: the RL algorithm only becomes useful when the rest of the robotic learning system is engineered to make RL’s assumptions less unreasonable.

A useful way to frame the claim is to contrast two ways of using human data. In behavioral cloning or HG-DAgger-style imitation, human behavior is treated primarily as a target distribution: the robot should learn to act like the demonstrator, or like the corrected policy under human supervision. This can work when the task is mostly kinematic, when the expert demonstrations cover the relevant states, and when small action errors do not qualitatively change the future. But dexterous manipulation often violates all three assumptions. A slightly late grasp, a missed contact, or a poorly timed push can send the object into a state that was never demonstrated, and then pure imitation has no principled way to decide how to recover.

HIL-SERL instead uses human data to solve a different bottleneck: exploration. Demonstrations and interventions help the robot visit states near successful behavior and avoid catastrophic or unproductive regions of the state space. But the policy is not improved merely by copying the human. The actual improvement is driven by off-policy reinforcement learning using task reward. That distinction matters because the reward can, in principle, rank behaviors by outcome rather than by resemblance to a person.

Put differently, imitation answers the question: “What action would the human take here?” HIL-SERL’s RL component answers: “Which actions lead to higher long-term task success from here?” Those questions often agree, but not always. A robot may have different dynamics, latency, controller structure, gripper geometry, or visual processing than the human teleoperator. A behavior that looks human-like may be unnecessarily slow or brittle; a behavior that looks slightly strange may be more reliable for the robot. HIL-SERL’s thesis is that human guidance should shape the data distribution, while reward-backed dynamic programming should decide what is actually good for the robot.

This is why the system-level ingredients are not incidental details. Sparse rewards are hard to use unless the replay buffer contains meaningful transitions. Vision is hard to learn from scratch unless the representation is stabilized by pretrained visual features. Real-world trial-and-error is dangerous unless low-level controllers and human interventions keep the robot inside a recoverable operating regime. Online RL is slow unless actor and learner processes can collect, relabel, and train from experience efficiently. Each ingredient reduces a different practical failure mode:

• Pretrained visual features reduce the burden of learning perception entirely from sparse robot reward.
• Classifier rewards turn human-provided success criteria into scalable task feedback.
• Human interventions prevent long stretches of useless or unsafe exploration.
• Off-policy replay allows demonstrations, corrections, and autonomous rollouts to all contribute to the same learning process.
• Safe low-level control makes high-level exploration physically feasible on real hardware.
• Distributed actor–learner infrastructure improves wall-clock efficiency, which is crucial when robot time is the scarce resource.

The comparison with HG-DAgger or behavioral cloning is therefore not simply “RL beats imitation.” A more precise reading is: imitation with comparable human data can still inherit the fragility of supervised learning under distribution shift. HIL-SERL changes the role of that data. The human is no longer just a labeler whose actions define the desired policy; the human becomes part of a data-collection mechanism that keeps the replay buffer informative while reinforcement learning performs policy improvement.

This also explains why the reported outcomes are so striking. The paper reports that imitation-style baselines achieve roughly 49.7% average success and are about 1.8× slower in cycle time, while HIL-SERL reaches near-100% success after roughly 1–2.5 hours for most tasks, with the timing belt task taking longer. The timing-belt exception is important: it reminds us that the method is not magic. When the task demands particularly difficult predictive dynamics, precise timing, or rare successful contact sequences, even a well-designed RL system needs more experience.

The deeper takeaway is that sample-efficient real-world RL is less about a single algorithmic trick and more about aligning the entire pipeline with the structure of the learning problem. HIL-SERL makes RL practical by arranging for the robot to see useful states, receive usable reward, train from mixed sources of experience, and stay physically safe while improving. The reinforcement learner is the optimization engine, but the surrounding system determines whether that engine ever receives fuel of sufficient quality.

The visual below condenses this argument into a direct comparison. The key distinction is not merely the performance numbers, but the column labeled by what drives improvement: imitation methods improve by matching human behavior, while HIL-SERL improves by combining human-guided data collection with sparse-reward off-policy RL.

The bottom takeaway lines in the visual should be read as the paper’s central design philosophy. HIL-SERL is a recipe: pretrained vision, reward classification, RLPD-style off-policy learning, human interventions, safe controllers, and distributed replay are all arranged so that human data guides exploration, but reward-driven dynamic programming selects and reinforces successful robot behavior.

If the previous claim is that reinforcement learning becomes practical only when the whole robotic system is designed around it, the next step is to make precise what “the RL problem” actually is. In HIL-SERL, the authors are not treating dexterous manipulation as an abstract benchmark with clean simulator states and shaped rewards. They are solving a finite-horizon, vision-based, real-world robotic Markov decision process in which the robot must act from noisy observations, collect sparse success signals, and improve using data gathered on the physical system.

Formally, each manipulation task is modeled as an MDP,

$$\mathcal{M}=\{\mathcal{S},\mathcal{A},\rho,\mathcal{P},r,\gamma\}.$$

Here $\mathcal{S}$ is the observation/state space, $\mathcal{A}$ is the action space, $\rho$ is the initial-state distribution induced by resets, $\mathcal{P}$ is the unknown real-world transition dynamics, $r$ is the reward function, and $\gamma\in(0,1)$ is the discount factor. The word “unknown” matters: the robot does not get access to $\mathcal{P}$, and in the real world $\mathcal{P}$ includes contact dynamics, actuator delays, perception noise, object slippage, human reset variation, and all the other small effects that make manipulation difficult to model.

The observation $\mathbf{s}_t\in\mathcal{S}$ is not a compact simulator state such as object pose plus joint angles. It contains camera observations $\mathbf{o}^{img}$ together with proprioception $\mathbf{p}$, such as robot joint or end-effector information. This is one of the central reasons the setting is hard: the policy must infer task-relevant physical state from pixels, while also reacting quickly enough to contacts and object motion. A cup sliding, a card bending, or a toy slipping from the gripper may be obvious to a human observer, but to the policy it is just a change in high-dimensional image features.

The action $\mathbf{a}_t\in\mathcal{A}$ is typically a low-level command to the robot controller. In HIL-SERL-style systems, this is often decomposed into an end-effector command $\mathbf{a}_{eef}\in\mathcal{A}_1$, optionally combined with a gripper command $\mathbf{a}_{gripper}\in\mathcal{A}_2$. This design choice is subtle but important. The policy is not directly outputting motor torques for every actuator; it acts through a controller abstraction that makes exploration safer and more sample-efficient. The RL policy still learns meaningful manipulation behavior, but the low-level controller absorbs some of the burden of stable motion execution.

The learning objective is the standard discounted finite-horizon return:

$$\max_\theta\; \mathbb{E}_{\mathbf{s}_0\sim\rho,\;\mathbf{a}_t\sim\pi_\theta(\cdot\mid\mathbf{s}_t),\;\mathbf{s}_{t+1}\sim\mathcal{P}(\cdot\mid\mathbf{s}_t,\mathbf{a}_t)} \left[ \sum_{t=0}^{H}\gamma^t r(\mathbf{s}_t,\mathbf{a}_t) \right].$$

This equation says that we want policy parameters $\theta$ that produce actions leading to high cumulative reward over a horizon $H$. The expectation is over three sources of randomness: the reset distribution $\rho$, the policy’s own action sampling, and the environment transitions. In a real robot, all three are consequential. Resets may place objects slightly differently, stochastic policies explore different contacts, and the same command can produce different outcomes depending on friction, compliance, and object pose.

The reward in HIL-SERL is intentionally sparse. Instead of hand-engineering a dense reward based on distances, angles, or object poses, the system uses a learned visual success classifier:

$$r(\mathbf{s},\mathbf{a})=C_\omega(\mathbf{o}^{img})\in\{0,1\}.$$

The classifier $C_\omega$ looks at the image observation and predicts whether the task has succeeded. This has a major practical advantage: many real manipulation tasks have outcomes that are easier to recognize than to quantify continuously. It may be straightforward to label whether the object is inserted, opened, placed, or grasped, while being much harder to define a smooth reward that guides every intermediate contact-rich motion.

But sparse rewards also introduce a serious learning challenge. Most random behavior produces $0$ reward, especially in dexterous manipulation. From the perspective of a policy learning from scratch, the reward landscape may look almost empty until the robot accidentally succeeds. This is exactly why HIL-SERL cannot be understood as “just run RL on a robot.” The sparse classifier reward defines the goal cleanly, but the surrounding system—demonstrations, human interventions, off-policy replay, visual pretraining, and controller design—is what makes the sparse signal learnable.

The discount factor $\gamma$ adds another important bias: among policies that eventually succeed, the objective prefers policies that succeed sooner. If the same success occurs at time $t$ rather than $t+N$, its contribution is weighted by $\gamma^t$ instead of $\gamma^{t+N}$. Since $\gamma<1$,

$$\gamma^t>\gamma^{t+N}.$$

So the reward formulation does not merely encode “succeed eventually.” It also encodes a pressure toward efficient completion. In robotic manipulation this matters because slow, hesitant, or overly indirect behavior may be technically successful but practically undesirable. A policy that reacts quickly to object motion, recovers from small errors, and completes the task in fewer steps receives a larger return.

The visual summary that follows compresses this formulation into the core ingredients: the MDP definition, the observation and action structure, the discounted objective, the sparse classifier reward, and the timing preference induced by discounting. Read it as a map of the problem HIL-SERL is solving, not yet the algorithm used to solve it.

In particular, notice the separation between the unknown real-world dynamics and the learned policy objective. The robot samples trajectories from resets, its current policy, and physical transitions; the reward classifier then turns visual outcomes into binary reinforcement learning data. That distinction will become crucial in the next step, where we compare HIL-SERL against prior pipelines that either rely too heavily on imitation labels or struggle to make sparse-reward RL work from scratch.

With the robotic RL objective now explicit, the central difficulty becomes less about writing down the goal and more about constructing a learning pipeline that can actually make progress toward it on hardware. For dexterous manipulation, the objective is deceptively simple:

$$\max_\theta\;\mathbb{E}\left[\sum_{t=0}^{H}\gamma^t r(\mathbf{s}_t,\mathbf{a}_t)\right].$$

The policy $\pi_\theta$ should choose actions that accumulate reward over a finite horizon $H$, usually under sparse success signals: did the robot complete the task or not? But this objective hides three practical requirements. A successful method must reach success at least sometimes, improve the behavior beyond the data it was given, and recover from the states its own imperfect policy creates.

That last point is especially important. In real robot learning, the state distribution is not fixed. A policy trained on demonstrations will not only visit the clean, expert-like states in the dataset; it will also drift into awkward poses, partial grasps, collisions, occlusions, near-failures, and timing errors. These are precisely the states where robust manipulation is decided. A method that learns only from expert trajectories may perform well while it remains close to the expert distribution, but once it deviates, it often lacks the information needed to return to success.

This explains why behavior cloning and related imitation methods, including diffusion-policy-style action modeling, can be strong yet brittle. Their learning signal is supervised: match the demonstrated action $\mathbf{a}$ given an observed state $\mathbf{s}$. Formally, they optimize something closer to an action prediction loss than the return objective above. That makes them data-efficient and stable, but it also creates a ceiling: they are trained to reproduce the demonstrations, not to discover actions that are faster, more reliable, or better adapted to the current trial. If the human demonstrator takes a conservative path, pauses unnecessarily, or uses a strategy that works only under certain object poses, the cloned policy inherits those limitations.

There is also a distribution-shift failure mode. Suppose the policy makes a small early mistake, causing the object to rotate differently than in the demonstrations. The policy now observes a state $\mathbf{s}_t$ that was rare or absent in the dataset. The supervised target is either unavailable or extrapolated poorly. Even a highly expressive model can fail here, because the problem is not merely capacity; it is that the training data did not tell the learner what action leads back to high return from this off-distribution state.

Human-guided variants such as HG-DAgger address part of this problem by collecting corrective actions when the learned policy begins to fail. This improves coverage of the states induced by the current policy, which is a major advantage over static imitation. However, if those interventions are treated purely as supervised labels, the method still learns to imitate the human correction $\mathbf{a}_{itv}$ rather than evaluate whether that correction leads to better long-term outcomes. Human corrections can be noisy, delayed, inconsistent across operators, or locally reasonable but globally suboptimal. Supervised learning has no direct mechanism to ask: did this intervention actually increase return?

At the opposite extreme, reinforcement learning from scratch optimizes the right objective. In principle, it directly searches for actions that maximize $r(\mathbf{s}, \mathbf{a})$ over time. But sparse-reward robotic manipulation is a brutal exploration problem. The useful reward may appear only after a long sequence of precise contacts, object motions, and end-effector configurations. In a large continuous space $\mathcal{S}\times\mathcal{A}$, random exploration almost never discovers success, especially for long horizons $H$. The method is optimizing the right quantity, but it may receive too little informative signal to begin learning.

This motivates demonstration-bootstrapped off-policy RL methods such as SERL or RLPD-style training with a fixed demonstration buffer $\mathcal{D}_{demo}$. These approaches are more aligned with the return objective: demonstrations provide initial successful transitions, and off-policy RL can in principle improve beyond them by estimating values and optimizing actions. This is already a better compromise than pure imitation or pure scratch RL. But if the replay buffer contains only demonstrations plus autonomous rollouts, it may still miss the most valuable data: corrective transitions from the specific failure states produced by the current policy.

The core gap is therefore not simply “more data.” It is the absence of the right kind of data paired with the right kind of learning signal. Prior pipelines tend to lack at least one of the following:

• Reward optimization: imitation matches actions but does not directly optimize return.
• Exploration support: scratch RL optimizes return but rarely finds sparse successes.
• Corrective data at policy-induced states: demo-bootstrapped RL helps early learning but may not observe enough recoveries from the learner’s own mistakes.

This is exactly the opening that HIL-SERL exploits. Human interventions are not merely labels for supervised imitation; they become off-policy RL experience. The human helps the robot visit meaningful recovery trajectories, while the RL update still evaluates transitions through reward and bootstrapped value estimates. In other words, interventions supply exploration and corrective coverage, but the algorithm remains anchored to the objective

$$\mathbb{E}\left[\sum_{t=0}^{H}\gamma^t r(\mathbf{s}_t,\mathbf{a}_t)\right],$$

rather than collapsing into action matching.

The visual below condenses this comparison into a compact failure-mode table. The important pattern is that each prior pipeline solves one part of the problem while leaving another exposed: imitation has demonstrations but no direct reward optimization; HG-DAgger has corrective coverage but still treats corrections as labels; scratch RL has the right objective but insufficient exploration; demo-seeded RL has reward optimization but not necessarily intervention data from the current policy’s hardest states.

The bottom-line gap is the design target for the next section: demonstrations help initialize learning, but online human interventions target the current policy’s mistakes. HIL-SERL’s architecture is built around turning that insight into a practical distributed robot-learning system.

The failure modes we just discussed all point to the same engineering lesson: for real robots, the learning algorithm is only half the story. A vision-based dexterous manipulation policy must improve from scarce experience, but it also has to keep the robot moving safely, accept human corrections at the right moments, assign rewards from noisy observations, and update without interrupting control. HIL-SERL’s core architectural move is therefore to separate acting from learning: the robot runs a lightweight actor process in real time, while a separate learner process performs the expensive off-policy updates asynchronously.

At each control step, the actor observes a state consisting of image observations and proprioceptive information,

$$\mathbf{s}_t = (\mathbf{o}^{img}, \mathbf{p}),$$

where $\mathbf{o}^{img}$ is the visual input and $\mathbf{p}$ includes robot state such as end-effector pose, gripper state, or other low-dimensional proprioceptive features. The policy $\pi_\theta(\mathbf{a}\mid \mathbf{s}_t)$ proposes an action $\mathbf{a}_{RL}$, typically at roughly $10$ Hz. That rate matters: it is fast enough to support reactive manipulation, but slow enough that neural-network inference, sensing, and low-level control can be coordinated reliably on real hardware.

The human operator is not used as a source of labels in the usual behavioral-cloning sense. Instead, the human is inserted into the action execution path. When no intervention occurs, the robot executes the RL policy’s action. When the operator takes control, the executed action is replaced by the intervention action:

$$\mathbf{a}^* = \begin{cases} \mathbf{a}_{itv}, & I_t = 1,\\ \mathbf{a}_{RL}, & I_t = 0. \end{cases}$$

This distinction is subtle but crucial. The system does not merely record, “the human would have done this,” and then train the policy to imitate that action with supervised learning. Instead, it treats the executed intervention as part of the environment trajectory. The transition $(\mathbf{s}_t,\mathbf{a}^*,r_t,\mathbf{s}_{t+1})$ enters replay and is later consumed by the same off-policy RL machinery as ordinary experience. In other words, human control changes the data distribution, not the objective.

This is what makes the method different from a pure imitation-learning pipeline. A teleoperated action may be locally useful, but the learning signal still comes through Bellman backups: if the intervention leads to higher future return, the critic can assign value to nearby states and actions, and the actor can gradually move toward those high-value choices. Conversely, if an intervention is imperfect or only partially helpful, the RL update is not forced to clone it exactly. The data is useful because it reaches informative states and avoids catastrophic dead ends, not because every human action is assumed to be globally optimal.

The reward pathway is also deliberately separated from the control pathway. For sparse manipulation tasks, hand-designed reward functions are often brittle: small pose errors, occlusions, and contact-rich dynamics make geometric success tests hard to specify robustly. HIL-SERL instead uses a reward classifier $C_\omega(\mathbf{o}^{img})$ to detect task success from images, producing the sparse reward used by the learner:

$$r(\mathbf{s}_t,\mathbf{a}^*) \;\text{from}\; C_\omega(\mathbf{o}^{img}).$$

This does not make the problem dense or easy—the reward may still be sparse, delayed, and classifier-dependent—but it makes the system practical across visually defined tasks. The classifier becomes a reusable success detector, while the policy and critic learn how to reach states that trigger it.

The replay design reflects the two kinds of experience HIL-SERL wants to preserve. One buffer, $\mathcal{D}_{demo}$, is initialized with a small number of teleoperated demonstrations, often on the order of $20$--$30$ trajectories. During online training, it also stores intervention segments, because those segments are disproportionately informative: they occur precisely when the current policy is uncertain, unsafe, or about to fail. A second buffer, $\mathcal{D}_{RL}$, stores online transitions generated by the current policy, including transitions immediately before and after interventions. Those adjacent transitions are important because they capture the states where autonomy and human assistance meet—the boundary where the policy most needs improvement.

The learner samples mixed batches from these replay sources, updates the critic $Q_\phi$ and policy $\pi_\theta$, and periodically sends fresh parameters back to the actor. Because this happens asynchronously, the robot does not need to pause while gradients are computed. The actor may be running slightly stale parameters, but this is acceptable in an off-policy setting: the replay buffer already contains data from many behavior policies, including old policies, current policies, demonstrations, and human interventions. The algorithmic burden is then shifted to the off-policy update rule, which we will derive next.

There are a few assumptions hiding in this architecture. First, the low-level controller must execute actions consistently enough that replayed experience is meaningful. Second, the reward classifier must be reliable enough that success is not systematically mislabeled. Third, the intervention interface must allow the human to take over quickly, because delayed overrides can still produce bad data or unsafe contact. Finally, the learner must balance demonstration-like data and online data carefully: too much reliance on demonstrations can overfit to narrow trajectories, while too little can leave the robot exploring blindly in a sparse-reward task.

The visual below condenses this into a distributed actor-learner loop. The left side corresponds to real-time robot control: observations become state, the policy proposes $\mathbf{a}_{RL}$, the human can override with $\mathbf{a}_{itv}$, and only the selected action $\mathbf{a}^*$ is executed. The reward classifier supplies the sparse success signal, and the resulting transition is written into replay.

The right side represents the slower learning process. Demonstrations, intervention segments, and ordinary online transitions are sampled together to update $Q_\phi$ and $\pi_\theta$; updated parameters then flow back to the robot. The key takeaway is that HIL-SERL does not bolt human help onto RL as a separate supervised module. It turns human help into off-policy reinforcement learning data, allowing the same actor-critic machinery to learn from both autonomy and intervention.

With the distributed architecture in place, the next question is what the learner actually optimizes while the robot is collecting experience, humans are intervening, and demonstration data is sitting in replay. HIL-SERL does not introduce a new reinforcement learning objective for interventions. Its learning rule is much closer to a standard off-policy maximum-entropy actor-critic method, with one crucial practical choice: every minibatch is deliberately mixed between prior human data and online robot experience.

The starting point is the Bellman backup. For a transition $(\mathbf{s}, \mathbf{a}, \mathbf{s}')$, the critic should estimate the immediate reward plus the discounted value of what the current policy would do next:

$$r(\mathbf{s},\mathbf{a})+\gamma\mathbb{E}_{\mathbf{a}'\sim\pi_\theta(\cdot\mid\mathbf{s}')}\left[\bar Q_\phi(\mathbf{s}',\mathbf{a}')\right].$$

Here $\bar Q_\phi$ denotes a target critic, usually an exponential moving average of the learned critic. This target network is a stabilizing device: without it, the critic would be chasing a target that changes too quickly because the same network would be used both to predict and to define its own regression label. The expectation over $\mathbf{a}'\sim\pi_\theta(\cdot\mid\mathbf{s}')$ is also important. The backup is not asking, “what did the human or robot actually do next in the dataset?” It asks, “under the current policy, what action would we sample next, and how valuable would that be?”

That distinction is what makes the update off-policy. The dataset may contain actions from several behavior sources: scripted resets, initial demonstrations, human interventions, failed autonomous attempts, and successful autonomous recoveries. But the critic target evaluates continuation under the current learned policy $\pi_\theta$. This is exactly the property HIL-SERL needs: human intervention data can be reused as reinforcement learning experience without converting it into a supervised behavioral cloning target.

The critic is trained by regressing $Q_\phi(\mathbf{s},\mathbf{a})$ toward this one-step target:

$$\mathcal{L}_Q(\phi)=\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}')\sim\mathcal{D}_{batch}}\left[\left(Q_\phi(\mathbf{s},\mathbf{a})-\left(r(\mathbf{s},\mathbf{a})+\gamma\mathbb{E}_{\mathbf{a}'\sim\pi_\theta(\cdot\mid\mathbf{s}')}\left[\bar Q_\phi(\mathbf{s}',\mathbf{a}')\right]\right)\right)^2\right].$$

Intuitively, this loss says: given the state-action pairs that actually occurred, estimate how good they were according to reward plus future policy value. Demonstration actions and intervention actions can therefore become high-value evidence if they lead to good outcomes. But they are not treated as commands that the policy must imitate forever. This matters in dexterous manipulation, where the human may intervene only in rare recovery states, may use actions that are safe but not optimal, or may teleoperate with latency and discontinuities. A supervised objective would tend to clone these artifacts; an off-policy RL objective can instead learn their long-term consequences.

The actor update then changes the policy to choose actions that the critic currently rates highly, while preserving entropy:

$$\mathcal{L}_\pi(\theta)=-\mathbb{E}_{\mathbf{s}\sim\mathcal{D}_{batch}}\left[\mathbb{E}_{\mathbf{a}\sim\pi_\theta(\cdot\mid\mathbf{s})}\left[Q_\phi(\mathbf{s},\mathbf{a})\right]+\alpha\mathcal{H}(\pi_\theta(\cdot\mid\mathbf{s}))\right].$$

The negative sign appears because we usually write training as minimization. Minimizing $\mathcal{L}_\pi$ is equivalent to maximizing critic value plus an entropy bonus. The entropy term $\alpha\mathcal{H}(\pi_\theta(\cdot\mid\mathbf{s}))$ discourages premature collapse to a deterministic policy, which is especially useful when the robot is still discovering contact-rich strategies. In manipulation, many states are ambiguous from vision alone, and small exploratory differences in pose, timing, or force can determine whether a grasp, insertion, or reorientation succeeds.

The subtlety is that both the critic and actor expectations are taken over $\mathcal{D}_{batch}$, not over only the most recent autonomous rollouts. HIL-SERL follows the RLPD-style idea of mixing demonstration and online replay inside every update:

$$\mathcal{D}_{batch}=\lambda\mathcal{D}_{demo}+(1-\lambda)\mathcal{D}_{RL},\qquad \lambda=0.5.$$

With $\lambda=0.5$, half of each training batch comes from human-provided data and half comes from robot-collected RL data. This is a simple design choice, but it has large consequences. If the learner trains mostly on online replay early on, it may drown in failures before it has enough signal to improve. If it trains mostly on demonstrations forever, it may fail to adapt to the distribution induced by its own policy. Equal mixing keeps the learner anchored to competent behavior while still forcing it to evaluate and improve on the states it actually visits.

There are also failure modes hidden in this formulation. The critic can overestimate unseen actions, especially with high-dimensional visual observations and sparse rewards. The actor may exploit those errors by choosing actions that look valuable only because the critic is poorly calibrated. Demonstration mixing reduces, but does not eliminate, this risk by repeatedly exposing the critic to meaningful state-action regions. Target networks, entropy regularization, and continued online data collection all work together to keep the Bellman updates from drifting too far away from grounded robot experience.

The key takeaway is that HIL-SERL’s human-in-the-loop mechanism is not “human labels plus RL.” It is off-policy RL with human data in the replay distribution. Human demonstrations and interventions shape the empirical distribution of transitions used for Bellman regression; the actor is still optimized through the critic, not through direct imitation. This is what lets the method combine the reliability of human guidance with the adaptability of reinforcement learning.

The visual below condenses this derivation into a top-to-bottom flow: first define the one-step Bellman target, then regress the critic toward that target, then improve the policy under the learned critic. The replay mixture at the bottom is not an implementation detail tacked on afterward; it is the sampling distribution underlying both losses.

Reading the equations this way also clarifies why RLPD is such a natural fit for HIL-SERL. The mathematical update remains a standard actor-critic update, but the batch construction ensures that every gradient step sees both human competence and robot experience. That mixture is the bridge between safe early learning and eventual autonomous improvement.

With the Bellman target and actor objective in place, the algorithmic step in HIL-SERL is surprisingly simple: do ordinary off-policy actor-critic learning, but draw each minibatch from a mixture of replay sources. The key point is that demonstrations are not treated as a separate supervised imitation loss, and online rollouts are not treated as a separate RL phase. They are all just transitions of the form $(\mathbf{s}, \mathbf{a}, r, \mathbf{s}')$ that can participate in the same Bellman backup.

Concretely, each update constructs a minibatch by sampling some fraction from a demonstration buffer and the rest from the online RL buffer:

$$\mathcal{D}_{batch} = \lambda \mathcal{D}_{demo} + (1-\lambda)\mathcal{D}_{RL}.$$

Here $\lambda$ is not a probability distribution over individual transitions in a formal dataset-mixture sense so much as a practical batching rule: if the batch size is $B$, sample approximately $\lambda B$ transitions from $\mathcal{D}_{demo}$ and $(1-\lambda)B$ transitions from $\mathcal{D}_{RL}$. The RL buffer may contain autonomous policy data, human intervention data, and later successful trajectories produced by the improving policy. Once sampled, however, the update does not remember which transition came from which source.

That uniform treatment matters. In behavior cloning, a demonstration action is interpreted as a label: “in this state, copy this action.” In RLPD-style learning, the demonstration action is instead interpreted as an action whose long-term value should be evaluated. The critic asks: if the robot took this action in this state, received reward $r(\mathbf{s}, \mathbf{a})$, and then continued according to the current policy, what return would we estimate? The target is

$$r(\mathbf{s},\mathbf{a}) + \gamma \mathbb{E}_{\mathbf{a}' \sim \pi_\theta} \left[ \bar Q_\phi(\mathbf{s}',\mathbf{a}') \right].$$

This is the same target whether $\mathbf{a}$ came from an expert demonstration, a human correction, a partially trained robot policy, or a failed autonomous attempt. The critic loss is therefore

$$\mathcal{L}_Q(\phi) = \mathbb{E}_{\mathbf{s},\mathbf{a},\mathbf{s}'} \left[ \left( Q_\phi(\mathbf{s},\mathbf{a}) - \left( r(\mathbf{s},\mathbf{a}) + \gamma \mathbb{E}_{\mathbf{a}'\sim\pi_\theta} [ \bar Q_\phi(\mathbf{s}',\mathbf{a}') ] \right) \right)^2 \right].$$

The actor is then improved using the critic’s judgment. Rather than explicitly cloning demonstration actions, the policy is optimized to choose actions that the learned $Q$-function rates highly, while retaining entropy for exploration and robustness:

$$\mathcal{L}_\pi(\theta) = - \mathbb{E}_{\mathbf{s}} \left[ \mathbb{E}_{\mathbf{a}\sim\pi_\theta} [ Q_\phi(\mathbf{s},\mathbf{a}) ] + \alpha \mathcal{H} \bigl( \pi_\theta(\cdot \mid \mathbf{s}) \bigr) \right].$$

This distinction is subtle but important. Demonstrations shape the value function by providing high-quality state-action coverage early in training. Online data then corrects the critic where the robot’s own distribution differs from the demonstrator’s. If the policy discovers a better action than the one in the demonstration, the actor objective can prefer it. That is precisely why RLPD can move beyond pure imitation: it uses demonstrations as off-policy experience, not as immutable targets.

The target network update completes the standard off-policy actor-critic loop:

$$\bar Q_\phi \leftarrow \tau Q_\phi + (1-\tau)\bar Q_\phi.$$

This slow-moving target $\bar Q_\phi$ stabilizes bootstrapping. Without it, the critic would chase a target that changes too rapidly as $Q_\phi$ itself is updated. In real robotic manipulation, where rewards are sparse, perception is noisy, and datasets are small, such stabilization is not a minor implementation detail; it is part of what makes repeated updates from limited physical experience feasible.

There are also several assumptions hiding inside this compact update. The replayed transitions must be compatible with the current Markov state representation, the reward function must be consistently computable for both old demonstrations and new online data, and the action space must match across demonstrator, human intervention, and robot policy. If demonstrations use a different controller interface or coordinate frame than online actions, the “same replay format” assumption breaks. Similarly, if the demonstration buffer dominates too much, the policy may remain conservative; if online data dominates too early, the learner may forget rare but valuable successful behavior.

A useful way to think about the mixture ratio $\lambda$ is as a knob controlling the bias-coverage tradeoff:

• More demonstration replay gives stable access to competent behavior and successful outcomes.
• More online replay adapts the value function to the states the current policy actually visits.
• Mixing both allows the critic to compare expert-like behavior, intervention behavior, autonomous failures, and autonomous successes under one value scale.

The visual below condenses this entire update into a single algorithmic box: sample from the demonstration buffer, sample from the RL buffer, merge the transitions, compute the same Bellman target, update the critic, update the actor, and softly update the target critic. The highlighted sampling and gradient steps are meant to emphasize the central design principle: the replay source changes, but the loss does not.

This is the algorithmic bridge to the human-in-the-loop part of HIL-SERL. Once we accept that any valid transition can be replayed through the same off-policy update, a human correction no longer needs to be framed as a supervised label. It can be logged as another transition in $\mathcal{D}_{RL}$, assigned reward through the same reward mechanism, and used by the same Bellman backup. That is the conceptual move that makes interventions more than emergency controls: they become reusable reinforcement learning data.

Once we have an off-policy actor–critic update that can mix demonstrations and online experience, the next question is subtle but central: what exactly should we do with human corrections? A tempting answer is to treat the human’s action as a label and train the policy to imitate it. HIL-SERL instead makes a different choice: a human correction is treated as an executed action in the environment, producing a real next state and reward, and therefore becomes ordinary off-policy reinforcement learning data.

At every control step, the robot policy still proposes an action,

$$\mathbf{a}_{RL}\sim\pi_\theta(\cdot\mid\mathbf{s}_t).$$

This matters because the system is not switching into a separate “teleoperation dataset collection” mode. The learned policy remains in the loop, generating behavior and exposing its own weaknesses. The human only intervenes when the current behavior is likely to fail, unsafe, or inefficient. In other words, interventions are concentrated near the policy’s decision boundary: places where autonomous control is not yet reliable.

The executed action is then defined by a simple gate:

$$\mathbf{a}^*= \begin{cases} \mathbf{a}_{itv}, & I_t=1,\\ \mathbf{a}_{RL}, & I_t=0. \end{cases}$$

Here $I_t$ indicates whether the human is currently intervening. If $I_t=0$, the robot executes its own policy action. If $I_t=1$, the human action $\mathbf{a}_{itv}$ overrides the policy. In HIL-SERL, interventions are also temporally bounded: if an intervention starts at $t_i$, it lasts for at most $N$ steps, so $I_t=1$ only for $t_i\le t<t_i+N$. This prevents the human from silently taking over the whole episode and encourages the learned policy to resume control quickly.

The key design decision is what happens to the transition after the action is executed. If the human intervened, HIL-SERL stores the resulting transition in both the demonstration replay and the online replay:

$$(\mathbf{s}_t,\mathbf{a}^*,r(\mathbf{s}_t,\mathbf{a}^*),\mathbf{s}_{t+1}) \in \mathcal{D}_{demo}\cap\mathcal{D}_{RL} \quad\text{if }I_t=1.$$

Autonomous transitions before and after the correction are stored only in $\mathcal{D}_{RL}$. So the replay buffers distinguish where the data came from, but the critic still sees a transition: state, action, reward, next state. The human action is not treated as a privileged target that must be copied. It is treated as an action whose value should be inferred from its consequences.

This is the important distinction from methods such as HG-DAgger-style human-guided dataset aggregation. In a DAgger-like formulation, intervention data often becomes supervised learning data: “in state $\mathbf{s}_t$, the correct action was $\mathbf{a}_{itv}$.” That can be useful, but it also introduces a strong assumption: the human action is locally optimal, and the policy should reproduce it directly. HIL-SERL avoids adding a supervised action-label loss. Instead, it asks the critic to evaluate whether the intervened action led to better long-term outcomes.

That evaluation happens through the same Bellman backup used for ordinary off-policy RL:

$$\mathcal{L}_Q(\phi) = \mathbb{E}_{\mathcal{D}_{batch}} \left[ \left( Q_\phi(\mathbf{s},\mathbf{a}) - \left( r(\mathbf{s},\mathbf{a}) + \gamma \mathbb{E}_{\mathbf{a}'\sim\pi_\theta} \left[ \bar Q_\phi(\mathbf{s}',\mathbf{a}') \right] \right) \right)^2 \right].$$

This equation is doing more than assigning credit to the immediate human action. It propagates value backward from future outcomes. If a correction prevents a grasp from slipping and leads, several seconds later, to task success, the critic can assign higher value to the corrected transition even when the immediate sparse reward is zero. Conversely, if a human action looks reasonable locally but does not improve the long-horizon return, the RL objective is not forced to imitate it.

This gives HIL-SERL a useful asymmetry. Human corrections are high-value exploration events because they push the robot out of failure modes and into recoverable parts of the state space. But they are not treated as unquestionable ground truth. Their usefulness is mediated by reward, dynamics, and bootstrapped value estimates. In dexterous manipulation, where there may be many ways to recover an object, regrasp, or reposition a hand, this is a much better fit than rigidly cloning every human command.

There are also practical failure modes this design helps avoid:

• Over-imitation of interventions: if the policy simply clones human actions, it may learn intervention-specific behavior rather than autonomous recovery behavior.
• Distribution mismatch: human corrections occur in states induced by a partially trained policy, not necessarily in clean expert demonstrations.
• Sparse reward credit assignment: Bellman backups can connect a short correction to delayed task success.
• Noisy human input: an imperfect correction can still be evaluated by its realized consequences instead of being accepted as a perfect label.

The visual below compactly summarizes this data flow. The robot timeline alternates between autonomous regions, where $\mathbf{a}_{RL}$ is executed and stored in online replay, and intervention regions, where $\mathbf{a}_{itv}$ becomes the executed action $\mathbf{a}^*$. The intervention transitions are deliberately sent to both replay sources, making them available both as demonstration-like high-quality data and as part of the online distribution encountered during training.

The right side emphasizes the conceptual punchline: off-policy evaluation, not imitation. The case equation defines which action was actually executed, while the Bellman critic loss explains how that action is judged. The crossed-out supervised action-label loss is not just a cosmetic detail; it marks the methodological boundary between “copy the human” and “learn from what happened after the human acted.”

With interventions now understood as experience rather than labels, the full HIL-SERL loop becomes conceptually simple: the robot is always learning from the consequences of executed actions, regardless of whether those actions came from the policy or from a human takeover. The human is not training a behavior cloning model at every correction. Instead, the human temporarily changes the behavior policy that generates data, and the off-policy learner later decides how that data should shape the value function and policy.

The training loop has three coupled components. First, the system needs a way to assign rewards in the real world without hand-coding every success condition. HIL-SERL therefore trains a visual reward classifier $C_\omega(\mathbf{o}^{img})$ from labeled images. This classifier turns camera observations into sparse task rewards:

$$r(\mathbf{s}_t, \mathbf{a}^*) \text{ from } C_\omega(\mathbf{o}^{img}).$$

This is a pragmatic compromise. The reward is still sparse and imperfect, but it is much easier to obtain than a dense manually engineered reward for dexterous manipulation. The important assumption is that success is visually recognizable: for example, whether an object has been inserted, grasped, placed, opened, or otherwise brought into the desired configuration. If the classifier is poorly calibrated, the RL loop can optimize the wrong signal; if it is too conservative, learning may be unnecessarily slow.

Second, HIL-SERL seeds replay with a small number of teleoperated demonstrations, typically on the order of 20--30 episodes. These demonstrations are not treated as a final supervised dataset. They are used as high-value off-policy trajectories inside the replay mixture, so the critic can see successful outcomes early and avoid the cold-start problem of sparse-reward exploration. This is where the method inherits the spirit of RLPD: demonstration data and online robot data are sampled together for actor-critic updates, allowing the policy to improve beyond the demonstrator while still being anchored by useful successful experience.

The online phase is the key closed loop. At each time step, the robot observes a state containing both visual and proprioceptive information, such as

$$\mathbf{s}_t = (\mathbf{o}^{img}, \mathbf{p}),$$

then samples an action from the current stochastic policy:

$$\mathbf{a}_{RL} \sim \pi_\theta(\cdot \mid \mathbf{s}_t).$$

A human supervisor watches execution and sets an intervention indicator $I_t$. If $I_t = 0$, the robot executes its own action. If $I_t = 1$, the human provides an intervention action $\mathbf{a}_{itv}$. The actually executed command is the arbitration rule

$$\mathbf{a}^* = I_t\mathbf{a}_{itv} + (1-I_t)\mathbf{a}_{RL}.$$

This small equation captures a major design choice: HIL-SERL does not pause learning when the human intervenes. The world still transitions from $\mathbf{s}_t$ to $\mathbf{s}_{t+1}$, the reward classifier still assigns a reward, and the resulting transition is still valid off-policy RL data. The replay buffer therefore records what actually happened, not what the policy would have done without help.

There are two buffers with different roles. The online replay buffer $\mathcal{D}_{RL}$ stores all executed transitions, including both autonomous and intervened steps. The demonstration buffer $\mathcal{D}_{demo}$ contains the initial teleoperated demonstrations and may also receive intervention segments. This matters because interventions are often locally informative: they occur near states where the policy is brittle, unsafe, stuck, or about to fail. Adding those segments to the demonstration side of the mixture increases their training influence without converting the algorithm into pure imitation learning.

Periodically, the learner runs an RLPD-style update using mixed minibatches from $\mathcal{D}_{demo}$ and $\mathcal{D}_{RL}$. The critic $Q_\phi$ is trained off-policy using Bellman targets, the actor $\pi_\theta$ is updated to choose actions with high predicted value, and the target critic $\bar Q_\phi$ stabilizes bootstrapping. The crucial point is that human corrections affect learning through the value function: they create trajectories with better outcomes, and the critic propagates those outcomes backward through state-action space. The policy is then optimized against that critic.

A subtle failure mode appears when interventions are too long or too successful too early. Suppose the robot begins in a poor state, the human takes over for a long segment, and the episode succeeds. If the replay data do not make clear which actions were responsible for recovery, bootstrapping can overestimate the value of early autonomous actions that merely preceded human rescue. In other words, the learner may conclude, “my behavior led to success,” when success was actually caused by extensive human control. This is why intervention design, replay weighting, and careful monitoring of intervention frequency matter in practice.

The stopping condition is also behavioral rather than purely algorithmic. HIL-SERL is working when interventions become less frequent, task success stabilizes, and the learned policy begins to handle perturbations that previously required human correction. The loop is therefore not just “collect data, update model.” It is a real-time training protocol in which policy competence changes the distribution of future human effort.

The visual below compresses this into a single pseudocode-style training loop: initialize the reward classifier and demonstrations, run the robot under policy-human action arbitration, store every executed transition off-policy, and repeatedly call the RLPD update. The highlighted action arbitration line is the hinge of the method, because it is where human control and autonomous policy execution become one stream of experience.

The warning at the bottom is equally important. HIL-SERL’s strength is that humans can rescue the robot and keep learning productive, but those rescues must still be interpreted carefully by the critic. The algorithm succeeds when interventions provide targeted corrective data—not when they hide the policy’s weaknesses by always carrying the episode to success.

The training loop we just described can make HIL-SERL look deceptively simple: collect demonstrations, allow interventions, add everything to replay, and run an off-policy actor-critic update. But in real robotic manipulation, the algorithm is only half the story. The other half is the MDP interface: what the policy observes, how actions are executed, how rewards are computed, and how safety is maintained while the robot is still incompetent.

This matters because vision-based dexterous manipulation is not merely “RL with pixels.” A raw camera stream contains many nuisance variables—background, lighting, irrelevant geometry, robot self-occlusion, object pose ambiguity—and the reward signal is often sparse. If we expose the learner to the wrong state representation or an unsafe action interface, no amount of clever replay mixing will reliably produce sample-efficient learning. HIL-SERL’s practical success comes from designing the environment boundary so that sparse-reward off-policy RL becomes possible, rather than trying to solve every difficulty inside the neural network.

The perception stack is deliberately constrained. Instead of feeding arbitrary full-resolution images to the policy, HIL-SERL crops task-relevant regions from wrist and side cameras, resizes them to $128 \times 128$, and encodes them with an ImageNet-pretrained visual backbone $F_\psi(\mathbf{o}^{img})$. This is a strong inductive bias: the policy does not need to relearn generic edge, texture, and shape detectors from a few thousand robot transitions. It can spend its limited online data learning which visual features matter for control.

The visual embedding is then fused with proprioception $\mathbf{p}$, giving the actor and critic an observation $\mathbf{s}$ that combines appearance and robot state. Conceptually, the actor and critic receive

$$\mathbf{s} = \big(F_\psi(\mathbf{o}^{img}), \mathbf{p}\big),$$

and learn the usual policy and value functions,

$$\pi_\theta(\mathbf{a}\mid \mathbf{s}), \qquad Q_\phi(\mathbf{s}, \mathbf{a}).$$

This fusion is important because manipulation is neither purely visual nor purely proprioceptive. Vision tells the robot where the object, hand, and task-relevant geometry appear to be; proprioception tells it where the arm and gripper currently are, how they are moving, and how close they are to commanded references.

A subtle but important design choice is relative proprioception. HIL-SERL randomizes the robot’s initial end-effector pose and then expresses later pose features relative to that episode-start frame. This prevents the policy from overfitting to absolute coordinates that may be arbitrary or task-irrelevant. The learner is encouraged to represent behavior like “move forward relative to the starting approach pose” or “rotate relative to the initial grasp frame,” rather than memorizing a fixed global workspace location. For manipulation tasks where the setup changes slightly across trials, this can be the difference between a brittle policy and one that generalizes.

The reward design is also intentionally simple. Rather than hand-shaping dense rewards for distance, alignment, contact, force, or object progress, HIL-SERL uses a sparse visual classifier reward:

$$r(\mathbf{s}, \mathbf{a}) = C_\omega(\mathbf{o}^{img}).$$

Here $C_\omega$ predicts from the image whether the task is in a successful state. This is a pragmatic compromise. Dense hand-shaped rewards can make exploration easier, but they often encode the designer’s mistaken assumptions about the task. In dexterous manipulation, reward shaping can accidentally prefer visually plausible but physically wrong behaviors, such as touching the object without controlling it, moving toward a target while losing the grasp, or satisfying a geometric proxy without achieving the true task.

So the principle is not “make the reward informative at all costs.” Instead, HIL-SERL keeps the reward close to the real task objective and uses other mechanisms to compensate for sparsity:

• Demonstrations seed replay with successful or near-successful behavior.
• Human interventions prevent catastrophic drift and add corrective off-policy data.
• Off-policy RL propagates sparse success information backward through replay.
• Safe low-level control makes exploration physically tolerable on real hardware.

The action interface is therefore just as important as the observation and reward interface. Contact-rich manipulation benefits from impedance control and reference limiting, because the robot must be compliant enough to survive imperfect actions but precise enough to exploit contact. In more dynamic settings, the controller may include feedforward wrenches expressed in the end-effector frame. These choices shape the effective action space $\mathcal{A}$: the learned policy does not directly command arbitrary unsafe torques, but rather produces commands that pass through a stabilizing control layer.

The visual summary below consolidates these system choices as a pipeline. The key idea is that HIL-SERL is not simply “SAC plus demonstrations plus interventions.” It is an engineered loop in which perception, proprioception, reward classification, actor-critic learning, and safe execution are all chosen to support one another.

Read the pipeline from left to right: task-focused camera observations and proprioception become a compact observation $\mathbf{s}$; the actor and critic learn from that representation; the sparse classifier supplies the reward; and the low-level controller turns learned actions into safe robot behavior. The bottom callouts emphasize the larger design principle: sample efficiency comes from the interaction between relative state encoding, classifier-based sparse rewards, and human-provided off-policy experience, not from any single component in isolation.

With the main ingredients of the real-robot loop in place—visual representations, reward detection, relative actions, low-level stabilization—we can now address a small but important mismatch in the action space. The robot’s end-effector motion is naturally continuous: small Cartesian displacements and rotations are well suited to an actor-critic policy such as the RLPD-style continuous controller. But the gripper is different. In many manipulation setups, the gripper command is not a smooth real-valued control variable in the same sense; it is closer to a small menu of discrete decisions: open, close, or sometimes stay.

This matters because treating the entire robot action as one homogeneous continuous vector can blur two very different learning problems. End-effector control asks, “Which infinitesimal motion should I take next?” Gripper control often asks, “Should I commit to a contact mode change now?” That second question is more combinatorial and timing-sensitive. Closing one timestep too early may bump the object away; closing one timestep too late may miss the grasp. Opening too soon may drop the object before placement. So although the gripper has fewer possible actions, its value can be highly discontinuous in state.

HIL-SERL handles this by splitting the hybrid action into two coupled decision processes. The continuous component remains governed by the RLPD actor-critic update:

$$\mathbf{a}_{eef} \in \mathcal{A}_1,$$

while the gripper command is modeled as a discrete action:

$$\mathbf{a}_{gripper} \in \mathcal{A}_2.$$

The key modeling move is that these are not two separate environments. They share the same state distribution, transition dynamics, reward, and discount factor. They are two views of the same physical task:

$$\{\mathcal{S},\mathcal{A}_1,\rho,\mathcal{P},r,\gamma\} \quad\text{for }\mathbf{a}_{eef}, \qquad \{\mathcal{S},\mathcal{A}_2,\rho,\mathcal{P},r,\gamma\} \quad\text{for }\mathbf{a}_{gripper}.$$

This formulation is subtle. The transition $\mathcal{P}$ still depends on the joint executed robot action, including both motion and gripper state. But for learning, HIL-SERL assigns the continuous part to a continuous actor and the discrete part to a separate gripper critic. In other words, the system factorizes the policy implementation while still learning from the same off-policy replay stream.

For the discrete gripper, HIL-SERL uses a DQN-style value function $Q_{\theta_g}(\mathbf{s},\mathbf{a}_{gripper})$. Since $\mathcal{A}_2$ is small, we do not need a policy network that samples a continuous action. We can simply evaluate all candidate gripper commands and choose the one with the largest predicted value. The training objective uses a target network $Q_{\theta_g'}$, giving the Bellman-style loss

$$\mathcal{L}_g(\theta_g) = \mathbb{E}_{\mathbf{s},\mathbf{a},\mathbf{s}'} \left[ \left( r(\mathbf{s},\mathbf{a}) + \gamma Q_{\theta_g'} \left( \mathbf{s}', \arg\max_{\mathbf{a}'\in\mathcal{A}_2} Q_{\theta_g}(\mathbf{s}',\mathbf{a}') \right) - Q_{\theta_g}(\mathbf{s},\mathbf{a}_{gripper}) \right)^2 \right].$$

This is essentially a double-DQN-style target: the online gripper critic $Q_{\theta_g}$ selects the maximizing next gripper action, while the target critic $Q_{\theta_g'}$ evaluates it. That separation helps reduce overestimation bias, which can be especially harmful in sparse-reward manipulation. If the critic becomes overconfident that “close” is valuable in too many states, the robot may prematurely grasp empty space or clamp before alignment. If it overvalues “open,” it may never commit to stable contact.

At inference time, the action selection rule is cleanly factorized:

$$\mathbf{a}_{gripper} = \arg\max_{\mathbf{a}\in\mathcal{A}_2} Q_{\theta_g}(\mathbf{s},\mathbf{a}), \qquad \mathbf{a}_{eef}\sim\pi_\theta(\cdot\mid\mathbf{s}).$$

The continuous end-effector action is sampled from the learned motion policy, while the gripper command is chosen greedily with respect to the learned discrete critic. This is a practical hybrid controller: stochastic continuous control gives flexibility in motion, while discrete value maximization gives decisive gripper timing.

An important consequence is that human interventions remain reinforcement learning data, not merely supervised labels. If a human takes over and closes the gripper at a key moment, that transition enters the replay buffer with its state, action, reward, and next state. The gripper critic learns from the eventual value of that decision through Bellman backup. It is not simply trained to imitate “the human closed here”; it learns whether closing in that kind of state improves expected task return. This distinction is central to HIL-SERL’s ability to improve beyond demonstrations and interventions.

The visual below compresses this idea into the two-lane structure of the controller. A shared observation $\mathbf{s}$—vision plus proprioception—feeds both the continuous motion policy and the discrete gripper critic. The blue lane corresponds to $\mathcal{A}_1$, where RLPD-style actor-critic learning remains appropriate for $\mathbf{a}_{eef}$. The orange lane corresponds to $\mathcal{A}_2$, where the gripper critic scores a small set of commands such as open, close, and stay.

The equations in the visual summarize the coupled-MDP interpretation, the DQN-style gripper loss, and the inference-time factorization. The main takeaway is that HIL-SERL does not force dexterous manipulation into a single monolithic action model. It preserves the continuous RL machinery where it works well, and adds a small discrete value-learning problem exactly where the robot’s action space becomes categorical.

Once we separate dexterous control into a continuous motion MDP and a smaller discrete gripper MDP, the remaining question is operational: at every real robot timestep, which part of the system chooses what, what happens when a human intervenes, and how do those choices become training data? HIL-SERL’s answer is deliberately simple. The learned motion policy $\pi_\theta$ proposes continuous end-effector motion, while a separate gripper critic $Q_{\theta_g}$ chooses among a tiny set of discrete gripper commands.

This split matters because the two control problems have very different geometry. End-effector motion is naturally continuous: the robot must adjust pose, velocity, or delta commands smoothly in response to visual observations. Gripper control, however, is often sparse and mode-like: open, close, or stay. Treating those gripper decisions as continuous outputs can make learning unnecessarily brittle, especially in bimanual manipulation where opening or closing at the wrong moment can destroy the trial.

So HIL-SERL executes a hybrid action of the form

$$\mathbf{a}_t = (\mathbf{a}_{eef,t}, \mathbf{a}_{gripper,t}),$$

where the motion component is sampled from the actor,

$$\mathbf{a}_{eef,t} \sim \pi_\theta(\cdot \mid \mathbf{s}_t),$$

and the gripper component is selected greedily from the discrete critic,

$$\mathbf{a}_{gripper,t} = \arg\max_{\mathbf{a}\in\mathcal{A}_2} Q_{\theta_g}(\mathbf{s}_t,\mathbf{a}).$$

Here $\mathcal{A}_2$ is small. With one gripper, it contains three options: open, close, and stay. With two grippers, each gripper independently has three options, producing $3^2=9$ joint gripper actions. That is small enough that exact maximization is trivial; there is no need to sample, relax, or approximate the discrete argmax.

A subtle but important point is that the gripper critic is not trained as a supervised classifier of “the correct gripper action.” Human interventions do not simply become labels saying “do this gripper command in this state.” Instead, they become off-policy reinforcement learning transitions. The replay buffer records what state the system was in, what action was actually executed, what reward followed, and what next state resulted. This preserves the RL semantics: an intervention is valuable not because it is expert-labeled in isolation, but because it changes the future trajectory and therefore the return.

This is especially important for gripper timing. In manipulation, the best time to close a gripper may be defined by delayed consequences: a grasp that looks reasonable immediately may fail one second later, while a seemingly conservative “stay open” action may enable a better approach. A supervised loss would compress this into local imitation. The gripper critic instead asks: if I choose this discrete gripper action in this state, what return should I expect under the future policy?