The starting point for ACT is a subtle mismatch between how manipulation tasks are described in everyday language and how they actually unfold in contact with the world. We might casually say that a robot should “open a condiment cup,” but physically the task is not a single action. It is a chain of small, tightly coupled substeps: the right gripper tips the cup, nudges it toward the left gripper, both grippers lift the cup together, and then the left gripper pries off the lid while the right gripper holds the container steady. Each substep changes the object’s pose, the contact state, and the locations where force is being applied. None of these substeps is individually spectacular, but the whole task succeeds only if the robot passes through all of them without breaking the chain.
The central difficulty is that fine manipulation has very low tolerance for geometric error. A cup that is grasped a few millimeters too high, too far back, or at the wrong angle can slip, buckle, or fail to transfer into the other gripper. A lid-prying motion that is slightly misaligned can miss the ridge entirely or tear the thin plastic in an unpredictable way. These are not small numerical errors that merely reduce task quality; they are bistable, contact-rich failures that can terminate the attempt. The condiment cup is deformable, the lid is compliant, friction is hard to model, and the force required changes depending on exactly how the cup is held. Because the dynamics are so sensitive to state, hand-engineered controllers built from rigid assumptions about object geometry are brittle.
This is why fine manipulation must be treated as a closed-loop perception-and-control problem. The robot cannot simply play back a fixed trajectory, because tiny deviations in the object, the gripper, or the environment shift the entire contact interaction. It must observe the current state and react. In the condiment-cup example, that means the policy needs to see whether the cup is still tilted correctly, whether the transfer into the left hand is progressing, and whether the lid has actually separated. A controller that ignores this feedback will fail as soon as reality deviates from the exact trajectory it was tuned for.
One traditional solution is to make the hardware and sensing precise enough that open-loop or lightly supervised methods can work. Expensive robot arms, stiff transmissions, high-resolution encoders, precise calibration, and high-end force or vision sensors can all reduce uncertainty. But that approach makes fine manipulation inaccessible: the hardware is costly, the calibration is fragile, and reproducing results across laboratories is difficult. Low-cost arms, in contrast, have backlash, joint compliance, noisy actuators, and less reliable absolute positioning. Their raw pose estimates may be off by more than the acceptable tolerance for a given substep. Under a conventional precision-first approach, those errors appear disqualifying.
ACT proposes to invert that intuition. Instead of asking the hardware to be precise enough for a hand-designed controller, the idea is to learn a visuomotor policy that can absorb the hardware’s sensing and actuation uncertainty through closed-loop behavior. Humans do something similar: our hands are not calibrated to micron accuracy, but we use vision, touch, and reactive adjustment to open flimsy containers, manipulate deformable objects, and recover from small slips. The hypothesis is that end-to-end imitation learning can transfer this adaptive, feedback-driven competence to imprecise, low-cost robots.
The proposed system has two halves. The first is ALOHA, a low-cost bimanual teleoperation setup in which a human controls two follower arms through two leader arms, collecting demonstrations in joint space while multiple cameras observe the workspace. The second is ACT, a generative imitation learning algorithm that learns a closed-loop policy from those demonstrations. ACT explicitly addresses the fact that behavioral cloning can drift into states the demonstrator never visited; action chunking and temporal ensembling stabilize that closed-loop execution, while a conditional VAE gives the policy a way to represent the multimodal, fine-grained behavior needed for contact-rich manipulation.
A compact way to hold the whole argument in view is the visual below. Its left panel situates the problem with a low-cost bimanual workspace: two arms, a shared manipulation area, and several camera views that provide the closed-loop visual feedback the policy needs. The right panel compresses the condiment-cup task into four sequential stages, using simple amber cup outlines and blue grippers to emphasize the changing contact geometry. Under each stage, a red dashed tolerance band labeled with the scale of millimeter error marks the boundary between success and failure, with a red cross branching away from the task chain. That visual structure makes the core message tangible: fine manipulation is not one hard motion, but a fragile sequence of low-tolerance contact events.
The closed-loop arrow at the bottom of that visual is just as important as the step sequence itself. It indicates that the robot must continuously circle back through perception and action rather than execute a fixed script. Each stage in the condiment-cup example is an opportunity for feedback to correct small errors before they propagate. Once we appreciate this chain structure, the next question becomes unavoidable: if a learned policy is copied from demonstrations, why does it still drift off the demonstrated chain? That is the compounding-error failure mode, and it is the precise problem ACT is designed to address.
If the previous section framed fine manipulation as a hardware and perception problem, the learning side has its own subtle failure mode. Behavioral cloning is attractive because it reduces skill acquisition to supervised learning: collect demonstrations, extract observation-action pairs, and train a policy to imitate the demonstrated action at each observed state. In principle, a policy is trained so that is close to the expert action . On held-out demonstration data, this often looks excellent. The policy may achieve low validation error and appear ready for deployment. The difficulty is that behavioral cloning measures the wrong kind of error: it optimizes one-step agreement under the distribution of states visited by the expert, not the distribution of states the learned policy will actually visit.
During closed-loop execution, the policy is no longer being evaluated on states sampled from the demonstration distribution. It observes its own resulting states, one after another, and each action determines the next observation. If the policy makes a small mistake at time , then the robot ends up in a state that is slightly different from the expert's . The next observation is therefore slightly outside the training distribution. The policy may still behave reasonably, but its error at that new state can be larger than its error at the original training state. That larger error pushes the robot even further from the demonstrated trajectory. This feedback loop is the core of compounding errors.
Mathematically, the problem comes from the mismatch between the training distribution and the policy-induced distribution. Behavioral cloning assumes, implicitly, that the states encountered during training and testing are drawn from the same distribution. In sequential control, that assumption is violated almost immediately. If is the state distribution of the expert and is the state distribution induced by the learned policy, then even a policy with small average error under can have much larger error under . The policy has no reason to know how to recover from states the expert never visited. In the worst case, a per-step error can lead to an expected trajectory divergence that grows like , where is the task horizon. This is far worse than the linear accumulation one might naively expect, because each error does not just add displacement; it also moves the agent into a region where the policy is less reliable.
Low-cost hardware makes this failure mode particularly severe. Inexpensive servo motors, 3D-printed linkages, cable-driven transmissions, and commodity cameras introduce backlash, compliance, noisy sensing, and imperfect calibration. An action that is correct in the demonstration may produce a slightly different robot motion in deployment. That hardware-induced deviation is exactly the kind of small state perturbation that behavioral cloning is poorly equipped to handle. Fine manipulation compounds the issue because the acceptable state error is often tiny: grasping a small object, inserting a peg, or coordinating two hands requires millimeter-level or even sub-millimeter accuracy. A policy that would have succeeded on a precise industrial arm can fail repeatedly on low-cost hardware because the combination of mechanical noise and learned-policy drift pushes the robot outside the narrow region where its cloned behavior is valid.
Another way to understand the failure is that behavioral cloning learns a conditional distribution but does not learn a robust state-to-state transition strategy. It is reactive rather than predictive. When a human teleoperates a robot, the human continuously corrects small errors without needing to think about them. Those corrections appear in the demonstration data, but only as local adjustments around a successful trajectory. The data rarely contains examples of large deviations or recovery from serious mistakes, because the human demonstrator is usually competent and keeps the robot near the intended path. As a result, the learned policy may be locally competent but globally fragile. It can imitate the expert near the demonstrated path, yet it has little capacity to return to that path once the robot has drifted.
This explains a common empirical observation: cloned policies can have excellent validation accuracy in the lab and still fail quickly in deployment. The first few actions may look plausible, but small mismatches accumulate into hesitation, overshoot, oscillation, or a wrong grasp. The robot may pause because the policy is uncertain in an unfamiliar state, then choose an action that creates an even less familiar state. With bimanual tasks, the problem is amplified because errors in one arm change the object pose and the configuration of the other arm, multiplying the number of out-of-distribution states. The policy may not simply fail; it may fail in a way that is hard to predict from the demonstration data alone.
There are a few important takeaways. Behavioral cloning is a one-step imitation objective, not a trajectory-level objective. It assumes the state distribution is fixed, but sequential control makes the state distribution depend on the policy itself. Hardware noise, visual noise, and tight manipulation tolerances shorten the horizon over which compounding errors become catastrophic. Finally, collecting more demonstration data can help, but it does not by itself solve the distribution-shift problem unless the data include meaningful recovery behavior.
The visual below condenses this feedback loop into a single diagrammatic story. It contrasts the expert trajectory with the policy’s actual rollout, showing how a small initial deviation at one step creates a new state that is slightly off the demonstrated path. From there, the policy’s next error is drawn from a less reliable part of the learned mapping, producing a larger deviation. The shaded widening region between the expert path and the actual path is a compact way to represent the compounding effect: the gap does not merely persist, it grows. By making the loop explicit—execute action, land off-trajectory, observe unfamiliar state, make larger error—the diagram reinforces why validation accuracy can be misleading and why a better behavioral-cloning objective must address temporal consistency, not just one-step action matching.
If compounding errors are the disease, then the first step toward a cure is being very precise about what is being imitated. The previous section established that a policy trained by behavioral cloning can drift: one mildly wrong action lands the robot in an observation it has rarely seen, the next action is even worse, and the failure cascades. To discuss fixes, we need a common notation for observations, actions, trajectories, and the objective that behavioral cloning actually optimizes.
At a given control time , let
be the observation visible to the robot, and let
be the action selected by the policy. In ALOHA-style manipulation, an observation usually includes one or more camera images as well as proprioceptive quantities such as joint angles and gripper positions. An action is typically a joint-space command, often a vector of target joint positions or small deltas for the arms and grippers. The spaces and are deliberately abstract in the notation because the exact observation preprocessing and action representation can vary between tasks, but the imitation-learning structure remains the same.
A demonstration is then a finite sequence
where is the episode length. The dataset is a collection of such expert trajectories:
This formulation is deliberately reward-free. Unlike reinforcement learning, behavioral cloning does not require a scalar reward signal or a value function. It simply asks the policy to reproduce the expert’s action distribution.
The most basic behavioral-cloning policy is a stationary mapping from the current observation to an action distribution,
with parameters . Training minimizes the negative log-likelihood of the demonstrated actions:
This looks simple, but it hides an important assumption: the policy makes only a one-step prediction. At runtime, the predicted action changes the world to , and the policy must then act from that new observation. The training objective never explicitly accounts for this feedback loop, which is exactly why distribution shift can cause failures even when the one-step validation loss is low.
A key notational idea in ACT is to replace the one-step prediction with a chunk. Instead of predicting only , the policy predicts a sequence of future actions from the current observation:
The integer is the chunk size or prediction horizon. In a low-level manipulation task, a chunk might cover a few hundred milliseconds of motion. Predicting a short chunk rather than a single action encodes temporal consistency: neighboring actions in a smooth manipulation trajectory are highly correlated, and committing to a short sequence prevents the policy from changing its mind at every control tick. At the same time, the chunk is not so long that the robot blindly executes an entire episode without feedback.
There is a second bookkeeping issue that becomes important later. If the policy produces overlapping chunks over time, then the same physical action can be predicted from several different starting observations. For example, the action at time might be predicted as part of the chunk emitted at time , but it might also appear at the second position of the chunk emitted at time . A useful notation for this is
meaning the predicted action for timestep produced by a chunk that started at timestep . Keeping track of the starting index is what makes temporal ensembling possible: instead of executing one noisy prediction, the robot can average several predictions for the same future action.
For bimanual tasks, the action vector is usually the concatenation of commands for both arms. If the robot has two arms and each arm has its own joint configuration and gripper state, then
This compact notation hides significant physical complexity: low-cost arms may have backlash, joint elasticity, and different calibration between arms, yet the policy must coordinate both hands for precise tasks such as threading a cable or transferring a delicate object. The observation can similarly combine multiple camera viewpoints and two sets of proprioceptive measurements.
The visual below condenses this setup into a single glance. It shows a trajectory as an alternating sequence of observations and actions, the policy as a mapping from the current observation to a chunk of future actions, and the dataset as many such expert trajectories. It also highlights the two forms of prediction that matter for ACT: a standard one-step action and the overlapping chunk-based action sequence. This bookkeeping is the foundation for both the action-chunking architecture and the temporal ensembling rule discussed next.
The notation from the previous section treats a manipulation trajectory as an abstract sequence of observations and actions. To make that abstraction concrete, ACT relies on a particular source of real robot demonstrations: ALOHA, a low-cost bimanual teleoperation setup. The hardware matters more than it might first appear, because fine manipulation on inexpensive arms is not just a smaller version of industrial manipulation. Hobby-class arms have joint backlash, link deflection, nonuniform friction, and limited encoder resolution. A command that sends every joint to exactly the same angle can produce slightly different Cartesian poses from one trial to the next. For tasks like threading a cable through a clip, inserting a battery, or zipping a pouch, those millimeter-level inconsistencies are often larger than the clearance the robot is trying to exploit.
The key idea behind ALOHA is to remove the need for an autonomous controller to be perfect at the start. Instead, a human operator performs the task through leader-follower teleoperation. The system has two pairs of arms: two leader arms that the human moves, and two follower arms that execute the corresponding motion in the workspace. The leader arms are backdrivable, meaning the operator can move them freely by hand. Their joint encoders are read continuously, and the measured joint angles are sent as position targets to the matching joints on the follower arms. Because the leader and follower arms have the same kinematic structure, the motion can be mirrored directly in joint space, without solving an inverse-kinematics problem from a desired end-effector pose.
This joint-space choice is subtle but important. Teleoperating in Cartesian space would require the operator to specify only where the wrist should go, leaving the arm to resolve how the elbow, shoulder, and wrist should be arranged. For many fine tasks, however, the whole arm posture matters. A human naturally uses a particular elbow angle to avoid an obstacle, to maintain a stable grip, or to apply a small lateral force. ALOHA captures that full posture directly by recording joint angles. It also sidesteps the inverse-kinematics instabilities and singularities that plague low-cost arms, making the whole pipeline easier to deploy.
The bimanual aspect is equally central. Many manipulation tasks are not two independent single-arm problems. One hand often stabilizes an object while the other hand performs a precise insertion, pulls a flap, or aligns a connector. ALOHA lets one operator control both leader arms simultaneously, so the recorded data preserves the natural temporal coordination between the two hands. Each demonstration therefore contains synchronized joint-angle streams for both follower arms, including
Once a policy produces future actions at every step, the immediate question is how to turn that redundant stream of predictions into a single command for the robot. The simplest answer—execute all actions, then re-plan—turns the policy partially open-loop for steps. That may be acceptable for short horizons, but it throws away the possibility of reacting to fresh observations during the chunk. The alternative is to query the policy at every timestep, so consecutive chunks overlap heavily. At time the model predicts a chunk beginning at , and at time it predicts another chunk that shares future timesteps with the previous one. Temporal ensembling is the rule used by ACT to combine those overlapping estimates into a smooth executed action.
Formally, let denote the prediction for timestep produced by the chunk whose first predicted action is at timestep . After running the policy at time , the controller has access to estimates for the current action made at several recent start times:
The first estimate is the freshest, produced from the current observation. The last is the oldest, produced steps ago when the current time was only the final element of the predicted chunk. ACT fuses these estimates with an exponentially decaying weighted average:
where is the age of the prediction and controls how fast older predictions are discounted. At the beginning of an episode, or after a reset, fewer than overlapping chunks are available; the same normalized formula naturally uses whatever estimates exist.
The exponential weighting captures a simple prior: newer predictions were conditioned on observations closer to the present state, so they are usually more trustworthy. Older predictions can be stale if the scene has shifted, contact forces have changed, or the arm has moved into a different configuration. Discounting them with makes the ensemble react more to the latest model output while still retaining some evidence from the recent past. The parameter sets a bias–variance trade-off. If is very small, the ensemble is nearly a uniform average over the whole chunk horizon, producing strong smoothing but slower reactions to fast disturbances. If is large, only the newest prediction matters and the method approaches ordinary single-step behavioral cloning at inference time.
The combination of action chunking and temporal ensembling is subtle and important. Chunking gives the policy a way to commit to a consistent short-horizon trajectory, which reduces the effective task horizon and helps combat compounding errors. Temporal ensembling prevents that commitment from becoming rigid. The controller still observes the environment and queries the model every step, so the policy remains closed-loop. It simply executes an exponentially weighted consensus of the most recent chunks instead of the newest raw prediction.
This also gives a useful interpretation: temporal ensembling is a test-time ensemble across time rather than across models. The same policy, with slightly different observation histories, makes several predictions for the same action. Averaging those predictions reduces variance, much like averaging an ensemble of classifiers, but it does so without training multiple networks or running additional forward passes beyond one per timestep. In manipulation, lower variance often translates into smoother joint commands, less jitter, and more consistent contact behavior.
There are failure modes worth keeping in mind. Averaging in action space assumes that the midway point between two action vectors is meaningful. This is usually reasonable for continuous joint-position targets, which is the regime of ALOHA and many low-cost arms, but it would be questionable for discrete modes or actions with strong non-convex constraints. Averaging can also introduce a mild lag when the correct action changes rapidly, because the executed command is pulled toward older predictions. The smoothing parameter is therefore an important inference-time knob: smaller creates smoother but more conservative motion, while larger increases reactivity but sacrifices some of the variance reduction.
The visual below condenses this idea into the overlapping-chunk picture. It shows several action chunks as horizontal bars, each starting one step after the previous one. A vertical slice at the current timestep crosses one estimate from each active chunk, and those estimates are combined with weights that decay as the slice moves from the newest chunk to the oldest. The key takeaway from the image is that no single chunk has exclusive authority over the action sent to the robot; the executed action is a moving, exponentially weighted consensus that continuously shifts
With the inference procedure now in hand, the natural question is whether action chunking and temporal ensembling actually matter when the policy is deployed on real hardware. Training loss and qualitative smoothness are useful, but the paper’s central empirical claim is about task success: can a low-cost bimanual system perform fine manipulation reliably enough to complete a sequence of delicate, contact-rich tasks?
The evaluation protocol is deliberately close to what a practitioner would care about. For each task, the robot is trained on a relatively small set of human teleoperated demonstrations—around fifty episodes per task in the ALOHA setup—and then evaluated over a fixed number of real-world rollouts. Success is binary: either the task is completed within the trial window, or it is not. This matters because a policy that looks accurate in a mean-squared-error sense can still fail the actual task if it hesitates, drifts out of distribution, or never commits to a coherent motion.
The comparison in the main results is not against weak baselines. The paper contrasts ACT with at least two strong same-data imitation learners. BC-ConvMLP is a standard behavior-cloning policy that maps visual observations and joint states to the next action, reflecting the conventional one-step regression approach. BeT, or Behavior Transformer, is a transformer-based policy that predicts actions with discretized bins and offsets, explicitly modeling multimodality but still generating actions one step at a time. ACT combines three ingredients that neither baseline has all at once: short-horizon action chunks, a conditional VAE for multimodal action distributions, and temporal ensembling at inference.
The headline result is a large and consistent gap in success rate across bimanual manipulation tasks. On precise tasks—such as opening a cup, placing an eggplant into a pan, or inserting bread into a toaster—ACT completes the task far more often than the baselines. The failures of the baselines are often not random; they tend to exhibit systematic behaviors such as stopping slightly too early, moving too cautiously near contact, or drifting a few millimeters off the correct insertion pose and then failing to recover.
That pattern is exactly what the action-chunking formulation is designed to prevent. A one-step policy trained with behavior cloning is evaluated at every step under its own previous errors. In a high-precision task, even small deviations from the demonstrated state distribution can push the policy into regions
After examining how chunk size and temporal ensembling change ACT’s behavior, it is tempting to treat those two design choices as the main source of success. But both of them sit on top of a more fundamental modeling decision: the policy is not a deterministic regressor. The next ablation set asks what happens when we remove the probabilistic objective, and separately, what happens when we change how frequently the learned policy is allowed to issue fresh control commands. Together, these two axes clarify that ACT needs both a suitable action distribution and a fast enough control loop.
The first issue is that fine manipulation often produces multimodal demonstrations. A low-cost bimanual setup has substantial joint backlash, compliance, and sensing noise. Two human demonstrations may start from nearly the same visual scene, but diverge at a contact point: one demonstration approaches an insertion from the left, another from the right, and a third pauses briefly before pushing. A standard behavioral cloning model trained with mean-squared error or a deterministic action chunk decoder tends to average these demonstrated future actions. In many states, that average is not a physically plausible action at all. It may place the gripper in the middle of a constrained opening, or command both arms to move in a way that cancels out the small corrective motion needed to make contact. This mode-averaging failure is especially expensive in contact-rich tasks
Having isolated the CVAE objective and high-frequency control in the ablations, the natural question is whether ACT is simply a collection of individually useful tricks. The more useful summary is that each design choice in ACT addresses a specific failure mode that appears when fine-grained bimanual manipulation meets low-cost hardware. Low-cost arms tend to have backlash, compliance, noisy proprioception, and limited torque bandwidth. They can still perform precise tasks, but the control stack must be more patient and more structured than a standard behavior-cloning pipeline.
Behavioral cloning on raw per-step actions is especially fragile in this setting. If the policy predicts one action at a time, a small error at time changes the observation at , pushing the policy into a slightly different state distribution. The next prediction then has a chance to make another small error, and the trajectory diverges over time. This is the classic compounding error problem. One way to reduce it is not to ask the policy to make a fresh decision at every instant, but to commit to a short, coherent plan and re-plan only after executing part of it. That is the first core idea behind action chunking.
Instead of predicting a single action , ACT predicts a chunk of future actions from the current observation. This has two effects. First, the policy makes fewer independent decisions over a fixed episode, which slows the accumulation of distribution shift. Second, each predicted chunk is forced to be internally consistent over a short horizon, which acts as a mild regularizer against jittery or contradictory commands. The trade-off is that committing to a chunk can cause discontinuities at chunk boundaries if a new chunk replaces the old one abruptly. The solution is temporal ensembling: when several overlapping chunks contain predictions for the same time step, ACT averages them with weights that emphasize the most recent, most confident parts of each chunk.
Formally, if a time step is covered by several chunks , and each chunk contributes a predicted action with weight , the executed action is
This is not a post hoc smoothing filter added for cosmetic reasons. It is a direct response to the chunking mechanism: chunking gives robustness over longer horizons, while temporal ensembling restores smoothness at the transitions between chunks. In the low-cost ALOHA setting, this combination produces visibly smoother bimanual motion than either raw single-step control or naive chunk replacement.
The third component, the conditional VAE objective, handles a different problem. Demonstration chunks are not unique: a human operator may use several equally valid strategies for the same task, or may grip an object from a slightly different angle on different demonstrations. A deterministic regression objective will average these modes together, producing blurry or infeasible actions. ACT instead encodes an entire action chunk into a compact latent variable , conditioned on the current observation. During training, the encoder sees the ground-truth action chunk and learns a posterior distribution over plans. The decoder reconstructs the chunk from the observation and the sampled plan. The loss combines reconstruction quality with a KL term that pulls the posterior toward a learned or fixed prior:
At inference time, the encoder is removed and the policy samples from the prior before decoding the chunk. This gives ACT the ability to commit to one latent plan rather than averaging incompatible plans, while the KL term prevents the latent space from becoming degenerate.
The demonstration data matters just as much as the policy architecture. ALOHA’s leader–follower joint-space teleoperation produces dense sequences of joint targets rather than end-effector poses or image-space waypoints. Joint-space control is natural for low-cost arms because the action is directly what the robot needs to execute; there is no external calibration, retargeting, or inverse-kinematics mismatch. The human operator supplies the fine-grained bimanual coordination, while the arms record high-frequency joint commands. This shifts the learning problem from “invent a control law” to “compress and reproduce a demonstrated control law,” which is exactly what action chunking and the CVAE are designed to do.
The ablations confirm that these components are not interchangeable. Removing the CVAE objective forces a deterministic bottleneck over multimodal demonstrations, which degrades performance on tasks that require subtle alternative strategies. Removing high-frequency control or replacing it with slower, smoother commands sacrifices the fine corrections that make low-cost hardware capable of precise insertion or cable routing. The result is a coherent design loop: teleoperation supplies dense bimanual supervision, chunking reduces effective decision frequency, temporal ensembling smooths the output, the CVAE captures multimodality, and high-frequency joint-space execution preserves the fine-grained signal.
The visual below compresses that loop into three interlocking ideas. At the center is action chunking, which reduces the planning horizon and slows compounding error. Around it, temporal ensembling averages overlapping predictions into a smooth command stream, and the CVAE latent plan supplies a compressed, multimodal representation of each chunk. The supporting conditions are equally important: joint-space teleoperation provides the right data, while high-frequency low-level control preserves the manipulations that the policy produces. Taken together, ACT is not a single architectural trick but a carefully matched set of choices that make imitation learning viable on inexpensive, imperfect hardware.
