HaWoR: Putting Egocentric Hands Back in the World

Picture someone wearing head-mounted camera glasses in a kitchen. They pick up a kettle from the counter, turn around to face a cup behind them, and pour. The glasses record the whole thing from roughly the wearer's eye position. Now suppose you want the answer to a simple-sounding question: where did the right hand go during those few seconds? Not where it appeared in the image, but where it travelled through the room, in meters.

A modern single-image hand reconstructor such as HaMeR or WiLoR answers a different question. Given one frame, it returns a 3D hand mesh placed relative to the camera that took the picture. Its output is a wrist orientation and position expressed in the camera frame, plus the articulation of the fingers. That is genuinely useful, but the camera here is strapped to a head, and heads move constantly. Each frame's answer is relative to a coordinate frame that has shifted since the previous frame.

A thought experiment makes the problem concrete. Let the wearer hold the kettle perfectly still at waist height while turning their head 90° to the left. In the room, the hand has not moved at all. In the camera frame, though, the hand starts near the center of view and slides steadily toward the right edge as the head turns. A per-frame camera-frame reconstruction will faithfully report that sweep. If you naively string those per-frame outputs together, you get a hand trajectory that is really a record of head rotation. The method has not made an error on any individual frame; the error comes from interpreting a sequence of camera-relative answers as if they shared one fixed frame.

The fix is a standard composition of rigid transforms, and it tells us exactly what extra information we need. Write the camera-frame hand orientation of hand $i$ at frame $t$ as $\Phi_t^{c_t,i}$ (an axis-angle vector, treated as a rotation matrix when we multiply) and its camera-frame root position as $\Gamma_t^{c_t,i}$. Write the camera's pose in the world at that frame as a rotation $R_{c_t}$ and a translation $t_{c_t}$, mapping camera coordinates into world coordinates. Then the hand's world orientation $\Phi_t^i$ and world root translation $\Gamma_t^i$ are

$$\Phi_t^{i} = R_{c_t}\,\Phi_t^{c_t,i}, \qquad \Gamma_t^{i} = R_{c_t}\,\Gamma_t^{c_t,i} + t_{c_t}.$$

The first equation rotates the hand's orientation out of the camera's axes and into the room's axes. The second takes the hand's position as seen from the camera, rotates that offset into world axes, and adds the camera's own location. Apply it to the thought experiment: as the head turns, $\Gamma_t^{c_t,i}$ swings sideways, but $R_{c_t}$ turns by exactly the opposite amount, so the product $R_{c_t}\Gamma_t^{c_t,i} + t_{c_t}$ stays fixed. The world position comes out constant, which is the truth.

This composition also shows where things can go wrong. The world trajectory is only as good as both factors. A perfect camera-frame hand paired with a camera rotation that is off by a few degrees produces a world hand that drifts. A perfect camera pose paired with a jittery camera-frame hand produces a world hand that jitters. And if the camera translation $t_{c_t}$ is right in direction but wrong in scale, every reach and every pour is stretched or shrunk. In the kettle example the head turns roughly half a circle and the pouring hand can swing out of the field of view entirely, so all three failure modes are in play at once.

For robotics, the distinction is not academic. If you want to learn a manipulation policy from egocentric human video, or retarget a human demonstration to a robot arm, the robot needs to know where the wrist went relative to the counter, the kettle and the cup. A trajectory relative to a swinging head describes the person's gaze as much as their hand. So a world-space hand reconstruction needs two ingredients from the same video: good hands in the camera frame, and a good camera-to-world pose $R_{c_t}, t_{c_t}$ for every frame. Getting the second ingredient from a head-mounted monocular camera, with large hands moving through the view and the hands often leaving it, turns out to be much harder than it is for the third-person, full-body settings where world-grounded reconstruction was first developed.

The paper identifies three obstacles that make egocentric hands harder than the earlier world-grounded body settings, plus a fourth problem that comes from the hands themselves.

The first is metric scale. The standard way to get $R_{c_t}, t_{c_t}$ from a single moving camera is visual SLAM, and monocular SLAM has a well-known blind spot: it can recover the shape of the camera's path and the relative depths of the scene, but not their absolute size. A video of a dollhouse filmed from 10 cm away and a real house filmed from 10 m away can produce the same images. The SLAM translation is therefore only correct up to an unknown factor $\alpha$, and the metric translation is

$$t_{c_t}^{\text{metric}} = \alpha\, t_{c_t}^{\text{SLAM}}.$$

For a person walking across a room, an error of 10% in $\alpha$ shifts a 5 m path by half a meter, which is noticeable but leaves the gist intact. For hands it is much worse. A reach to the kettle covers a few tens of centimeters, and in $\Gamma_t^i = R_{c_t}\Gamma_t^{c_t,i} + t_{c_t}$ the camera term $t_{c_t}$ is added directly to the hand position. Because the head itself moves during the reach, a mis-scaled $t_{c_t}$ injects error of the same order as the motion we are trying to measure.

The second obstacle is visibility. A head-mounted camera sees what the wearer is looking at, and people do not watch their hands all the time. In the kettle example, as the wearer turns toward the cup, the pouring hand can swing out of the field of view or sit at the very edge of the frame, partly cut off. For those frames a camera-frame hand network has nothing to work with, so $\tilde{\Theta}_t^i$ and $\Gamma_t^{c_t,i}$ are simply missing for runs of consecutive frames. Any world trajectory built frame by frame from them has holes exactly when the hand is doing something interesting.

The third obstacle is priors. When observations are missing or noisy, a motion prior learned from motion-capture data can fill in plausible movement, which is how body methods cope with occlusion. Large-scale hand mocap, however, is scarce compared with body mocap, and hand motion combines a moving wrist with articulated fingers interacting with objects. Strong learned priors for hand motion are therefore harder to come by.

There is also a failure that the hands cause in the camera estimate itself. SLAM assumes the scene is static and estimates the camera by tracking how background pixels move between frames. In egocentric video the hands often cover a large part of the image and move independently of the camera. Pixels on a moving hand give the SLAM system false evidence about how the camera moved, so the very object we want to reconstruct corrupts the camera pose we need in order to place it.

It helps to see where existing methods sit relative to these obstacles. World-grounded body methods solve world placement, but for full bodies seen from a third-person view. GLAMR and WHAM regress global trajectories directly. SLAHMR and PACE optimize SLAM cameras jointly with body motion priors, which handles occlusion through the prior but is slow. TRAM combines SLAM with a metric depth network to recover scale. None of them reconstructs hands. Single-image hand methods such as HaMeR and WiLoR reconstruct detailed hands but work per frame in the camera frame, so they jitter in video, fail when the hand is truncated at the image border, and produce nothing when it is out of view.

Method family	World frame?	Hands?	Out-of-view frames?
GLAMR, WHAM (regression)	yes	no (body)	partial (body occlusion)
SLAHMR, PACE (SLAM + prior optimization)	yes	no (body)	via prior, slow
TRAM (SLAM + metric depth)	yes	no (body)	no
HaMeR, WiLoR (single image)	no (camera)	yes	no, per frame
HaWoR	yes	yes	yes (infiller)

The table shows the gap. Body methods give world grounding without hands, and hand methods give hands without world grounding. Simply bolting a hand model onto a body-style SLAM pipeline runs straight into the issues above: unknown scale magnified by small motions, missing frames, weak priors, and hand pixels contaminating the camera estimate. HaWoR's design responds to each of these in turn, and it starts by deciding how to split the problem.

Before splitting anything, we need a precise target. HaWoR represents each hand with MANO, a parametric hand model widely used in hand reconstruction. MANO turns a small set of numbers into a full triangle mesh of a hand. The articulated pose $\Theta_t^i \in \mathbb{R}^{15\times 3}$ gives an axis-angle rotation for each of 15 finger joints, so it says how the fingers bend. The shape coefficients $\beta_t^i \in \mathbb{R}^{10}$ say how the hand is built: long or short fingers, a broad or narrow palm. Two more quantities place that posed hand in space: a wrist orientation $\Phi_t^i \in \mathbb{R}^3$ and a root translation $\Gamma_t^i \in \mathbb{R}^3$. That is 61 numbers per hand per frame, and from them MANO produces the mesh and its joint positions.

The task can now be stated exactly. The input is an egocentric RGB video $\mathbf{V} \in \mathbb{R}^{T\times H\times W\times 3}$ of $T$ frames, each $H$ by $W$ pixels. The output, for every tracked hand $i$ (left or right), is the sequence

$$\{\Theta_t^i,\ \beta_t^i,\ \Phi_t^i,\ \Gamma_t^i\}_{t=0}^{T}$$

with $\Phi_t^i$ and $\Gamma_t^i$ expressed in world coordinates. Two details matter. The sequence covers every frame, including frames in which the hand cannot be seen. And "world coordinates" here means a fixed frame anchored to the room, with translations in meters, so that a 30 cm reach to the kettle comes out as 30 cm.

One could imagine a single network that eats the video and emits world-space MANO parameters directly. HaWoR does not do that. Instead it decouples the problem, using the composition from the first section as the seam. That equation says the world hand is a camera-frame hand composed with a camera-to-world pose. Each factor can be estimated by a separate module, each module can be trained on data suited to it, and each can be checked against its own benchmark. The decomposition has four parts:

1. A hand motion network $\mathcal{M}$ estimates the camera-frame hand for each frame: pose $\tilde{\Theta}_t^i$, shape $\tilde{\beta}_t^i$, orientation $\Phi_t^{c_t,i}$ and translation $\Gamma_t^{c_t,i}$. The tilde marks quantities predicted by this network.
2. A metric egocentric SLAM module estimates the camera pose $R_{c_t}, t_{c_t}$ for each frame, with translations in meters.
3. The two are composed: $\Phi_t^i = R_{c_t}\Phi_t^{c_t,i}$ and $\Gamma_t^i = R_{c_t}\Gamma_t^{c_t,i} + t_{c_t}$.
4. A motion infiller $\mathcal{F}$ fills frames where the hand was not observed, producing the completed sequence $\hat{\Theta}_t^i, \hat{\beta}_t^i, \hat{\Phi}_t^i, \hat{\Gamma}_t^i$. The hat marks infiller outputs.

Why is this split easier than the whole? Each subproblem matches one of the obstacles. The camera-frame hand network is a problem the field already solves well for single images, and its training data (hand datasets with 3D annotations) is reasonably plentiful. Camera estimation is a geometry problem that SLAM handles well once the hands are kept from confusing it and the missing scale is supplied. Infilling is a sequence-completion problem that becomes tractable once the camera's motion has been factored out. A single end-to-end network would have to learn all three at once from egocentric video with world-space hand labels, which is the scarcest kind of data in the whole setup.

There is one subtlety in the ordering, and it is easy to miss. The modules are not independent black boxes run in parallel. The hand network's output is used to tell the SLAM module which pixels to ignore, and the SLAM module's metric camera poses are needed before the infiller can work in a frame that does not move with the head. So the decomposition is also a data flow with a definite order. Laying that order out explicitly makes the rest of the method much easier to follow.

Follow the kettle video through the system from left to right. It enters two branches, a hand branch and a camera branch, and they meet again at the end.

The hand branch starts with tools HaWoR does not invent. An off-the-shelf detector from WiLoR finds hands in each frame, and a multi-object tracker (BoT-SORT) links the detections over time into one bounding-box sequence per hand, so the right hand in frame 40 is known to be the same hand as in frame 41. The crops along each track go into the hand motion network $\mathcal{M}$. Inside it, a pretrained vision transformer turns each crop into feature tokens, a temporal attention module shares information between neighbouring frames, a MANO head reads out hand parameters, and a second temporal attention module smooths the pose sequence. The output is

$$\mathbf{V} \xrightarrow{\mathcal{M}} \{\tilde{\Theta}_t^i, \tilde{\beta}_t^i, \Phi_t^{c_t,i}, \Gamma_t^{c_t,i}\},$$

a camera-frame MANO sequence for each hand, with gaps wherever the tracker lost the hand. In the kettle video, that gap falls during the turn toward the cup.

The camera branch needs those hand estimates before it can begin. Each reconstructed hand mesh is projected back into its image to produce a binary mask $M_t$ marking the hand pixels. The masked frames go into DROID-SLAM, a learned dense visual SLAM system, which returns camera rotations $R_{c_t}$, translations $t_{c_t}$ and per-pixel depths $d_t$, all up to an unknown scale. Separately, a monocular metric depth network, Metric3D, predicts depth in meters $D_t$ for keyframes. An adaptive sampling module (AdaSM) chooses which pixels to trust when comparing $d_t$ with $D_t$, and a robust fit produces the scale $\alpha$. Multiplying the SLAM translations by $\alpha$ gives a metric camera trajectory:

$$\mathbf{V}, M_t \xrightarrow{\text{SLAM}+\alpha} \{R_{c_t}, t_{c_t}\}.$$

The two branches merge in three steps. First, each hand's camera-frame sequence is re-expressed in a canonical frame using a transform $T^{c_t2cano,i}$ built from the metric camera poses. This frame is anchored to the hand's own starting pose, so the head's motion no longer appears in the sequence. Second, the infiller $\mathcal{F}$, a transformer encoder, fills the missing frames in that canonical space. Third, a transform $T^{cano2w,i}$ maps the completed sequence back into world coordinates, giving $\hat{\Theta}_t^i, \hat{\beta}_t^i, \hat{\Phi}_t^i, \hat{\Gamma}_t^i$ for every frame. The output pairs the world-space hands with the camera trajectory, which is what the paper's figures render: hand meshes on a floor plane with camera frustums overhead.

Two dependencies in this diagram are easy to overlook, and both shape the design. The dashed arrow from the hand branch into SLAM means $\mathcal{M}$ must run first. If the hand network is badly wrong in some frame, its mask is wrong too, and SLAM can see hand pixels it should have ignored. The arrow from the metric trajectory into the canonical transform means scale errors do not stay inside the camera branch. They flow into the infiller's input and from there into the final world hand. The modules are separate, but their errors propagate along these arrows.

With the full flow in place, we can open each box. The first is $\mathcal{M}$, since everything else waits on it.

HaWoR does not build its hand network from scratch. It starts from WiLoR, a recent single-image hand reconstructor, and reuses WiLoR's pretrained vision transformer (ViT) backbone. Given a crop around hand $i$ in frame $t$, the ViT splits the crop into patches and returns a set of feature tokens $f_t^i$ that encode what the hand looks like in that frame. On a single image this backbone is very strong. The problem is what happens when you run it independently on every frame of a video.

Three failures show up, and the kettle video triggers all of them. First, jitter. Each frame's estimate carries its own small errors, and because nothing ties neighbouring frames together, those errors do not cancel. A hand holding the kettle steady comes out trembling. Second, truncation. As the wearer turns, the right hand slides toward the image border and part of it falls outside the crop, so the network must guess the missing fingers from half a hand. Third, there is no motion prior. A per-frame model does not know that a wrist cannot teleport 10 cm between frames 33 ms apart, so it cannot use the previous frame to reject an implausible answer.

The Image Attention Module (IAM) addresses truncation and part of the jitter. It applies self-attention across time to the image tokens of nearby frames of the same hand:

$$\hat{f}_t^i = \mathrm{IAM}\big(f_{t-k}^i, \dots, f_{t+k}^i\big).$$

Self-attention lets each token in frame $t$ look at tokens from neighbouring frames and pull in whatever is relevant. Consider the frame in which the right hand is cut off at the border. Its own tokens describe only the visible half of the hand. Through IAM, they can attend to tokens from a frame or two earlier, when the hand was still fully in view, and borrow the appearance evidence for the missing fingers. The output tokens $\hat{f}_t^i$ are thus image features enriched with temporal context.

A MANO head then turns these tokens into hand parameters. Following WiLoR, it uses a learnable query token that cross-attends to $\hat{f}_t^i$ and regresses the articulated pose $\tilde{\Theta}_t^i$, the shape $\tilde{\beta}_t^i$, and the hand's orientation $\Phi_t^{c_t,i}$ and translation $\Gamma_t^{c_t,i}$ in the camera frame. At this point each frame has a full MANO estimate, informed by its neighbours' appearance.

Appearance alone has a limitation, and it motivates the second module. Image tokens mix the hand with whatever surrounds it: a kettle handle, a patterned countertop, motion blur. The paper notes that features from the ViT carry this baked-in appearance and background, which does not express hand motion directly. The Pose Attention Module (PAM) therefore applies temporal self-attention to the sequence of MANO pose parameters themselves:

$$\{\tilde{\Theta}_t^i\}_t \leftarrow \mathrm{PAM}\big(\{\tilde{\Theta}_t^i\}_t\big).$$

Working in pose space, PAM sees only how the joints are bending over time. It can learn which pose sequences are typical of real hands, for example that fingers curl smoothly around a handle rather than flickering between grasp and open hand, and it can pull an outlier frame back toward its neighbours. This is a learned hand-motion prior, built into the network rather than added as a separate optimization step.

The training data reflects the egocentric goal. The network sees about one million video frames: 573K from HOT3D (egocentric recordings of everyday hand activity), 165K from ARCTIC (two hands manipulating objects), 169K from DexYCB (hand grasping) and 66K from HO3D (hand-object interaction). The pretrained ViT backbone stays frozen, so what is learned is the temporal machinery and the head on top of WiLoR's features. An important caveat follows from the design. IAM and PAM can only bridge short truncations and noisy frames, because they need nearby observations to attend to. When the hand leaves the view for longer, as during the turn in the kettle video, $\mathcal{M}$ still produces nothing, and the gap waits for the infiller.

Training $\mathcal{M}$ means deciding what "correct" looks like for a video clip of a hand. HaWoR uses three kinds of supervision at every frame and sums them over the clip:

$$\mathcal{L}_{\mathcal{M}} = \sum_{t=1}^{T}\big(\lambda_1\mathcal{L}_{3D}^t + \lambda_2\mathcal{L}_{2D}^t + \lambda_3\mathcal{L}_{MANO}^t\big).$$

Each term checks the prediction against ground truth in a different space, and the weights $\lambda_1 = 0.05$, $\lambda_2 = 0.01$ and $\lambda_3 = 0.001$ set how much each contributes to the gradient. Because the sum runs over all frames of the clip, the temporal modules IAM and PAM receive gradient from every frame they influence, which is what lets them learn to share information along the sequence.

The first term compares 3D joint positions:

$$\mathcal{L}_{3D}^t = \|J_{3D}^t - \tilde{J}_{3D}^t\|_1.$$

Here $J_{3D}^t$ are the ground-truth 3D hand joints at frame $t$, and $\tilde{J}_{3D}^t$ are the joints of the MANO mesh built from the network's predicted parameters. This term asks whether the predicted hand has its knuckles and fingertips in the right places in 3D. Using an L1 norm instead of a squared error matters in practice. With an L1 penalty, the gradient from one badly wrong joint has bounded size, so an occasional bad annotation or a frame with a severely occluded fingertip pulls on the network no harder than an ordinary error. A squared loss would let such outliers dominate the update.

The second term does the same in the image:

$$\mathcal{L}_{2D}^t = \|J_{2D}^t - \tilde{J}_{2D}^t\|_1.$$

The predicted 3D joints are projected into the image with the camera model, and $\tilde{J}_{2D}^t$ is compared with the annotated 2D keypoints $J_{2D}^t$. A 3D loss alone can be satisfied by a hand that has the right shape but sits slightly off the pixels; the 2D term keeps the reconstructed mesh aligned with what the camera actually saw. That alignment pays off twice in HaWoR, because the hand masks fed to SLAM are made by projecting these meshes. A mesh that drifts off the hand pixels would leave real hand pixels unmasked.

The third term supervises the MANO parameters directly:

$$\mathcal{L}_{MANO}^t = \|\Theta_t - \tilde{\Theta}_t\|_2^2 + \|\beta_t - \tilde{\beta}_t\|_2^2.$$

Joint positions do not pin down every MANO parameter. Twisting a finger about its own long axis, for example, barely moves the joint centers, so joint losses alone leave such parameters loosely constrained. Matching the pose $\Theta_t$ and shape $\beta_t$ directly closes that gap and keeps the predictions in a plausible part of MANO's parameter space.

It is tempting to read the weights as a ranking of importance, with 3D joints fifty times more important than MANO parameters. That reading is unsafe. The three terms are measured in different units (3D distances, pixel distances, axis-angle radians and shape coefficients), so their raw magnitudes differ and the weights partly just balance those scales. The paper reports the values but does not derive them, so treat them as tuned hyperparameters.

The optimization details are conventional: AdamW with learning rate $10^{-5}$, 250K iterations, batch size 32, four NVIDIA A800 GPUs, about two days. The small learning rate fits the setup, since the ViT backbone stays frozen and the new temporal modules are trained on top of features that already work well per frame. With a trained $\mathcal{M}$, we have camera-frame hands and, by projecting them, the masks $M_t$ that the camera branch needs.

To see why hands disturb camera estimation, it helps to know roughly what DROID-SLAM does. For pairs of frames, a learned network predicts dense correspondences: for each pixel in one frame, where the same scene point appears in the other. Alongside each correspondence it predicts a confidence weight, collected per frame as $w_t$. A dense bundle adjustment (DBA) layer then solves for the camera poses $R_{c_t}, t_{c_t}$ and per-pixel depths $d_t$ that make these correspondences geometrically consistent. Concretely, it minimizes a reprojection error. A pixel's depth and the relative camera motion predict where that pixel should land in the other frame. The error is the distance from that predicted location to the location the network observed, weighted by $w_t$. Pixels with high confidence pull hard on the solution, and pixels with low confidence barely matter.

That optimization rests on a static-world assumption. A correspondence is only explained by camera motion if the scene point itself stayed put. In a typical third-person video the moving person occupies a modest part of the frame, and the static background dominates. Egocentric video breaks this balance. As the wearer reaches for the kettle, the hand can fill a large part of the image, and it moves independently of the head. Its pixels produce correspondences that are perfectly real but describe hand motion, not camera motion. DBA cannot tell the difference, so it bends the camera estimate to partly explain them. If the hand sweeps right while the head is still, the solver concludes that the camera turned a little left.

HaWoR's fix uses the fact that it already knows where the hands are. The camera-frame meshes from $\mathcal{M}$ are projected into each image to form a binary mask $M_t$, equal to 1 on hand pixels and 0 elsewhere. Following TRAM, the mask is applied in two places, which is why the paper calls it dual masking:

$$\hat{I}_t = (1 - M_t)\cdot I_t, \qquad \hat{w}_t = (1 - M_t)\cdot w_t.$$

The first equation blanks out hand pixels in the image $I_t$ before DROID-SLAM extracts features, so the correspondence network never matches hand texture between frames. The second multiplies the confidence map by the same complement, so any correspondence that starts on a hand pixel enters DBA with weight exactly zero. Both are needed. Masking only the image leaves the network free to assign confident correspondences near the blanked region, where blurred hand edges can leak through. Masking only the confidences still lets hand appearance shape the features that background correspondences are computed from. Together they ensure that only background pixels decide the camera motion.

In the kettle example, this matters most in the moments just after the wearer picks up the kettle. The hand and kettle are close to the camera, large in the frame and moving fast. With masking, the head's rotation as the wearer turns around is estimated from the counters, cabinets and walls, which are the things that actually stay still.

The mask is only as good as the hand network behind it. If $\mathcal{M}$ misses a hand or places it a few pixels off, some hand pixels remain unmasked and keep their influence. This is one reason the 2D joint loss from the previous section matters downstream. There is also a limit to what masking can achieve. It removes a bias in the trajectory's shape, but DROID-SLAM's poses and depths are still only defined up to an unknown scale. Masking makes the turn come out right as a shape; it does not tell us whether the wearer's head moved 20 cm or 2 m. That is the job of the next step.

Masked DROID-SLAM produces, for each keyframe $t$, a depth $d_t(p)$ at every pixel $p$, but in arbitrary units. If the true depth of a countertop pixel is 0.8 m, SLAM might report 2.3, or 0.04; all that matters to SLAM is that depths and translations are mutually consistent. A single unknown number $\alpha$ converts all of them to meters at once. If we can find it, multiplying every SLAM translation by $\alpha$ gives the metric camera trajectory.

To find $\alpha$, HaWoR borrows absolute scale from a different source. Metric3D is a monocular depth network trained on large and varied datasets to predict depth in meters from a single image. Following TRAM, HaWoR runs it on each SLAM keyframe to get a metric depth map $D_t(p)$. Now there are two depth maps of the same image: one in meters but computed from a single frame and possibly noisy, the other geometrically consistent across the video but unscaled. If both were perfect, every pixel would satisfy $D_t(p) = \alpha\, d_t(p)$, and plotting $D_t$ against $d_t$ would give a straight line through the origin whose slope is $\alpha$.

The obvious approach is least squares: choose $\alpha$ to minimize $\sum_p (D_t(p) - \alpha d_t(p))^2$ over all pixels. In egocentric video this is fragile for two reasons. Hand pixels are wrong for both maps. SLAM has no reliable depth there, because they were masked out of bundle adjustment, and the hand is very close to the camera. The second reason is subtler. Metric depth networks are least reliable at the extremes of their range: very close to the camera, where an egocentric view often sees the hands, a held object or the near edge of a counter, and very far away, such as a window or a distant wall. Under a squared loss, a few such pixels with large residuals can tilt the fitted slope substantially.

HaWoR's adaptive sampling module (AdaSM) handles both problems by choosing which pixels take part in the fit:

$$S_t = \{\, p \mid p \notin M_t,\ D_{min} < D_t(p) < D_{max} \,\}.$$

A pixel enters the sampled set $S_t$ only if it is not on a hand, according to the same mask $M_t$ used for SLAM, and if its metric depth lies inside a trusted interval $(D_{min}, D_{max})$. The bounds are not guessed; the paper chooses them by optimizing scale accuracy on its egocentric training data. In the kettle scene, this keeps pixels on the counter, the cabinets and nearby walls, and drops the hand, the kettle when it is held close to the face, and anything visible through a far doorway.

Even inside $S_t$ some pixels will disagree, for example on a reflective kettle body or a textureless cabinet door. HaWoR therefore replaces the squared error with a robust penalty:

$$E(\alpha) = \sum_{p \in S_t} \rho_{GM}\big(D_t(p) - \alpha\, d_t(p)\big), \qquad \rho_{GM}(x) = \frac{x^2}{x^2 + \sigma^2}.$$

The Geman-McClure function $\rho_{GM}$ behaves like a scaled square for small residuals, so pixels that agree with the line still pull $\alpha$ precisely into place. As the residual grows past roughly $\sigma$, the penalty levels off toward 1. A pixel whose residual is enormous therefore adds almost no more cost than one that is merely wrong, and its pull on $\alpha$ fades instead of growing. In the scatter-plot picture, the robust fit follows the dense band of inliers while the outliers sit off the line without tilting it.

Once $\alpha$ is fitted, every SLAM translation is scaled by it to give metric $t_{c_t}$. Together with the masked rotations $R_{c_t}$, this is the camera half of the world-space composition. Two caveats belong next to this result. The scale is only as good as Metric3D's predictions inside the trusted range, so a scene dominated by unusual geometry or materials can still mislead it. And because one $\alpha$ is fitted for the sequence, the method assumes SLAM's internal scale stays consistent over the clip. When those conditions hold, the reach toward the kettle can finally be measured in centimeters rather than arbitrary units.

With metric camera poses in hand, we could compose every visible frame into world space and be done, except for the frames where the hand was never seen. In the kettle video the right hand drops out of view for a stretch during the turn. Something must guess what it did there, and the frame of reference in which we ask for that guess matters a great deal.

Consider asking a network to fill the gap in camera-frame coordinates. Before the gap, the hand is at the right of the image. After it, the hand reappears somewhere else because the head has turned roughly half a circle. The camera-frame trajectory across the gap mixes the hand's real motion with the head's rotation, and the head's contribution differs in every training sequence. A network would have to learn hand motion and head motion together, from data in which the two are tangled differently each time. Raw world coordinates have the opposite problem. They are free of head motion, but each sequence starts at an arbitrary position and orientation in some room, so the same reach looks numerically different depending on where the person happened to stand.

HaWoR's answer is a canonical frame for each hand's sequence, anchored to the hand itself at the first frame. In this frame the hand at $t = 0$ sits at the origin with identity orientation, and every later frame is expressed relative to that starting pose. To build it, first write the hand's world pose at frame 0 using the composition from the first section and the metric camera from the previous step. Its world orientation is $R_{c_0}\Phi_0^{c_0,i}$ and its world position is $R_{c_0}\Gamma_0^{c_0,i} + t_{c_0}$. A point $x$ in camera frame $c_t$ maps to world as $R_{c_t}x + t_{c_t}$. To express that world point relative to the frame-0 hand, subtract the frame-0 hand position and undo the frame-0 hand rotation. Collecting terms gives the camera-to-canonical transform $T^{c_t2cano,i} = [R^{c_t2cano,i} \mid t^{c_t2cano,i}]$:

$$R^{c_t2cano,i} = (R_{c_0}\Phi_0^{c_0,i})^{-1} R_{c_t}, \qquad t^{c_t2cano,i} = (R_{c_0}\Phi_0^{c_0,i})^{-1}\big(t_{c_t} - t_{c_0} - R_{c_0}\Gamma_0^{c_0,i}\big).$$

Applying it to the camera-frame hand at frame $t$ gives the canonical orientation and translation:

$$\Phi_t^{cano,i} = R^{c_t2cano,i}\,\Phi_t^{c_t,i}, \qquad \Gamma_t^{cano,i} = R^{c_t2cano,i}\,\Gamma_t^{c_t,i} + t^{c_t2cano,i}.$$

It is worth checking that this does what we claimed at $t = 0$. For the orientation,

$$\Phi_0^{cano,i} = (\Phi_0^{c_0,i})^{-1}R_{c_0}^{-1}R_{c_0}\Phi_0^{c_0,i} = I.$$

For the translation, $t_{c_0} - t_{c_0}$ cancels, leaving

$$\Gamma_0^{cano,i} = (R_{c_0}\Phi_0^{c_0,i})^{-1}\big(R_{c_0}\Gamma_0^{c_0,i} - R_{c_0}\Gamma_0^{c_0,i}\big) = 0.$$

Every sequence therefore starts from exactly the same state, whatever the person's head was doing and wherever they stood.

Now return to the kettle. In canonical space, the trajectory of the right hand reads as "starting from here, the hand moved forward 25 cm, rotated its wrist as it lifted, swung to the left...". Since $R_{c_t}$ and $t_{c_t}$ enter every frame's transform, the head's rotation during the turn has been removed. What remains is hand motion in a standard starting pose, which is a much more regular object to learn about. A reach looks like a reach no matter which way the person faced when it began.

The transform back is just the frame-0 hand's world pose, $T^{cano2w,i} = [R_{c_0}\Phi_0^{c_0,i} \mid R_{c_0}\Gamma_0^{c_0,i} + t_{c_0}]$, which the infiller's output passes through to land in world coordinates. One dependency is worth stating plainly. Both directions are built from the metric camera poses, so the canonical sequence is only as clean as SLAM and the scale fit made it. A residual scale error appears as a stretched canonical trajectory, and the infiller sees that stretched trajectory as its input.

In canonical space, the missing stretch of the kettle trajectory becomes a familiar problem from character animation: motion in-betweening. You know where the hand was before the gap and where it reappeared after, and you need plausible frames in between. HaWoR builds its infiller $\mathcal{F}$ in the style of a conditional motion in-betweening model. It is a transformer encoder that reads the whole sequence, including the gap, and outputs a complete sequence.

The input is hand $i$'s $T$-frame MANO sequence from $\mathcal{M}$, with missing frames initially set to zero, along with a record of which frames were visible. Before the network sees anything, two preparation steps happen. First, every frame is moved into canonical space with $T^{c_t2cano,i}$ from the previous section. Second, the missing frames are filled with a cheap initial guess by interpolating between the nearest visible frames on either side. Translation $\Gamma$ and shape $\beta$ are vectors in ordinary Euclidean space, so linear interpolation (LERP) is appropriate for them. Orientation $\Phi$ and the per-joint rotations in $\Theta$ are rotations, and linearly blending rotation parameters can pass through implausible intermediate orientations. They are interpolated with spherical linear interpolation (SLERP), which moves along the shortest arc between two rotations at constant angular speed.

The paper's reason for the initialization is practical: the network only has to learn a correction to a reasonable guess instead of producing motion from nothing. In the kettle example, interpolation alone would move the hand along a smooth path from where it left the frame to where it came back. That path is roughly right, but real hand motion during a turn and pour is not a constant-speed glide, and the wrist rotates as the kettle tips. The transformer's job is to turn that straight-line guess into something that moves like a hand.

The network itself is compact. Each frame's MANO vector is projected by a linear layer to a token of dimension $d_{model}$, and a positional embedding is added, because self-attention by itself ignores order and the model needs to know which frame comes when. The tokens pass through a stack of multi-head self-attention layers, so every frame, visible or not, can draw on every other frame in the sequence. A fully connected decoder maps each output token back to MANO parameters in canonical space. Finally $T^{cano2w,i}$ maps the completed sequence into world coordinates:

function INFILL(sequence of MANO params, visible[t])
    canonicalize every frame with T^{c_t2cano,i}
    for each t with visible[t] = false:
        Γ_t, β_t ← LERP between nearest visible frames
        Φ_t, Θ_t ← SLERP between nearest visible frames
    tokens_t ← Linear(MANO vector_t) + PosEmb(t)        # dimension d_model
    tokens ← stacked multi-head self-attention(tokens)
    {Φ̂_t, Θ̂_t, β̂_t, Γ̂_t} ← FC(tokens_t)               # canonical space
    return T^{cano2w,i} applied to every frame
end function

Training needs sequences where the hand is out of view but its true motion is still known. HOT3D provides this. It was recorded with Aria glasses alongside third-person cameras and has MANO annotations in world coordinates, including for frames where the hand is outside the egocentric camera's field of view. HaWoR trains on those real out-of-view segments. To get more data, it also takes ordinary sequences and randomly masks continuous runs of middle frames, always keeping the start and end frames as context, which mirrors the test-time situation of a gap bounded by visible frames. The paper uses 110 HOT3D videos for training and 27 for validation.

The loss compares the infiller's outputs with ground truth on all four MANO quantities, using L1 distances:

$$\mathcal{L}_{\mathcal{F}} = \sum_t \big(\gamma_1\|\Gamma_t - \hat{\Gamma}_t\|_1 + \gamma_2\|\Phi_t - \hat{\Phi}_t\|_1 + \gamma_3\|\Theta_t - \hat{\Theta}_t\|_1 + \gamma_4\|\beta_t - \hat{\beta}_t\|_1\big),$$

with $\gamma_1 = 0.05$ for translation, $\gamma_2 = 2.0$ for orientation, $\gamma_3 = 2.0$ for pose and $\gamma_4 = 0.05$ for shape. As with the hand network, these weights also absorb unit differences, so they are best read as tuned values, not a ranking. Training uses AdamW with an initial learning rate of $10^{-4}$, decayed by a factor of 0.9 every 100 steps, for 1500K iterations at batch size 32 on a single A800, which takes about a day.

One boundary is worth stating. The infiller produces plausible motion consistent with the frames around the gap and with what it learned from HOT3D. It does not observe the hand during the gap. For the kettle pour, if the person did something unusual out of view, such as a quick wobble or a regrip, the infiller has no evidence of it and will produce a typical motion instead.

Every piece is now in place, and writing the whole method as one procedure makes the ordering constraints explicit:

function HAWOR(V)
    tracks ← BoT-SORT(WiLoR-detect(V))                   # per-hand box sequences
    for each hand i:
        {Θ̃_t, β̃_t, Φ_t^{c_t}, Γ_t^{c_t}} ← M(crops_i)      # camera frame, with gaps
    M_t ← project all reconstructed hands into frame t
    R_{c_t}, t_{c_t}, d_t ← DROID-SLAM(Î_t, ŵ_t)          # masked, up to scale
    D_t ← Metric3D(keyframe t)
    S_t ← {p ∉ M_t, D_min < D_t(p) < D_max}
    α ← argmin over α of Σ_{p∈S_t} ρ_GM(D_t(p) − α d_t(p))
    t_{c_t} ← α · t_{c_t}                                 # metric camera
    for each hand i:
        {Θ̂_t, β̂_t, Φ̂_t, Γ̂_t} ← INFILL(hand i sequence)   # canonical → world
    return world hands, {R_{c_t}, t_{c_t}}
end function

Read it as three blocks, each consuming the output of the one before. The first block runs entirely in the camera frame. Detection and tracking split the video into per-hand crop sequences, and $\mathcal{M}$ turns each sequence into MANO parameters relative to the camera, leaving gaps where the hand was not visible. Nothing here knows how the head moved, and nothing needs to.

The second block converts the video into a metric camera trajectory, and it depends on the first block in a specific way: the masks $M_t$ come from projecting the hands that $\mathcal{M}$ just reconstructed. Masked DROID-SLAM then estimates rotations, unscaled translations and depths from background pixels only. Metric3D supplies metric depths, AdaSM picks the trustworthy pixels, and the scale is the minimizer of the robust energy:

$$\alpha^* = \arg\min_{\alpha} \sum_{p \in S_t} \rho_{GM}\big(D_t(p) - \alpha\, d_t(p)\big).$$

This is a one-dimensional robust regression, so it is cheap to solve. Once found, it rescales every translation.

The third block needs both of the others. Each hand's camera-frame sequence is moved into its canonical frame using the metric camera poses, the infiller completes the missing frames, and the frame-0 world pose of the hand maps everything back into world coordinates. In the kettle video, this is where the gap during the turn is filled and the whole right-hand trajectory lands in meters, relative to the kitchen.

Notice what the procedure does not contain: an iterative optimization that jointly adjusts hands, cameras and motion priors until they agree. Optimization-based approaches such as SLAHMR (and the HMP-SLAM baseline the paper builds in that style) refine all of these together over many iterations. HaWoR instead runs its learned modules forward, with the scale fit as its only optimization, and a cheap one. The paper reports the consequence in runtime. HaWoR needs a single forward pass of about 40 ms per frame, against about 160 ms per frame for HMP-SLAM, a 75% reduction.

That figure deserves two qualifications. First, 40 ms per frame is roughly 25 frames per second, slower than a typical 30 fps camera, and the paper itself says the runtime is still far from real time. Second, the pipeline as written is a batch procedure. Bundle adjustment works over keyframes, the scale is fitted for the sequence, and the infiller attends across the whole gap, including frames after it. So HaWoR in this form suits offline processing of recorded egocentric video, such as building datasets of human demonstrations, more than live teleoperation. The paper names foundation models that estimate world-space cameras directly as a possible route toward real-time operation.

The modular structure also gives a clear way to evaluate the method. Each block produces an intermediate result that can be scored on its own: camera-frame hands against hand benchmarks, camera trajectories against ground-truth poses, and world-space hands against world-space annotations. The paper's experiments follow this structure.

The first test isolates the hand network. DexYCB is a dataset of hands grasping objects, filmed by static cameras, with annotations of which frames are occluded. Since the cameras do not move, world placement is not an issue here, and the benchmark measures camera-frame hand reconstruction directly. The metric is PA-MPJPE: the mean distance in millimeters between predicted and ground-truth joints after a Procrustes alignment removes global rotation, translation and scale. What remains is error in the hand's articulation and shape. AUC is the area under the curve of the fraction of joints within a distance threshold, so higher is better.

Method	All	50–75% occluded	75–100% occluded
WiLoR (single image)	5.01 / 90.0	5.42 / 89.2	5.68 / 88.7
Deformer (video)	5.22 / 89.6	5.70 / 88.6	6.34 / 87.3
HaWoR	4.76 / 90.5	5.03 / 89.9	5.07 / 89.9

The entries are PA-MPJPE in mm and AUC. On the full test set HaWoR is ahead of WiLoR by about a quarter of a millimeter. The more telling pattern is how each method degrades as occlusion increases. WiLoR's error rises from 5.01 to 5.68 mm between all frames and the most occluded bucket. Deformer, a video-based method, rises further, to 6.34. HaWoR's error barely moves, reaching 5.07 at 75–100% occlusion, where its advantage over WiLoR grows to about 0.6 mm. Since HaWoR starts from WiLoR's own backbone, the gain has to come from what was added on top, and the pattern matches what IAM and PAM were designed for: borrowing evidence from neighbouring frames when the current frame shows little of the hand. This table does not isolate the two modules; the ablations later do.

The second test isolates the camera. On HOT3D, which provides ground-truth camera trajectories from Aria glasses, the paper reports two trajectory errors. ATE (absolute trajectory error) first aligns the estimated trajectory to ground truth with a transform that includes the best possible scale, so it measures only whether the shape of the path is right. ATE-S uses HaWoR's own estimated scale, so it measures shape and metric size together. The paper reports both in millimeters, and splits ATE-S by how far the camera traveled (short, medium and long displacement).

Method	ATE	ATE-S short / medium / long / all
DROID + Metric3DV2	3.80	14.28 / 21.56 / 29.10 / 21.07
HaWoR w/o AdaSM	3.36	– / – / – / 20.97
HaWoR	3.36	9.31 / 15.86 / 19.26 / 14.61

Reading the table means comparing pairs of rows that differ by a single change. Plain DROID-SLAM plus Metric3D (in the TRAM style) and HaWoR without AdaSM differ mainly in the hand masking, and ATE drops from 3.80 to 3.36. That is the shape improvement promised by keeping hands out of bundle adjustment. Their ATE-S values, however, are nearly identical (21.07 against 20.97), because masking does nothing to fix scale. HaWoR with and without AdaSM have the same ATE, as expected since AdaSM only changes $\alpha$, but ATE-S drops from 20.97 to 14.61, a reduction of about 30%.

Two further observations come from this table. First, look at the magnitudes. The ATE-S values are four to six times the ATE values, so for egocentric camera trajectories, getting the scale right is the larger problem by a wide margin. Second, the improvement holds across all displacement ranges, and the gap is widest for long trajectories (29.10 against 19.26). The paper also compares other metric depth networks in the same role: ZoeDepth gives an ATE-S of 43.58 and DepthAnythingV2 27.49 with plain DROID-SLAM, both worse than Metric3DV2, which supports the choice of Metric3D as the source of scale.

For the kettle example, these two tables cover the first two ingredients separately. The hand network should keep the right hand's shape stable even as it slides to the image edge during the turn, and the camera branch should get the head's turn right in both shape and size. What these tables do not yet show is whether the composition of the two, plus the infiller, produces a good world-space hand.

Because HaWoR is the first method aimed at this exact task, the paper has to build its own baselines. It takes strong camera-frame hand reconstructors (HaMeR, WiLoR and HandDGP), combines each with DROID-SLAM camera trajectories, and composes the two exactly as in the first section's equations. It also builds HMP-SLAM, an optimization-based baseline in the style of SLAHMR. HMP-SLAM combines DROID-SLAM with HMP, a learned hand motion prior, and optimizes the hands to agree with both the camera trajectory and the prior. This gives a fair picture of what the obvious alternatives achieve on HOT3D egocentric sequences.

The metrics test different failure modes, so it helps to know what each one punishes before reading the numbers. All are computed on 100-frame segments.

• PA-MPJPE (mm) aligns each frame's predicted hand to ground truth before measuring, so it only sees articulation and shape errors.
• W-MPJPE (mm) aligns each segment only at its first frame and then measures joint error in world coordinates for the rest. Any drift in hand or camera position accumulates into it.
• WA-MPJPE (mm) aligns the whole segment at once, which forgives a consistent offset but still punishes a trajectory with the wrong shape.
• RTE (%) is the root translation error as a percentage of how far the hand actually moved.
• Accel (m/s²) compares accelerations, so it rises with jitter.

Method	PA-MPJPE	W-MPJPE	WA-MPJPE	RTE	Accel
HaMeR-SLAM	9.39	156.03	43.37	4.77	19.25
HandDGP-SLAM	17.88	154.30	42.93	3.18	20.17
WiLoR-SLAM	6.00	151.67	39.49	2.99	8.02
HMP-SLAM	10.51	119.41	39.46	2.79	5.50
HaWoR	4.79	33.20	11.27	0.78	5.41

The W-MPJPE column carries the headline. The three regression baselines cluster around 150 mm, and the optimization baseline brings that down to 119 mm. HaWoR is at 33 mm, about 3.6 times lower than the best baseline. Since W-MPJPE is the metric most sensitive to drift, this is where the masked SLAM and the scale fit show up most clearly. A wrist that is 15 cm off after a few seconds is unusable for placing a hand on a kettle handle; one that is 3 cm off is in the right neighbourhood. WA-MPJPE tells a consistent story: about 11 mm against roughly 40 for every baseline, so even after the best rigid alignment the baselines' trajectories have the wrong shape. RTE falls from 2.79% to 0.78% of the hand's displacement.

The other two columns help separate causes. PA-MPJPE ranks WiLoR-SLAM second at 6.00 mm, consistent with WiLoR being a strong per-frame reconstructor. Yet WiLoR-SLAM's world-space error is no better than HaMeR-SLAM's, which shows that good per-frame hands do not produce good world trajectories on their own. HMP-SLAM is the reverse case. Its motion prior gives it smooth motion (Accel 5.50, close to HaWoR's 5.41) and the best world error among the baselines, but its per-frame articulation is worse (PA-MPJPE 10.51). HaWoR matches the optimization baseline's smoothness while being the most accurate on articulation, without running an optimization loop.

The paper's qualitative figures show the same thing on the kettle sequence. For the person who picks up the kettle, turns around and pours, the baseline trajectories scatter or sit offset from ground truth, while HaWoR's reconstructed right hand follows the ground-truth path through the turn. The paper also shows qualitative results on EPIC-KITCHENS videos, which HaWoR was not trained on, where it reconstructs plausible hands and cameras even when hands leave the frame.

One caveat about this comparison deserves to sit next to the numbers. HaWoR's hand network and infiller were both trained on HOT3D data, and the world-space evaluation is on held-out HOT3D sequences, so HaWoR sees test data from the same distribution it trained on. The baselines are off-the-shelf components assembled for this task. The large margins are therefore strong evidence that the design works on HOT3D-like egocentric video; the EPIC-KITCHENS results suggest generalization, but they are qualitative.

An ablation removes one component and measures what breaks. Before reading the tables, it is worth predicting from the design what each component should affect. PAM is a prior on pose sequences, so removing it should hurt smoothness (Accel) and accumulated drift (W-MPJPE) more than per-frame articulation (PA-MPJPE). The pretrained ViT supplies per-frame visual understanding, so removing it should hurt everything, PA-MPJPE included. The infiller only acts where the hand is missing, so its effect should show up on out-of-view segments.

The first ablation varies the hand network inside the full HaWoR pipeline and reports the same world-space metrics as before:

Variant	PA-MPJPE	W-MPJPE	WA-MPJPE	RTE	Accel
w/o pretrained ViT	7.59	86.80	19.46	1.26	9.09
w/o IAM & PAM	5.07	44.60	13.85	0.93	8.42
w/o PAM	4.80	36.32	12.40	0.88	6.03
Full	4.79	33.20	11.27	0.78	5.41

Read from the bottom up. Removing PAM leaves PA-MPJPE essentially unchanged, 4.80 against 4.79, yet Accel rises from 5.41 to 6.03 and W-MPJPE from 33.20 to 36.32. That is exactly the motion-prior signature: per-frame shapes are just as good, but the sequence is jerkier and drifts more. Removing IAM as well raises PA-MPJPE to 5.07 and W-MPJPE to 44.60, a jump that fits IAM's role in rescuing truncated and occluded frames, which in egocentric video are common enough to move the averages. Training without the pretrained ViT is the largest single hit, with PA-MPJPE at 7.59 and W-MPJPE at 86.80. Most of HaWoR's per-frame quality comes from starting on WiLoR's backbone.

There is a second lesson hidden in the middle row. Even without IAM and PAM, the hand network is essentially per-frame, yet the pipeline scores 44.60 mm on W-MPJPE. The WiLoR-SLAM baseline in the previous section, also per-frame hands on DROID-SLAM, scored 151.67 mm. The ablation is not a controlled comparison with that baseline (the network here was still trained on HaWoR's data), but the size of the gap suggests that much of HaWoR's world-space advantage comes from the camera side and the infiller, not only from the temporal modules in the hand network.

The second ablation targets the infiller, evaluated only on HOT3D segments where the hand is genuinely out of view. It compares two simple strategies against the learned infiller: freezing the last visible pose for the whole gap, and the LERP/SLERP interpolation that the infiller itself uses as its starting guess. A new metric enters here. FID (Fréchet Inception Distance, computed here over MANO parameters) compares the distribution of generated poses with the distribution of real ones; lower means the filled-in motion looks more like real hand motion.

Fill strategy	FID	PA-MPJPE	W-MPJPE	WA-MPJPE	RTE
Last pose	1.52	7.83	116.79	78.78	13.04
LERP/SLERP	1.42	6.33	75.01	49.16	9.39
Infiller	0.57	6.22	66.25	37.22	7.41

Freezing the last pose is clearly bad, since a hand that stops dead and then teleports produces large world errors. Interpolation fixes much of that, cutting W-MPJPE from about 117 to 75 mm. The learned infiller improves every metric again, but unevenly. On PA-MPJPE it barely beats interpolation (6.22 against 6.33), while FID drops from 1.42 to 0.57. The infiller's main contribution is making the hidden stretch move like a real hand instead of a constant-speed glide, together with a moderate improvement in where the hand goes (W-MPJPE 75 to 66, WA-MPJPE 49 to 37).

Notice also the absolute numbers. On invisible segments the infiller's W-MPJPE of 66 mm is about double the full-pipeline figure of 33 mm from the main table. That is expected, since these are the hardest frames, but it gives a concrete sense of reliability: infilled frames are noticeably less certain than observed ones. For the kettle sequence, the part of the pour that happened outside the camera's view is HaWoR's best guess, about twice as uncertain as the parts it saw.

Run the kettle video through HaWoR once more, this time keeping track of which module handles which difficulty. As the wearer grabs the kettle, the hand fills much of the frame. Dual masking keeps those pixels out of DROID-SLAM, so the head's turn is estimated from the counters and walls. The turn comes out with the right shape, and the robust scale fit against Metric3D depths makes it the right size, so the reach and the pour are measured in centimeters. During the turn the right hand slides to the image edge, and IAM and PAM keep its shape stable as it truncates. Then it leaves the view entirely, and the infiller bridges the gap in a canonical frame anchored to the hand's own starting pose. The result is the sequence the first section asked for,

$$\{\hat{\Theta}_t^i, \hat{\beta}_t^i, \hat{\Phi}_t^i, \hat{\Gamma}_t^i\}_{t=0}^{T},$$

in world coordinates and in meters, for every frame.

The paper is direct about where this pipeline breaks, and each limitation follows from a design decision covered above. First, HaWoR relies on off-the-shelf detection and tracking. If the tracker swaps left- and right-hand identities, for example when the hands cross or one briefly disappears, $\mathcal{M}$ receives crops of the wrong hand under the wrong label. Those errors flow into the masks, the infiller and the world trajectory, and the paper shows such failures. Second, each hand is modelled independently, with no constraint that the two hands not intersect. When the hands interact closely, as in washing hands or a two-handed grasp, the reconstructed meshes can interpenetrate. Third, the runtime of about 40 ms per frame is roughly four times faster than optimization-based reconstruction but, in the paper's words, still far from real time. The paper suggests foundation models that estimate world-space cameras directly as a path to faster inference, and modelling two-hand interaction to address interpenetration.

For robotics, it helps to be clear about what the output offers. What follows is our framing of the use cases, not claims made in the paper. Methods that learn manipulation from human video, and pipelines that retarget human demonstrations onto robot hands, need roughly three things: where the wrist went in a fixed frame ($\hat{\Gamma}_t^i$), how it was oriented ($\hat{\Phi}_t^i$), and what the fingers were doing ($\hat{\Theta}_t^i$), all on a metric scale that matches the robot's workspace. HaWoR produces exactly these from a single head-mounted camera, which makes ordinary egocentric recordings, including ones like EPIC-KITCHENS where the paper shows qualitative results, a potential source of demonstrations.

The evidence also suggests how much to trust each part of a reconstructed clip. Observed frames on HOT3D-like video carry world joint errors of a few centimeters over 100-frame segments. Infilled frames are noticeably less certain, with roughly double the world error in the paper's ablation, and they show typical motion, not whatever the person actually did out of view. A practical consumer of HaWoR output would therefore check each clip for left/right identity swaps (for example, a sudden jump in a wrist's world position), for hand-hand interpenetration during bimanual contact, and would mark infilled segments so they can be weighted down or excluded when precise contact matters. Scenes very different from HOT3D, or where Metric3D's depth is unreliable, deserve the same caution about scale.

The overall result is a decomposition that makes a hard problem tractable. A world-space hand is a camera-frame hand composed with a camera pose. HaWoR gets the first factor from a temporally aware hand network, the second from SLAM kept away from the hands and anchored to meters, and fills the gaps in a frame where the camera's motion no longer appears. Each piece can be checked on its own, and that is also how you would judge whether a particular reconstruction is good enough for the robot that will learn from it.