Augmentic's first step toward photoreal robot simulation

Every robot shipping today is being trained and tested in a generic warehouse, a clean kitchen, lighting that never changes. Real sites have a ton of surprises.

At Augmentic we believe robotics companies can scale 10x faster when they can train and test with confidence inside their own sites, under every condition a robot might exist in: morning sun through the loading dock, the night shift under LEDs, the aisle that was rearranged last week.

That is why we are building Augmentic. We want creating a hyperrealistic, physically accurate digital twin of a real site to be as seamless as recording a video of it, so every release can be tested before it ships, every situation that has not happened can be generated, and every failure can be captured and replayed in simulation.

This post is the first step on the visual half of that promise. Here are the results of V0 of our photoreal renderer.

Introducing the renderer

⇆

Engine referenceOur output · 4 steps

Augmentic turns a phone scan of a real site into a digital twin that runs in Isaac Sim, MuJoCo or Gazebo. The physics engine owns the scene: geometry, objects, contacts, camera pose. What it cannot cheaply give you is the image a real camera would record there, under the lighting of that shift, with the glare, shadow and noise a perception stack has to survive. That is the job of our learned renderer.

The renderer reads the engine’s G-buffer, the per-pixel surface colour, normal, depth and roughness that any deferred rasteriser already produces, and generates the photoreal frame. It also reads its own previous frame, reprojected into the new camera view with the engine’s depth and pose, so appearance carries from frame to frame.

How it works

The simulator keeps control of everything that matters to the task. The renderer only learns appearance.

• The temporal path is the part we think is different in kind, not just in degree. Rather than hand the network the raw previous frame and let it learn where things moved, we warp the previous output into the new view using the depth and camera pose the engine already knows exactly.
• The network then sees previous appearance aligned with current geometry, and a per-pixel irradiance estimate that tells it how lit each surface was a frame ago. This is the same mechanism we use to measure temporal consistency below, which is why we can report a motion-compensated number.
• Training runs in two stages on one GPU.
  • Stage one learns G-buffer to image with the temporal channels zeroed, 15k steps. With no previous frame to lean on, the network has to learn lighting and materials from geometry alone, so the G-buffer stays its primary source of truth.
  • Stage two adds the temporal channels using the engine’s previous frame, 5k steps. The previous frame is dropped 10% of the time, so the network learns to use it for consistency without becoming dependent on it.
• The loss is a noise-prediction MSE plus a perceptual LPIPS term on a decoded crop. The training set is 2,400 frames from twelve scenes; two further scenes are used for validation and two held-out layouts for the evaluation below.

Measured results

We evaluated three things separately:

1. Similarity to the engine reference
2. Behaviour over long sequences
3. Inference time

The references are Unreal Engine 5.6 renders of two held-out warehouse layouts with their own exposure settings, 1,000 frames each, with a walking robot-height camera. Camera 1 is daylight, camera 2 is night. The first frame uses the engine’s frame 0 as context; every later frame uses only the renderer’s own previous output. Nothing else from the reference enters the loop.

Image fidelity across a full sequence

LPIPS measures differences in learned visual features (AlexNet backbone), PSNR measures pixel-level reconstruction, SSIM measures structure. All three are averaged over every one of the 1,000 frames, not a selection of stills.

1 step · 9.8 fps

4 steps · 5.7 fps

Engine reference

Temporal consistency, measured with motion compensation

Frame-to-frame flicker matters to a policy as much as per-frame fidelity does, and it is usually reported badly or not at all. A raw difference between consecutive frames mixes real camera motion with flicker. Because we already reproject the previous frame with exact depth and pose, we can separate them: we warp the previous output into the current view, compare it with the current output over the valid mask, and do the same for the engine reference. The ratio tells you how much more the rendered video changes than the scene itself does. At four steps, our output changes roughly six times more than the engine reference after motion is removed.

Inference cost on one GPU

Each value is the mean over 999 frames of a 1,000-frame sequence, with GPU execution synchronised before and after. The timing covers reprojection of the previous frame, the ControlNet and UNet passes for each step, and the VAE decode. It excludes loading the G-buffer from disk, metric computation and video writing. Hardware is a single RTX PRO 6000 Blackwell, bf16 for the networks, fp32 for the decoder.

The curve is roughly 0.09 s fixed plus 0.02 s per denoising step. The fixed part is dominated by the full-precision VAE decode and is the obvious next target: a half-precision or distilled decoder would about double the frame rate at every step count without touching the renderer’s weights.

See the output

Our output on the left, the engine reference on the right, both at 24 fps playback. Twelve seconds from the daylight sequence at four steps, starting three seconds in.

What needs to improve

Compared with FrameDiffuser

Fine detail and lighting both matter to a robot, and we are short on both.

Ours · frame 500

Reference

Ours · frame 1000

Reference

• Hallucination where the G-buffer is ambiguous. Flat ribbed surfaces, blank walls and floor paper get filled with the textures the model knows. More scene variety and self-conditioned training are the fix.
• Fine detail. Box labels, floor markings and small objects are softened. Part of this is a floor set by the Stable Diffusion VAE, which on our frames reconstructs at about LPIPS 0.05 even with a perfect latent.
• Temporal stability. Four to six times the reference’s motion-compensated change. We will report this number for every future version.

If you have a site and want a digital twin of it set up, book a call with us.