Real-World Autonomous Execution

Videos shown at 2× speed.

Intro

A Shared Core Claim

WoVR+ retains WoVR’s core premise: controlling hallucination enables reliable policy optimization in imagination.

ACWMs may generate visually plausible yet physically incorrect rollouts, resulting in spurious success signals under the policy’s actions.

If world models inevitably hallucinate, how can RL remain reliable under imperfect imagined dynamics?

Additional Challenges in a Broader Setting

Modern VLAs require temporally, spatially, and cross-modally consistent multi-view observations and proprioception. Exploration-induced action jitter can degrade world-model predictions, while autoregressive errors accumulate over long-horizon, multi-stage tasks. Reliable WM-RL must therefore align the interaction interface, preserve effective exploration, and organize stage-wise learning.

Contributions

From WoVR to WoVR+

WoVR: The Shared WM-RL Framework

WoVR treats the world model as a generative simulator and controls hallucination at three complementary levels: simulator-level control, interaction-level reshaping, and policy-model alignment.

Extensions and Scope

WoVR+ preserves this reliability-oriented framework while redesigning its learned environment, exploration, and imagined interaction protocol for multi-view VLA policies and long-horizon, multi-stage manipulation.

• Extended and redesigned. WoVR+ retains first-frame anchoring and extends it to an anchored sparse history. The single-view, delta-EEF formulation with vector action conditioning is replaced by a VLA-aligned multi-view interface with absolute joint-and-gripper commands, geometry-grounded action and camera conditioning, and command-based proprioceptive updates.
• New. Chunk-wise Flow-SDE exploration preserves the marginal exploration scale while enforcing temporal coherence across neighboring absolute-action targets.
• Broadened in scope. WoVR+ extends KIR’s reset-based principle from task-critical regions to long-horizon, multi-stage optimization, combining data-supported stage-entry resets, stage-specific rewards, local rollout horizons, and controllable stage sampling.

Methodology

VLA-Aligned Multi-View World Model

• Geometry-grounded control. Absolute joint commands are projected into per-view action maps. Camera ray maps and view embeddings condition a parallel DiT control branch for synchronized head and wrist prediction.
• Closed-loop context. A persistent first-frame anchor and sparse recent history preserve scene context and task progress. Proprioceptive feedback is approximated from shifted joint-and-gripper commands.
• Robust and efficient rollouts. Geometry-aware context corruption and variable temporal strides prepare the model for recursive prediction, while DMD2 distillation enables one-step generation for higher-throughput imagined interaction.

Temporally Coherent Exploration

• Coherent perturbations. One noise vector is shared across temporal positions at a selected Flow-SDE step, avoiding independent perturbations of neighboring absolute-action targets.
• Preserved exploration scale. Each action token retains the same marginal Gaussian noise scale. The method reshapes temporal covariance instead of simply reducing exploration strength.
• Imagined policy improvement. The action chunks drive closed-loop VLA–world-model interaction, and grouped imagined returns provide advantages for GRPO policy optimization.

Stage-Local Imagination for Long-Horizon Manipulation

• Direct access to every stage. GRPO rollout groups start from recorded stage-entry states and observation histories. Balanced stage sampling provides training opportunities without replaying preceding skills.
• Local learning signals. Stage-specific completion rewards and short rollout horizons reduce reward sparsity and avoid long cross-stage generated prefixes. Steps after stage completion are masked.
• One end-to-end policy. All stages optimize the same VLA policy. Resets organize imagined training only; at deployment, the policy executes the complete task continuously without resets.

Results

Policy-Level Utility of the Learned Environment

Policy-Level Consistency

We evaluate two VLA policies on each of three tasks, with 120 real-robot and 120 imagined rollouts per task–policy pair. Each imagined rollout uses the initial images from its corresponding real-robot rollout. Success rates are strongly correlated across the six task–policy pairs (Pearson r = 0.970; Spearman ρ = 0.943).

Colors denote tasks; circles and squares distinguish the original and additional collections.

Synthetic-Data Utility

Using 500 environment rollouts to train the world model, we generate 512 synthetic trajectories from matched reset states. Success filtering retains 401 real and 480 synthetic trajectories for continued SFT. WM-BC improves success from 71.00% to 91.00% under fixed resets and from 72.66% to 76.56% under randomized resets. Filtered synthetic demonstrations provide useful but incomplete supervision, motivating direct WM-RL over both successful and failed rollouts.

Exploration in WM-RL

Success rates (%) with chunk-wise and i.i.d. Flow-SDE noise

Exploration	Updates	Adjust Bottle		Click Bell
		Fixed	Random	Fixed	Random
Initial SFT	0	79.00	70.70	71.00	72.66
Chunk-wise	10	76.00	76.95	87.00	81.25
Chunk-wise	20	80.00	80.86	89.00	86.72
Chunk-wise	30	85.00	86.33	87.00	86.33
Chunk-wise	40	86.00	88.28	88.00	89.45
Chunk-wise	50	93.00	91.80	92.00	93.75
I.I.D.	10	72.00	65.63	78.00	75.78
I.I.D.	20	73.00	77.34	71.00	68.36
I.I.D.	30	77.00	78.52	72.00	74.22
I.I.D.	40	70.00	68.36	81.00	74.61
I.I.D.	50	77.00	74.22	89.00	86.33

Fixed evaluation uses 100 shared initial states. The initial policy, world model, reward model, rollout budget, and GRPO configuration are held constant.

Chunk-wise exploration achieves higher final success rates in both tasks and both evaluation settings. At larger noise scales, it also produces broader executed EEF coverage, turning exploratory perturbations into coherent motion rather than high-frequency fluctuations.

Abstract

Action-conditioned world models can reduce costly real-robot interaction in reinforcement learning for Vision-Language-Action models (VLA-RL), but hallucinations can mislead policy optimization. Extending them to richer VLA interfaces and long-horizon tasks raises challenges in spatiotemporal and cross-modal consistency, exploration–reliability balance in RL, and compounding rollout errors.

We present WoVR+, extending WoVR’s hallucination-aware WM-RL framework by redesigning the learned environment and imagined interaction protocol: A VLA-interface-aligned world model provides multi-view prediction and proprioceptive updates while mitigating hallucinations. Chunk-wise noise preserves flow-based policy exploration while limiting prediction-degrading action jitter. For long-horizon tasks, stage-local imagined training combines stage-entry resets, stage-specific rewards, local horizons, and controllable sampling to shorten generated prefixes and balance training across stages.

WoVR+ improves success over SFT by 21.09% and 16.7% on RoboTwin and 6 real-robot tasks. Long-horizon phone-packaging success rises from 30% to 75%.

GT vs. WoVR+ World Model: Same-Action Replay

Ground-truth (GT) videos on the left; WoVR+ world-model predictions on the right, replaying the same recorded actions from the same initial observation.

Head, left wrist, and right wrist views, top to bottom. Videos shown at 2× speed.