Training Generalist Navigation Policies via Action Self-Distillation

TL;DR: A VLM teaches itself to navigate from language by explaining recorded demonstrations — no human goal or reasoning labels needed.

13real-world tasks, zero-shot
~100 hof navigation data (vs. 9,500 h for OmniVLA)
95%outdoor success
80%ablation success vs. 36% for SFT on Claude Opus 5 labels

Overview

Highlights

Self-distillation from trajectory-only data. One open-source VLM serves as both teacher and student. During training, the teacher sees the demonstrated trajectory and infers the navigation goal and a reasoning trace. The student learns to produce the same reasoning and trajectory from the camera image and the goal. No human-written goals or reasoning traces are used.
Steerable navigation with structured reasoning. In the same scene, the policy reaches different targets when given different directional, color, or object commands. Each action comes with a short trace covering perception, prediction, and planning. Training uses about 100 hours of navigation data, compared with 9,500 hours for OmniVLA.
Zero-shot real-world generalization. We evaluate on 13 real-world tasks with a FrodoBot rover. Although the training data is mostly outdoor, the policy handles unseen indoor scenes and multi-stage instructions. It improves success by 30 percentage points over OmniVLA indoors and over NaVILA on multi-stage tasks, and reaches 95% success outdoors.
Outperforms SFT on frontier-model labels. Fine-tuning on reasoning labels from Claude Opus 5 reaches 36% success on our ablation tasks; self-distillation reaches 80%. On the Left chair task, ASD finishes in 15 seconds, while prompting Claude Opus 4.7 to drive takes over 14 minutes.

How it works

Method

One open-source VLM plays both roles. Training uses only recorded trajectories — no human-written goals or reasoning traces.

1 · Self-teacher

Frozen copy of the same VLM

Image + GT trajectory

VLM

Task Perception Agents Prediction Planning

Sees the camera image with the demonstrated (GT) trajectory overlaid, and writes the goal and a reasoning trace.

2 · Student

Same VLM, learns from the teacher

Image + goal only

VLM

Perception Agents Prediction Planning Trajectory

KL on reasoning tokens CE on GT trajectory tokens

Sees only the camera image and the goal, and learns to produce the same reasoning and trajectory.

3 · Deploy

Student only, no teacher

“Navigate to the chair”

Student VLM

Perception Prediction Planning Trajectory

The student alone steers the robot from a camera image and a language command, with a short trace for each action.

Diagram: the same VLM acts as student and self-teacher. Both see the camera input; only the self-teacher also receives privileged information. Their outputs are compared, and the resulting paths are drawn toward a red car. — One VLM, two roles: the self-teacher also receives privileged information (the demonstrated trajectory); the student does not.

Training pipeline. Top: the Self-Teacher VLM (Qwen3.5-VL-9B) takes the GT trajectory, a self-teacher prompt and the image with the path drawn on it, and outputs a goal, perception and plan. Bottom: the Student VLM (Qwen3.5-VL-9B) takes the raw image, a student prompt and the extracted goal, and outputs perception, plan and trajectory. A KL term matches the student's reasoning to the teacher's, a CE term fits its trajectory to the GT, and an EMA link connects the two VLMs. — Training overview. The self-teacher (top) sees the demonstrated (GT) trajectory and extracts the goal and reasoning; the student (bottom) sees only the image, its prompt, and the extracted goal.

Zero-shot on a FrodoBot rover

Real-World Experiments

Multi-stage Hallway (multi-stage)

Multi-stage Sidewalk (multi-stage)

Indoor Right chair

Indoor Monitor

Indoor Chair

Outdoor Statue

Outdoor Car

Outdoor Tree

Indoor and outdoor clips run ASD side by side with the best baseline; the prompt and success rate are shown in each clip. Click a clip to pause it.

Five time-lapse composites of the rover approaching boxed targets: a blue chair and a monitor indoors, and a red car, a statue and a tree outdoors. — Rollout composites on two indoor (left) and three outdoor (right) tasks; the box marks the target.

Frame sequences of the two multi-stage rollouts. Top: navigate through the door, turn right, then go along the hallway. Bottom: navigate right to the sidewalk, and navigate along the sidewalk. — Frames from the two multi-stage rollouts.

Interactive · pick a capability

Steerability

Same scene, different prompt — pick one to see the rollout.

Navigate to the black or blue chair: two composite rollouts, each ending at the boxed chair of that color. — All eight prompts at a glance — color, direction, and object — in the same scene.

Navigate to the chair on the left, middle or right: three composite rollouts, each ending at the boxed chair in that position. — All eight prompts at a glance — color, direction, and object — in the same scene.

Explainable actions

Reasoning

Each action comes with a short trace covering the task, perception, and planning.

Robot camera view of blue office chairs along a lab workbench; the leftmost chair is boxed and the predicted path curves left toward it.

Robot camera view of three blue office chairs in front of white cabinets; the rightmost chair is boxed and the predicted path curves right toward it.

Robot camera view of a parking lot with a red car boxed on the left; the predicted path bends left toward it.

Robot camera view across concrete slabs toward a brick building; a stone statue is boxed on the left and the predicted path curves left toward it.

Numbers

Results

Success Rate

Ablation

Full result tables

Indoor language-conditioned navigation

	Directions			Objects				Multi-stage
Method	Straight door⁻	Left chair⁺	Right chair^*	Blue chair⁻	Trash bin⁻	Chair⁺	Monitor^*	Doorway⁺	Hallway^*	Avg.
OmniVLA	0.8	0.9	0.2	0.7	1.0	0.4	0.2	0.7	0.0	0.54
NaVILA	0.3	0.2	0.1	0.4	0.1	0.3	0.2	0.3	0.5	0.27
Qwen2.5-VL-ASD	0.9	1.0	0.8	1.0	0.9	0.9	0.7	0.8	0.6	0.84
Qwen3.5-VL-ASD	1.0	1.0	0.9	1.0	1.0	1.0	0.9	0.9	0.8	0.94

Outdoor language-conditioned navigation

Method	Sidewalk	Statue	Tree	Car	Avg.
OmniVLA	1.0	0.6	1.0	1.0	0.90
NaVILA	0.9	0.1	0.0	0.3	0.33
Qwen2.5-VL-ASD	1.0	0.8	1.0	1.0	0.95
Qwen3.5-VL-ASD	1.0	0.9	1.0	1.0	0.98

Foundation models (Left chair task)

Method	Success ↑	Completion time (s) ↓
OmniVLA	0.9	25.3
NaVILA	0.2	28.9
Qwen2.5-VL (vanilla)	0.0	—
Claude Opus 4.7	0.7	846.1
Qwen2.5-VL-ASD	1.0	15.3

Ablations

Method	Right chair	Chair	Monitor	Hallway	Car	Avg.
ASD, goal-only supervision	0.4	0.2	0.2	0.1	0.4	0.26
ASD, reasoning off at inference	0.1	0.2	0.2	0.0	0.2	0.14
Claude Opus 5 labels + SFT	0.3	0.5	0.3	0.3	0.4	0.36
Qwen2.5-VL labels + SFT	0.4	0.3	0.3	0.0	0.1	0.22
NaVILA (RECON + FrodoBots-2K SFT)	0.0	0.1	0.2	0.3	0.2	0.16
Qwen2.5-VL-ASD	0.8	0.9	0.7	0.6	1.0	0.80
Qwen3.5-VL-ASD	0.9	1.0	0.9	0.8	1.0	0.92

Open-loop trajectory prediction (2 s horizon, lower is better)

	RECON				FrodoBots-2K
Method	ADE	ATE	MAE	MSE	ADE	ATE	MAE	MSE
Gaussian Policy	0.226	0.321	0.144	0.059	0.034	0.046	0.019	0.002
Gaussian Mixture	0.260	0.364	0.166	0.084	0.036	0.050	0.020	0.002
Diffusion Policy	0.180	0.238	0.108	0.045	0.034	0.045	0.020	0.003
Qwen2.5-VL + SFT	0.192	0.247	0.115	0.042	0.064	0.061	0.027	0.007
Alpamayo-1 + SFT	0.303	0.362	0.188	0.091	0.093	0.106	0.055	0.041
OpenVLA + SFT	0.341	0.417	0.208	0.114	0.246	0.269	0.138	0.072
Qwen2.5-VL-ASD	0.174	0.227	0.104	0.037	0.031	0.042	0.017	0.004

Cite

BibTeX

@inproceedings{feng2026asd,
    title     = {Training Generalist Navigation Policies via Action Self-Distillation},
    author    = {Feng, Bowen and Wang, Keqin and Yang, Yulong and Joshi, Amogh
                 and Allen-Blanchette, Christine and Shah, Dhruv and Heide, Felix},
    booktitle = {Conference on Robot Learning (CoRL)},
    year      = {2026},
}