LFG: Learning to Drive is a Free Gift

Abstract

Ego-centric driving videos available online provide an abundant source of visual data for autonomous driving, yet their lack of annotations makes it difficult to learn representations that capture both semantic structure and 3D geometry. Recent advances in large feedforward spatial models demonstrate that point maps and ego-motion can be inferred in a single forward pass, suggesting a promising direction for scalable driving perception. We therefore propose a label-free, teacher-guided framework for learning autonomous driving representations directly from unposed videos.

Unlike prior self-supervised approaches that focus primarily on frame-to-frame consistency, we posit that safe and reactive driving depends critically on temporal context. To this end, we leverage a feedforward architecture equipped with a lightweight autoregressive module, trained using multi-modal supervisory signals that guide the model to jointly predict current and future point maps, camera poses, semantic segmentation, and motion masks.

Multi-modal teachers provide sequence-level pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation from raw YouTube videos without poses, labels, or LiDAR. The resulting encoder not only transfers effectively to downstream autonomous driving planning on the NAVSIM benchmark, surpassing multi-camera and LiDAR baselines with only a single monocular camera, but also yields strong performance when evaluated on a range of semantic, geometric, and qualitative motion prediction tasks.

We emphasize that the novelty of LFG lies more within the pretraining paradigm than the model architecture itself.

Method at a Glance

LFG Architecture. Starting from unposed single-view driving clips, a pretrained π³ backbone encodes N observed frames into latent scene tokens. A lightweight causal autoregressive transformer rolls out M future tokens, which a shared decoder maps to point maps, camera poses, semantic segmentation, confidence maps, and motion masks for all N+M frames. Multi-modal teachers provide pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation that transfers effectively to downstream planning.

Depth Estimation

Click on the image to cycle through: RGB Input → π³ Depth → LFG Depth

Scene 1 Scene 2

RGB Input

Current: Frame 1 Frame 2 Frame 3 Future: Frame 4 Frame 5 Frame 6

LFG predicts depth for both current (observed) and future (predicted) frames, while π³ requires all frames as input.

Semantic Segmentation

Click on the image to cycle through: RGB Input → SegFormer → LFG Semantics

Scene 1 Scene 2

RGB Input

Current: Frame 1 Frame 2 Future: Frame 4 Frame 6

LFG outperforms its SegFormer teacher on both current (observed) and future (predicted) frame predictions.

Motion Prediction

Click on the image to cycle through: RGB Input → LFG Motion → Pseudo GT

Scene 1 Scene 2

RGB Input

Frame 1 Frame 2 Frame 3

LFG correctly identifies dynamic objects (vehicles, pedestrians) and separates them from static scene elements.

Point Cloud Reconstruction

Click on the image to cycle through: LFG → π³

LFG

LFG preserves geometric structure and camera motion even when predicting future frames (shown in red poses).

Quantitative Results

NAVSIM Planning Benchmark

Single-camera LFG vs BEV-based baselines. Higher is better for all metrics.

L = LiDAR. 1Cam uses only the front-view camera with past temporal frames (3-frame input).

Data Efficiency Comparison

PDMS scores on NAVSIM with varying amounts of labeled training data.

LFG demonstrates strong data efficiency, matching full-data DINOv3 performance with only 10% labeled data.

Ablation Studies

Component and scaling ablations on NAVSIM (PDMS). Higher is better.

Scaling pretraining data and extending prediction horizon improve performance, while removing segmentation/motion supervision or the autoregressive head degrades results.

BibTeX

@inproceedings{strong2026lfg,
  title={Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos},
  author={Strong, Matthew and Chang, Wei-Jer and Herau, Quentin and Yang, Jiezhi and Hu, Yihan and Peng, Chensheng and Zhan, Wei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}