Vision Based Locomotion Policy Learning From Real World Robot Trajectories

via “pretrained generalist robot policy inference with multimodal task specification”

Generalist robot policy model from Open X-Embodiment.

Unique: Combines transformer-based sequence modeling with diffusion action heads to predict robot actions from 800K diverse trajectories, enabling zero-shot generalization to new tasks via language/goal conditioning without requiring robot-specific pretraining. The modular tokenizer design (separate observation, task, and action tokenizers) allows flexible composition of perception and instruction modalities.

vs others: Outperforms single-embodiment policies by leveraging diverse training data across 22+ robot platforms, and provides better task generalization than vision-only baselines by jointly modeling language instructions and visual observations through the transformer backbone.