Vision Based Locomotion Policy Learning From Real World Robot Trajectories

via “pretrained generalist robot policy inference with multimodal task specification”

Generalist robot policy model from Open X-Embodiment.

Unique: Combines transformer-based sequence modeling with diffusion action heads to predict robot actions from 800K diverse trajectories, enabling zero-shot generalization to new tasks via language/goal conditioning without requiring robot-specific pretraining. The modular tokenizer design (separate observation, task, and action tokenizers) allows flexible composition of perception and instruction modalities.

vs others: Outperforms single-embodiment policies by leveraging diverse training data across 22+ robot platforms, and provides better task generalization than vision-only baselines by jointly modeling language instructions and visual observations through the transformer backbone.

via “vision-language-model-grounding-to-physical-actions”

Google's vision-language-action model for robotics.

Unique: Grounds vision-language semantics to physical actions by co-fine-tuning on robotic trajectories, allowing the model to learn associations between abstract concepts and concrete motor commands within the same transformer architecture

vs others: Achieves tighter semantic grounding than systems that treat vision-language understanding and robot control as separate modules, by training them jointly on aligned robotic data

via “multi-task visual policy learning with task-agnostic world models”

* ⏫ 02/2023: [Grounding Large Language Models in Interactive Environments with Online RL (GLAM)](https://arxiv.org/abs/2302.02662)

Unique: DreamerV3's task-agnostic world model learns shared visual representations without explicit task conditioning, relying on the policy learning objective to extract task-relevant information from the shared latent space. This contrasts with task-conditioned approaches (e.g., MTRL baselines) that explicitly encode task identity, making DreamerV3 more flexible for discovering emergent task structure.

vs others: Achieves better sample efficiency and generalization than task-conditioned baselines by learning task-invariant visual dynamics, while avoiding the computational overhead of task-specific world models or explicit task embeddings.

via “vision-language grounding for robot tasks”

Dataset by cadene. 3,11,762 downloads.

Unique: Integrates natural language task descriptions with robot trajectories at scale, enabling direct training of vision-language models on real robot data without requiring manual annotation of individual frames

vs others: Provides language grounding for robot learning without the annotation overhead of frame-level language labels, making it practical for large-scale vision-language robot learning

via “physics-aware policy learning from high-dimensional visual observations”

* ⭐ 02/2022: [Magnetic control of tokamak plasmas through deep reinforcement learning](https://www.nature.com/articles/s41586-021-04301-9%E2%80%A6)

Unique: Trains end-to-end CNN policies directly on high-resolution camera images by leveraging Gran Turismo's differentiable physics engine, enabling gradient-based optimization of visual perception and control jointly rather than using separate perception and planning modules

vs others: Achieves better sample efficiency and generalization than modular approaches (separate perception + planning) because the visual features are optimized directly for control relevance rather than generic object detection

via “vision-based locomotion policy learning from real-world robot trajectories”

* ⭐ 02/2022: [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://proceedings.mlr.press/v164/jang22a.html)

Unique: Directly trains end-to-end visuomotor policies on real-world robot trajectories without simulation, using robust data augmentation and domain randomization techniques to handle the distribution shift between training and deployment environments. The approach captures implicit terrain understanding through visual features rather than explicit terrain classification.

vs others: Outperforms pure simulation-based approaches by training on real sensor data and terrain interactions, and exceeds hand-crafted controllers by learning adaptive behaviors from diverse demonstrations without manual parameter tuning.

via “end-to-end neural network policy learning for quadruped locomotion”

* ⭐ 10/2022: [Discovering faster matrix multiplication algorithms with reinforcement learning (AlphaTensor)](https://www.nature.com/articles/s41586-022%20-05172-4)

Unique: Learns locomotion policies entirely from raw sensor inputs to motor outputs via PPO without any hand-crafted features, inverse kinematics, or gait primitives, discovering natural gaits emergently through distributed RL training

vs others: Eliminates hand-coded controllers and gait libraries by learning end-to-end policies that adapt to new tasks and terrains, compared to traditional inverse kinematics and trajectory planning approaches

via “vision-language-action-model-transfer-to-robotics”

* ⭐ 07/2023: [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (RT-2)](https://arxiv.org/abs/2307.15818)

Unique: Directly grounds vision-language model representations in robot action spaces by learning a mapping from multimodal observations to motor commands, rather than treating robotics as a separate domain. Leverages internet-scale web knowledge (visual concepts, language semantics) to reduce dependence on large robot-specific datasets.

vs others: Achieves better generalization and sample efficiency than training robot policies from scratch or using task-specific imitation learning, by bootstrapping from foundation models while maintaining interpretability through language grounding.

via “retrospective trajectory optimization via policy gradient learning”

### Other Papers <a name="2023op"></a>

Unique: Applies policy gradient optimization directly to language model action logits using retrospective trajectory data, enabling agents to learn from their own execution history without external reward models or human feedback — a departure from supervised fine-tuning or RLHF approaches that require explicit human preferences

vs others: More sample-efficient than online RL methods because it reuses trajectories already generated during agent deployment, and more scalable than RLHF because it avoids human annotation bottlenecks by learning from task outcomes directly

via “vision-language-conditioned robotic manipulation control”

## Historical Papers <a name="history"></a>

Unique: Uses a unified transformer architecture with separate language and vision token streams fused via cross-attention, enabling a single model to handle diverse manipulation tasks across different robot morphologies without task-specific retraining. Discretizes actions into 8-bit tokens (256 bins per dimension) to leverage transformer's categorical prediction strengths rather than regressing continuous values directly.

vs others: Outperforms prior task-specific policies and vision-only baselines by jointly conditioning on language and vision, achieving 97% success on seen tasks and 76% on novel object generalizations — significantly higher than single-modality or non-transformer baselines on the same evaluation suite.