Efficient Fine Tuning For New Robot Embodiments And Observation Action Spaces

via “efficient fine-tuning for new robot embodiments and observation-action spaces”

Generalist robot policy model from Open X-Embodiment.

Unique: Implements modular fine-tuning where observation tokenizers, task tokenizers, and action heads can be independently retrained while freezing the transformer backbone, reducing fine-tuning data requirements from 100K+ trajectories to 10-500 by leveraging pretrained representations. Includes built-in task augmentation (language paraphrasing, image transformations) to artificially expand small datasets.

vs others: Requires 10-100x fewer demonstrations than training embodiment-specific policies from scratch, and provides better generalization than simple behavioral cloning by preserving the pretrained transformer's learned action distributions and task understanding.

via “co-fine-tuning-with-vision-language-preservation”

Google's vision-language-action model for robotics.

Unique: Implements co-fine-tuning by representing actions as text tokens within the language modeling framework, allowing the same transformer architecture to simultaneously optimize for vision-language understanding and robotic action prediction without separate policy heads

vs others: Preserves semantic understanding from web-scale vision-language pretraining better than standard fine-tuning by maintaining both vision and text encoder knowledge, while avoiding the computational overhead of separate policy networks or adapter modules

via “multimodal-grounding-of-language-in-action-space”

* ⭐ 07/2023: [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (RT-2)](https://arxiv.org/abs/2307.15818)

Unique: Learns joint embeddings across vision, language, and action modalities with explicit action grounding, enabling the model to map language semantics directly to motor commands rather than treating action prediction as a separate supervised learning problem.

vs others: Achieves better compositional generalization and language understanding than vision-only imitation learning, while being more sample-efficient than training separate language and action models due to shared multimodal representations.

via “cross-robot morphology action space abstraction and transfer”

## Historical Papers <a name="history"></a>

Unique: Uses a unified token-based action representation that abstracts away robot-specific details, allowing a single transformer policy to generate actions for diverse morphologies via lightweight morphology-specific decoders. This contrasts with prior approaches that train separate policies per robot or use explicit morphology-aware network branches.

vs others: Enables zero-shot or few-shot transfer to new robot morphologies without retraining the core policy, whereas task-specific or morphology-specific baselines require full retraining or extensive fine-tuning.