Dataset Loading And Preprocessing For Heterogeneous Task Formats

via “open x-embodiment dataset loading and preprocessing”

Generalist robot policy model from Open X-Embodiment.

Unique: Implements a modular data pipeline that handles 800K trajectories across 22+ robot platforms in heterogeneous formats (HDF5, TFRecord, RLDS) through standardized loaders and preprocessing steps. Supports lazy loading and on-the-fly augmentation to manage dataset scale without requiring full in-memory loading.

vs others: Handles significantly larger and more diverse datasets than single-robot datasets (e.g., MIME, Bridge), enabling better generalization through exposure to diverse embodiments and tasks. The standardized pipeline makes it easier to add new data sources compared to custom per-dataset loaders.