datasets

Loads datasets into memory as PyArrow Table objects via the Dataset class, enabling columnar storage with zero-copy access patterns. The ArrowDataset abstraction wraps PyArrow's Table API, providing lazy evaluation for transformations (map, filter, select) that are compiled into Arrow compute expressions rather than executed immediately. This approach enables efficient memory usage and fast iteration over structured data with native support for nested types, media features (images, audio), and distributed processing.

The IterableDataset class enables streaming data loading without materializing the full dataset in memory, using a buffer-based approach that fetches data in configurable chunks. Implements a generator-based iteration pattern where data is downloaded and processed on-the-fly, with optional local caching of streamed batches. This architecture supports infinite datasets and enables training on datasets larger than available RAM by trading off random access for sequential streaming efficiency.

The data_files module automatically discovers and matches data files based on glob patterns and file extensions, enabling loading of datasets split across multiple files (e.g., train_*.parquet, test_*.csv). The system supports hierarchical directory structures, multiple file formats in a single dataset, and custom pattern matching logic. It handles file listing, format detection, and split assignment automatically, abstracting away file system complexity.

The train_test_split() method partitions a dataset into multiple splits (train, test, validation) with configurable ratios and optional stratification. The system supports deterministic splitting via seed-based shuffling, stratified splitting to maintain class distributions, and custom split functions. The implementation returns a DatasetDict with named splits, enabling easy access to each partition throughout the training pipeline.

The DatasetCard class provides a structured format for dataset documentation following Hugging Face standards, including description, license, citations, and usage instructions. The system generates cards from templates and metadata, validates card structure, and publishes cards to the Hub alongside datasets. The architecture supports both manual card creation and automatic generation from dataset properties.

The load_dataset() function provides a single entry point for loading datasets from diverse sources (local files, Hugging Face Hub, remote URLs, custom scripts) by routing to appropriate DatasetBuilder implementations. The system uses a plugin architecture where each dataset is defined by a builder module (Python script or packaged module) that specifies download logic, data file patterns, and feature schemas. The API handles caching, version management, and automatic format detection, abstracting away source-specific complexity.

The map(), filter(), and select() operations compile transformations into a computation graph that is executed lazily, with each operation assigned a deterministic fingerprint based on the function code and input dataset state. This fingerprinting system enables automatic caching of intermediate results; if the same transformation is applied twice, the cached result is reused. The architecture stores transformation metadata (function hash, parameters) alongside cached data, enabling reproducibility and avoiding redundant computation across runs.

The Features class defines a schema for dataset columns with support for primitive types (int, string, float), nested structures (sequences, dicts), and media types (Image, Audio, Video). Each feature type includes encoding logic (serialization to Arrow format) and decoding logic (deserialization to Python objects or framework-specific formats). The system validates data against the schema during loading and provides automatic type conversion, ensuring type safety across the data pipeline.

The distributed module enables parallel processing of datasets across multiple workers (processes or machines) by automatically sharding data and coordinating transformations. Each worker receives a subset of the dataset based on its rank and world size, with built-in synchronization primitives to ensure consistent state across workers. The system handles distributed map operations, aggregations, and shuffle operations while managing communication overhead and load balancing.

The search module enables semantic search over datasets by building vector indices using Faiss (for in-memory similarity search) or Elasticsearch (for distributed search). The system computes embeddings for specified columns (via a user-provided embedding function), stores them in the index, and provides efficient nearest-neighbor retrieval. The architecture abstracts the underlying index backend, allowing seamless switching between Faiss (fast, single-machine) and Elasticsearch (distributed, persistent).

The set_format() method configures how dataset examples are returned when iterating, with specialized formatters for PyTorch (returns torch.Tensor), TensorFlow (returns tf.Tensor), JAX (returns jax.numpy arrays), and NumPy (returns NumPy arrays). Each formatter handles type conversion, batching, and padding automatically, enabling seamless integration with framework-specific training loops. The system maintains the underlying Arrow storage while providing framework-specific views on demand.

The push_to_hub() method uploads datasets to Hugging Face Hub repositories with automatic Git-based version control, enabling dataset versioning, branching, and collaboration. The system manages dataset files (Parquet, metadata) as Git LFS objects, tracks changes across versions, and provides dataset cards (documentation) with standardized metadata. The architecture integrates with the Hub API for authentication, access control, and dataset discoverability.

The batch() method groups dataset examples into fixed-size batches with optional dynamic padding to handle variable-length sequences. The system supports both static batching (fixed batch size) and dynamic batching (variable batch size based on token count), with automatic padding to the maximum length in each batch. The implementation integrates with the formatter system to return batches in framework-specific formats (PyTorch, TensorFlow, etc.).