Image Folder Dataset Loading And Caching

via “image-folder dataset loading and caching”

Dataset by Maynor996. 3,80,160 downloads.

Unique: Uses HuggingFace's Arrow-based columnar storage backend for zero-copy memory mapping of image metadata, enabling random access to 380K+ images without materializing the full dataset; integrates native streaming via the datasets library's built-in caching layer rather than requiring manual download orchestration

vs others: More memory-efficient than torchvision.ImageFolder for large-scale datasets because it leverages Arrow's columnar format and lazy evaluation, avoiding eager loading of image paths and metadata into Python objects