Diffusers

Provides a unified DiffusionPipeline base class that orchestrates end-to-end inference by composing modular components (UNet, VAE, text encoder, scheduler) into a single callable interface. The pipeline system extends ConfigMixin and ModelMixin, enabling automatic configuration serialization, device management, and gradient checkpointing across all sub-components. Pipelines are loaded via auto-detection (AutoPipeline) or explicit instantiation, with support for dynamic component swapping and memory-efficient execution hooks.

Implements a SchedulerMixin base class that abstracts noise scheduling algorithms (DDPM, DDIM, Euler, DPM++, LCM, etc.) behind a unified interface. Each scheduler manages timestep ordering, noise scale calculation, and the denoising step computation via a configurable noise schedule (linear, cosine, sqrt). Schedulers are swappable at runtime and support both deterministic and stochastic sampling strategies, enabling inference speed/quality trade-offs without changing the model or pipeline code.

Implements ConfigMixin and ModelMixin base classes that provide automatic configuration serialization, device management, and checkpoint loading/saving. Configurations are stored as JSON files alongside model weights, enabling reproducible inference and easy model sharing. The system supports loading from Hugging Face Hub, local files, or single-file checkpoints (safetensors), with automatic format detection and conversion.

Provides utilities for memory-efficient inference including gradient checkpointing, attention slicing, VAE tiling, and sequential model loading. Gradient checkpointing trades computation for memory by recomputing activations during backprop. Attention slicing reduces peak memory by processing attention in chunks. VAE tiling enables processing of large images by tiling the latent space. Sequential loading moves components between devices to reduce peak VRAM usage.

Provides hooks for profiling and optimizing inference performance, including memory profiling, latency measurement, and attention visualization. Hooks are registered on pipeline components and called at each denoising step, enabling real-time monitoring without modifying pipeline code. The system supports custom hooks for user-defined profiling or optimization logic.

Implements AutoPipeline class that automatically detects the correct pipeline variant based on model architecture and configuration. The system inspects model config files (config.json) to identify the model type (Stable Diffusion, SDXL, Flux, etc.) and selects the appropriate pipeline class. This enables loading any diffusion model with a single function call without specifying the pipeline type.

Provides a LoRA system that loads low-rank adaptation weights into model components (UNet, text encoder) via the PEFT library integration. LoRA weights are stored separately from base model weights, enabling efficient fine-tuning and inference with minimal memory overhead. The system supports loading multiple LoRA adapters with weighted fusion, enabling style mixing and multi-concept composition without retraining. Single-file loading via safetensors format enables direct checkpoint loading without conversion.

Implements ControlNet and IP-Adapter systems that inject spatial or semantic conditioning into the diffusion process. ControlNet uses auxiliary encoder-decoder networks to condition the UNet on edge maps, depth maps, pose, or other spatial controls. IP-Adapter conditions generation on image embeddings (CLIP image features) for style or content guidance. Both systems operate via cross-attention injection, enabling fine-grained control over generation without retraining the base model.

Provides image-to-image and inpainting pipelines that encode input images into latent space via VAE, add noise according to a strength parameter, and denoise using the diffusion process. Inpainting additionally uses a mask to preserve unmasked regions while regenerating masked areas. The latent space approach enables efficient editing without pixel-space operations, supporting variable image sizes and aspect ratios through latent tiling.

Implements SDXL pipelines that use a two-stage generation process: a base model generates low-quality images, and a refiner model upsamples and refines details. The pipeline manages separate text encoders (CLIP-L and OpenCLIP-G) for richer semantic understanding, supports negative prompts for both stages, and enables style/aesthetic guidance via prompt weighting. The refiner stage can be skipped for speed or applied selectively to high-quality base outputs.

Provides pipelines for Flux and Diffusion Transformer (DiT) models that replace the UNet with transformer-based architectures. These models use joint text-image token processing, enabling more efficient scaling and better semantic understanding. The pipeline system abstracts away transformer-specific details (token merging, attention patterns, sequence length management) behind the standard DiffusionPipeline interface.

Provides pipelines for video generation (text-to-video, image-to-video) and frame interpolation that extend the image diffusion process to temporal dimensions. Models like AnimateDiff and Stable Video Diffusion use temporal attention layers to maintain consistency across frames. The pipeline manages frame batching, temporal noise scheduling, and optional motion guidance for controlling video dynamics.

Implements multiple guidance techniques that steer generation toward text prompts or away from negative prompts. Classifier-free guidance (CFG) uses unconditional predictions to compute a guidance direction. Perturbed Attention Guidance (PAG) perturbs attention maps to amplify semantic features. These techniques are applied during the denoising loop via guidance scale parameters, enabling fine-grained control over prompt adherence without retraining.

Provides training scripts for DreamBooth (fine-tuning the entire UNet on a few images of a subject) and textual inversion (learning a new token embedding for a concept). Both techniques enable personalization without retraining the entire model. DreamBooth uses prior preservation to prevent overfitting, while textual inversion optimizes only the token embedding. Both output LoRA-compatible checkpoints or embedding files that can be applied to any model.