diffusers

Implements a DiffusionPipeline base class that orchestrates text encoders, UNet denoisers, VAE decoders, and schedulers as pluggable components. Pipelines inherit from both ConfigMixin and ModelMixin, enabling automatic configuration serialization, device management, and gradient checkpointing across heterogeneous model architectures. The system uses a component registry pattern where each pipeline declares its required components (e.g., text_encoder, unet, vae, scheduler) and automatically handles loading, device placement, and inference orchestration without requiring users to manually wire components.

Implements a SchedulerMixin base class with pluggable scheduler implementations (DDPM, DDIM, PNDM, Euler, DPM++, LCM) that abstract noise scheduling, timestep scaling, and denoising step computation. Each scheduler encapsulates a noise schedule (linear, cosine, sqrt) and provides methods like set_timesteps(), step(), and scale_model_input() that work identically across different sampling algorithms. The system decouples the diffusion process definition from the sampling strategy, allowing users to swap schedulers without modifying pipeline code or retraining models.

Implements classifier-free guidance (CFG) that trains the model to predict both conditional (text-guided) and unconditional (noise) predictions, then interpolates between them at inference time using a guidance scale parameter. The guidance direction is computed as (conditional_pred - unconditional_pred) * guidance_scale, amplifying the model's response to the text prompt. This enables fine-grained control over prompt adherence without requiring a separate classifier, allowing users to trade off prompt fidelity vs image diversity by adjusting a single scalar parameter.

Implements IP-Adapter that injects image embeddings from a frozen image encoder (CLIP ViT) into the UNet's cross-attention layers, enabling image-based conditioning alongside text prompts. IP-Adapter uses a lightweight adapter module that projects image embeddings to the same space as text embeddings, allowing seamless composition with text guidance. This enables image-to-image style transfer, image-based retrieval-augmented generation, and multi-modal prompting without modifying the base diffusion model or text encoder.

Implements ConfigMixin and ModelMixin base classes that provide automatic configuration serialization (save_config/from_config), model loading/saving (save_pretrained/from_pretrained), and device management (to/cpu/cuda). ConfigMixin automatically registers constructor parameters as configuration attributes, enabling full reproducibility of model instantiation. ModelMixin integrates with HuggingFace Hub for seamless checkpoint downloading and caching, supporting both PyTorch and SafeTensors formats. The system handles device placement, gradient checkpointing, and memory optimization transparently.

Provides memory optimization techniques including xFormers-based efficient attention (reduces attention memory from O(n²) to O(n)), gradient checkpointing (trades compute for memory by recomputing activations), and mixed-precision inference (FP16/BF16). The system automatically detects available optimizations (xFormers, Flash Attention, etc.) and applies them transparently. Inference hooks enable custom optimization strategies without modifying pipeline code, supporting techniques like token merging, attention slicing, and sequential processing.

Supports batch processing of multiple prompts or images in a single inference pass, enabling efficient GPU utilization and reduced latency per sample. The system manages batch dimension across all pipeline components (text encoder, UNet, VAE) with automatic padding and masking for variable-length inputs. Seed control enables deterministic generation for reproducibility and A/B testing, with per-sample seed support for batch generation. The pipeline automatically handles batch size optimization based on available VRAM.

Implements StableDiffusionPipeline that encodes text prompts using a frozen CLIP text encoder, projects embeddings into the UNet's cross-attention layers, and iteratively denoises a latent tensor conditioned on text. The pipeline uses a VAE encoder to compress images to latent space (4x downsampling), applies the diffusion process in latent space for efficiency, and decodes final latents back to pixel space using the VAE decoder. Cross-attention mechanisms in the UNet allow fine-grained control over which image regions attend to which prompt tokens, enabling semantic layout control.

Extends StableDiffusionPipeline to accept an input image and optional inpainting mask, encoding the image to latent space and initializing the diffusion process from a noisy version of that latent (rather than pure noise). For inpainting, the pipeline masks out regions to regenerate while preserving masked regions by blending original and denoised latents at each step. The mask is encoded as a spatial attention map that guides the UNet to focus regeneration on masked areas while maintaining coherence with unmasked regions.

Integrates ControlNet modules that accept spatial conditioning inputs (edge maps, depth maps, pose skeletons, semantic segmentation) and inject spatial information into the UNet via zero-convolution layers. ControlNet operates in parallel to the main UNet, processing conditioning inputs through a separate encoder and injecting features at multiple scales via residual connections. This enables precise spatial control over image generation without modifying the base diffusion model, allowing users to specify exact object positions, poses, or scene layouts.

Implements LoRA (Low-Rank Adaptation) training that decomposes weight updates into low-rank matrices (A and B), reducing trainable parameters by 100-1000x compared to full fine-tuning. During inference, LoRA weights are merged into the base model via W_new = W_base + (A @ B) * scale, enabling efficient model adaptation without storing separate checkpoints. The system integrates with PEFT library for automatic LoRA injection into UNet and text encoder, supporting multiple LoRA adapters that can be composed or swapped at inference time.

Implements DreamBooth training that fine-tunes a diffusion model on 3-5 images of a subject (person, object, style) using a rare token (e.g., 'sks person') paired with class-prior preservation. Class-prior preservation trains on unrelated images of the same class (e.g., 'person') to prevent language drift and maintain model generalization. The training objective combines subject-specific loss (matching rare token to subject images) with class-prior loss (maintaining diversity of class tokens), enabling the model to generate novel images of the subject in new contexts while preserving general image quality.

Implements Textual Inversion training that learns a small embedding vector (typically 1-10 tokens) representing a visual concept (style, object, attribute) by optimizing the embedding to match target images. The learned embedding is inserted into the text encoder's token space, enabling the model to generate images of the concept by using the learned token in prompts. Training optimizes only the embedding vector while keeping the text encoder and diffusion model frozen, making it extremely parameter-efficient (100-1000 parameters vs millions for LoRA).

Extends diffusion pipelines to generate video by applying the diffusion process across temporal dimensions, using temporal attention layers that enforce consistency across frames. The system supports frame-by-frame generation with optical flow-based warping for temporal coherence, or latent-space video diffusion that operates on sequences of latent frames. Temporal attention mechanisms (e.g., 3D convolutions, temporal transformers) enable the model to maintain object identity and motion consistency across generated frames without explicit motion specification.

Integrates Variational Autoencoders (VAE) that compress images to a low-dimensional latent space (4-8x spatial downsampling) and reconstruct images from latents. The VAE encoder maps images to a distribution (mean and log-variance) in latent space, enabling stochastic sampling; the decoder reconstructs images from latent samples. Diffusion operates in this compressed latent space rather than pixel space, reducing memory and compute by 16-64x while maintaining quality through the VAE's learned reconstruction. The system supports multiple VAE architectures (standard VAE, VAE-KL, VAE-VQ) with different compression-quality tradeoffs.