stable-diffusion-xl-1.0-inpainting-0.1

Generates new image content within user-defined masked regions using SDXL's dual-text-encoder architecture (OpenCLIP ViT-bigG and CLIP ViT-L) conditioned on text prompts. The model accepts a base image, binary mask, and text description, then uses latent diffusion to iteratively denoise only the masked area while preserving unmasked regions through concatenated conditioning. Implements the inpainting variant of SDXL-1.0 with specialized handling of mask-conditioned latent space.

Encodes text prompts through two independent text encoders (OpenCLIP ViT-bigG for semantic richness and CLIP ViT-L for alignment) producing separate embedding streams that are concatenated and fed into the diffusion UNet. Supports classifier-free guidance (CFG) with independent guidance scales for each encoder, enabling fine-grained control over prompt adherence vs. image quality trade-offs. Text embeddings are computed once and cached, reducing per-step computational overhead.

Implements the core diffusion process in compressed latent space (4x4x4 compression vs. pixel space) using a specialized UNet architecture with cross-attention layers for text conditioning. Starting from Gaussian noise, the model iteratively predicts and removes noise over 20-50 timesteps, with each step conditioned on the text embedding and current noise level (timestep embedding). Mask conditioning is applied by concatenating the masked latent representation to the UNet input, enabling region-specific synthesis while preserving unmasked areas.

Encodes input images into a compressed latent representation using a Variational Autoencoder (VAE) with 4x spatial downsampling (1024x1024 → 128x128 latent), enabling efficient diffusion in latent space. The encoder produces a distribution (mean and log-variance) that is sampled to create the latent vector. During generation, the decoder reconstructs high-resolution images from denoised latents. For inpainting, the encoder processes both the original image and mask, producing masked latents that guide the diffusion process.

Implements inpainting by concatenating the original image's encoded latent representation (outside the masked region) directly to the UNet input alongside the noisy latent being denoised. The mask is downsampled to latent resolution (4x4x4) and used to selectively blend the original latent with the denoised latent at each diffusion step, ensuring unmasked regions remain unchanged while masked regions are synthesized. This approach avoids separate masking networks and enables seamless boundary blending.

Supports generating multiple images in parallel (batch processing) with independent random seeds for each sample, enabling reproducible generation and efficient GPU utilization. The diffusion process is vectorized across the batch dimension, with separate noise schedules and random number generators per sample. Seed control ensures that identical prompts and parameters produce identical outputs, critical for A/B testing and debugging.

Provides multiple noise scheduling strategies (linear, quadratic, cosine, Karras) that define how noise is added and removed across diffusion timesteps. Users can specify the number of inference steps (20-50 typical) and the scheduler type, controlling the trade-off between generation quality and speed. The scheduler computes noise levels (alphas, betas) for each timestep, which are embedded into the UNet to condition the denoising process. Custom schedules can be implemented by extending the scheduler base class.

Supports multiple memory optimization techniques including CPU offloading (moving model components to CPU between uses), 8-bit quantization (reducing model weights from float32 to int8), and attention slicing (processing attention in chunks rather than all at once). These techniques can be combined to reduce peak VRAM usage from ~10GB to ~4-6GB, enabling inference on consumer GPUs. The diffusers pipeline automatically manages offloading and quantization through configuration flags.