Kandinsky-2

Converts natural language text prompts into images using a two-stage pipeline: text embeddings are first processed through a diffusion prior (1B parameters in v2.1+) that maps text space to CLIP image embeddings, then fed into a latent diffusion U-Net (1.2-1.22B parameters) operating in compressed latent space. Kandinsky 2.0 uses dual text encoders (mCLIP-XLMR 560M + mT5-encoder-small 146M) while v2.1+ uses XLM-Roberta-Large-ViT-L-14 (560M). The diffusion prior acts as a bridge between modalities, enabling more coherent image generation than direct text-to-pixel approaches.

Transforms existing images by encoding them into latent space via MOVQ encoder, then applying iterative diffusion steps guided by text prompts and a strength parameter (0-1) that controls how much the original image influences the output. The process uses the same diffusion prior and U-Net as text-to-image but initializes the noise schedule at a later timestep based on strength, allowing fine-grained control over preservation vs. modification. Supports both Kandinsky 2.0 (direct U-Net conditioning) and 2.1+ (diffusion prior + U-Net) architectures.

Classifier-free guidance (CFG) is implemented by computing both conditional (text-guided) and unconditional predictions, then scaling the difference: output = unconditional + guidance_scale * (conditional - unconditional). Higher guidance scales (10-15) increase semantic alignment with text prompts but reduce image diversity and may introduce artifacts. Lower scales (5-8) produce more diverse but less prompt-aligned images. Guidance scale is a hyperparameter exposed in all generation methods.

MOVQ (Multiscale Orthogonal Vector Quantization) is a 67M parameter encoder-decoder that compresses images into latent space for efficient diffusion processing. Unlike standard VAE, MOVQ uses vector quantization to discretize latent codes, improving reconstruction fidelity and reducing artifacts. Introduced in Kandinsky 2.1 as a replacement for VAE. The encoder downsamples images by 8x; the decoder upsamples latent codes back to pixel space with minimal quality loss.

Kandinsky 2.0 uses two text encoders in parallel: mCLIP-XLMR (560M parameters) for multilingual semantic understanding and mT5-encoder-small (146M parameters) for linguistic structure. Both encoders process the same text prompt independently, producing separate embeddings that are concatenated and fed into the U-Net. This dual-encoder approach enables strong multilingual support without requiring separate models per language. Kandinsky 2.1+ replaces this with a single XLM-Roberta-Large-ViT-L-14 encoder (560M).

Negative prompts are text descriptions of unwanted content (e.g., 'blurry, low quality, distorted'). During generation, the model computes predictions for both positive and negative prompts, then uses the difference to steer generation away from negative content. Implemented via classifier-free guidance: output = conditional_positive + guidance_scale * (conditional_positive - conditional_negative). Negative prompts are optional but widely used to improve quality by excluding common artifacts.

Fills masked regions of images by encoding the full image into latent space, zeroing out latent features corresponding to masked pixels, then running diffusion with text guidance to reconstruct masked areas while preserving unmasked context. The process uses the diffusion prior (v2.1+) or direct U-Net conditioning (v2.0) to guide generation toward text-aligned completions. Mask can be binary (0/255) or soft (grayscale 0-255) for graduated blending at boundaries.

Combines multiple images and text prompts by encoding each image into CLIP embeddings via the image encoder (ViT-L/14 in v2.1, ViT-bigG-14 in v2.2), interpolating or averaging embeddings, then using the diffusion prior to map the blended embedding to a coherent image. Supported in Kandinsky 2.1+ only. Allows weighted blending of image concepts (e.g., 0.7*image1 + 0.3*image2) with text guidance to steer the final output toward desired attributes.

Kandinsky 2.2 integrates ControlNet architecture to enable spatial conditioning of image generation via depth maps, edge maps, or other control signals. The control signal is encoded into a separate conditioning pathway that guides the diffusion U-Net without replacing text embeddings, allowing precise spatial control while maintaining semantic alignment with text prompts. Currently supports depth-based control; architecture extensible to other control modalities.

The get_kandinsky2() factory function provides a unified entry point for loading Kandinsky models with automatic device placement (CPU/CUDA), version selection (2.0, 2.1, 2.2), and task-specific configuration. The factory handles model weight downloading from Hugging Face Hub, caching, and memory-efficient loading. Abstracts version differences so users can switch between Kandinsky versions with a single parameter change without rewriting generation code.

Supports generating multiple images from a single prompt or multiple prompts in a single batch operation, with configurable batch size to fit available VRAM. Internally manages tensor allocation and GPU memory to prevent out-of-memory errors. Batch processing is more efficient than sequential generation due to amortized model loading and reduced overhead per image.

Encodes images into CLIP embedding space (ViT-L/14 in v2.1, ViT-bigG-14 in v2.2) to extract semantic features for image mixing, similarity comparison, or downstream tasks. The image encoder is frozen (not fine-tuned) and used as a feature extractor. Embeddings are 768-dimensional (ViT-L) or 1280-dimensional (ViT-bigG), enabling semantic operations in embedding space without pixel-level processing.

Kandinsky 2.1+ includes a trainable diffusion prior (1B parameters) that maps text embeddings to CLIP image embeddings. The prior can be fine-tuned on custom datasets to improve alignment between text and generated images for specific domains (e.g., product photography, character art). Training uses standard diffusion loss (MSE between predicted and actual noise) with text conditioning. Requires custom training code; not exposed via high-level API.

The core image generation component is a 1.2-1.22B parameter U-Net operating in latent space (encoded by MOVQ, 67M parameters). The U-Net uses cross-attention layers to condition on text embeddings (from dual encoders in v2.0, or from diffusion prior in v2.1+). Iterative denoising over 50-100 diffusion steps produces the final image. The architecture supports classifier-free guidance (CFG) to boost semantic alignment with text prompts by scaling the difference between conditional and unconditional predictions.