krita-ai-diffusion

Generates or modifies image content within Krita selections using diffusion models, with optional natural language prompts to guide generation. The plugin extracts the selection mask, encodes it as a conditioning signal, and passes it to the diffusion backend alongside the prompt embedding, enabling precise control over generation boundaries without manual masking workflows.

Extends image boundaries beyond the current canvas by generating new content in specified directions (up, down, left, right). The plugin detects canvas edges, creates temporary extended canvases with padding, applies diffusion conditioning to preserve edge coherence, and seamlessly merges generated content back into the original document. Supports multi-directional expansion in a single operation.

Abstracts backend infrastructure (local diffusion server, cloud API, or hybrid) behind a unified client interface, enabling users to switch between local and cloud execution without code changes. The plugin manages server lifecycle (installation, startup, shutdown), handles connection pooling and request routing, and provides fallback logic (e.g., fall back to cloud if local server unavailable). Supports both self-hosted backends (ComfyUI, Invoke) and cloud services (Replicate, RunwayML).

Discovers available diffusion models from registries (Hugging Face, CivitAI, etc.), downloads model weights with progress tracking and resume capability, verifies integrity using checksums, and caches models locally for reuse. The plugin maintains a model registry with metadata (architecture, size, download URL, checksum), handles partial downloads and network interruptions, and provides UI for browsing and installing models without command-line tools.

Enables users to save and load generation parameter presets (prompt, model, sampler, guidance scale, steps, seed, ControlNet settings, etc.) as named styles or configurations. The plugin stores presets in a local registry with metadata, provides UI for browsing and applying presets, and supports preset sharing via export/import. Presets can be organized into categories and tagged for easy discovery.

Enables advanced users to define custom generation workflows using a node-graph interface, where nodes represent diffusion operations (sampling, conditioning, upscaling, etc.) and edges represent data flow. The plugin provides a visual workflow editor with parameter binding, enabling users to create complex multi-step pipelines (e.g., generate → upscale → inpaint) without code. Workflows are stored as JSON and can be shared or version-controlled.

Provides intelligent autocomplete for generation prompts using embedding-based semantic search over a prompt database. As users type, the plugin suggests relevant prompt completions based on semantic similarity to the input, enabling faster prompt writing and discovery of effective prompt patterns. Suggestions are ranked by relevance and frequency, and users can customize the suggestion database.

Provides multi-language UI support with community-contributed translations, enabling users to use the plugin in their native language. The plugin uses a translation framework (e.g., gettext) with string extraction and community translation workflows, and supports dynamic language switching without restart. Includes fallback to English for untranslated strings.

Enables spatial control over generation by linking Krita layers to distinct text prompts, which are then converted to region-specific conditioning signals during diffusion. The plugin maintains a region registry that maps layer geometry to prompt embeddings, allowing users to define what should be generated in different areas of the canvas without manual mask creation. Regions can overlap, and the diffusion backend composites their conditioning signals.

Applies ControlNet models to guide diffusion generation using structural cues extracted from the canvas or user input. The plugin supports multiple ControlNet modes: scribble (user-drawn lines), line art (edge detection), canny (edge detection), pose (skeleton extraction), depth (depth map), normals (surface normals), and segmentation (semantic masks). Each mode extracts or generates the appropriate conditioning signal and passes it to the diffusion backend with configurable control strength.

Enables style transfer and reference-based generation using IP-Adapter, which encodes reference images into a style/aesthetic embedding that guides diffusion without replacing the base prompt. The plugin extracts image features using a CLIP-based encoder, generates IP-Adapter conditioning tokens, and blends them with text prompt embeddings at configurable weights. Supports multiple reference images with independent weight control.

Abstracts diffusion model architecture differences (SD1.5, SDXL, Illustrious, Flux) behind a unified interface, automatically detecting model type and selecting appropriate conditioning adapters, tokenizers, and inference pipelines. The plugin maintains a model registry with metadata (architecture, supported ControlNets, IP-Adapter availability, optimal resolution), and routes generation requests to the correct backend implementation without user intervention.

Interprets user brush strokes in real-time as generation guidance, updating the canvas incrementally as the user paints. The plugin monitors brush input, extracts stroke geometry and color information, encodes strokes as conditioning signals (similar to scribble ControlNet), and triggers generation updates at configurable intervals (e.g., every 500ms or every 10 strokes). Generated content is composited onto the canvas as a preview layer, allowing users to see results while painting.

Scales generated or existing images to 4K (2160p), 8K (4320p), or higher resolutions using diffusion-based upscaling with tile-based processing to manage memory constraints. The plugin divides large images into overlapping tiles, upscales each tile independently using a diffusion upscaler model, and blends tiles at boundaries to eliminate seams. Supports both 2x and 4x upscaling factors with configurable tile overlap and blending strategies.

Maintains an asynchronous job queue for generation requests, enabling batch processing, job history tracking, and preview management without blocking the Krita UI. The plugin queues generation jobs with parameters, executes them sequentially or in parallel (if hardware supports), stores results with metadata (parameters, generation time, seed), and provides a history interface for reviewing and re-running past generations. Supports batch generation of multiple variations with parameter sweeps.

Automatically scales generation resolution based on available VRAM and target image size, using tile-based layout to process images larger than the model's native resolution. The plugin estimates VRAM requirements, selects optimal tile size and overlap, and orchestrates multi-tile generation with boundary blending. Supports both upscaling (generating at lower resolution then upscaling) and native tiling (generating tiles at full resolution).