Instruction Conditioned Image Editing Via Diffusion Models

via “image-to-image transformation with style and content control”

Widely adopted open image model with massive ecosystem.

Unique: Uses VAE encoder to compress input images into latent space, then applies diffusion with text conditioning and a learnable strength parameter, enabling smooth interpolation between input preservation and prompt-driven transformation without requiring separate inpainting models

vs others: More flexible than traditional style transfer (which requires paired training data) and faster than iterative refinement approaches, while maintaining structural fidelity better than pure text-to-image generation

via “fast image generation with distilled diffusion steps”

Stability AI's 8B parameter flagship image generation model.

Unique: Applies knowledge distillation to compress diffusion steps from standard schedule to 4 steps while preserving the full 8.1B parameter model, enabling faster inference without architectural changes or separate lightweight model training

vs others: Faster than standard Stable Diffusion 3.5 Large with same parameter count, but slower than purpose-built fast models like LCM-LoRA or consistency models; trades speed for quality more conservatively than extreme distillation approaches

via “control-net guided image generation”

Stable Diffusion API — image generation, editing, upscaling, SD3/SDXL, video, and 3D models.

Unique: Implements ControlNet architecture as a separate conditioning branch that guides the diffusion process without modifying the base model, allowing multiple control types to be composed. Provides pre-computed control representations (canny edges, depth maps) rather than requiring users to generate them, reducing integration complexity.

vs others: More flexible than simple style transfer because it preserves spatial structure while allowing arbitrary text prompts; more accessible than training custom ControlNets because pre-built types are provided

via “image-to-image and inpainting with latent space editing”

Hugging Face's diffusion model library — Stable Diffusion, Flux, ControlNet, LoRA, schedulers.

Unique: Encodes reference images into VAE latent space, adds noise proportional to strength parameter, and denoises with text guidance, enabling controlled editing without full regeneration. Inpainting uses mask-guided latent blending to preserve masked regions while editing unmasked areas, whereas competitors often require separate inpainting models or post-processing.

vs others: More efficient than full regeneration; latent-space editing preserves content structure while enabling style/content changes. Inpainting with mask support is more precise than prompt-only editing, enabling pixel-level control without text descriptions.

via “image inpainting”

Stable Diffusion by Stability AI is a state of the art text-to-image model that generates images from text. #opensource

Unique: The inpainting feature is integrated into the same diffusion process as the text-to-image generation, allowing for a unified model that can handle both tasks without needing separate architectures.

vs others: More flexible than traditional inpainting tools because it can generate entirely new content based on textual prompts rather than relying solely on existing image data.

via “image-to-image-conditional-generation”

Diffusion Bee is the easiest way to run Stable Diffusion locally on your M1 Mac. Comes with a one-click installer. No dependencies or technical knowledge needed.

Unique: Implements VAE-based latent space encoding/decoding with configurable noise scheduling, allowing fine-grained control over how much of the original image structure is preserved versus how much creative freedom the diffusion process has. The strength parameter directly maps to the timestep at which diffusion begins, providing intuitive control.

vs others: More flexible than simple style transfer (which requires paired training data) and faster than full regeneration, while offering more control than cloud-based image editing tools that abstract away the strength/guidance parameters.

via “decomposed dual-branch diffusion inpainting with masked feature separation”

[ECCV 2024] The official implementation of paper "BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion"

Unique: Uses decomposed dual-branch architecture with dense per-pixel control injected at multiple UNet resolution levels, enabling plug-and-play integration without modifying base model weights. Unlike naive masking approaches, separates masked feature processing from latent noise processing, reducing learning burden and improving boundary quality.

vs others: Achieves higher inpainting quality than simple mask-based approaches (e.g., Inpaint-LoRA) while maintaining compatibility with any pre-trained diffusion model, and requires significantly less training data than full model fine-tuning approaches.

via “image-to-image transformation with text-guided refinement”

Kandinsky 2 — multilingual text2image latent diffusion model

Unique: Uses MOVQ encoder (67M parameters) instead of standard VAE for input image encoding, providing better reconstruction fidelity in latent space. Strength parameter controls noise schedule initialization, enabling smooth interpolation between preservation and regeneration without separate model variants.

vs others: Achieves finer control over image preservation than Stable Diffusion's img2img through explicit diffusion prior conditioning, and supports multilingual prompts natively unlike most open-source alternatives.

via “differential diffusion with region-specific generation control”

我的 ComfyUI 工作流合集 | My ComfyUI workflows collection

Unique: Provides differential diffusion workflows that expose per-pixel generation strength control, a capability unavailable in most commercial tools (Midjourney, DALL-E 3) and rarely documented in open-source implementations

vs others: More granular than inpainting masks (binary or soft) because differential diffusion allows continuous per-pixel strength variation; more flexible than ControlNet because it operates on the image itself rather than requiring separate control images

via “image-to-image generation with latent inpainting and mask-based conditioning”

State-of-the-art diffusion in PyTorch and JAX.

Unique: Implements mask-based latent blending where original latents are preserved in masked regions and only masked regions are denoised, enabling seamless inpainting without explicit boundary handling. Strength parameter controls the noise level of the initial latent, allowing fine-grained control over edit intensity.

vs others: More efficient than pixel-space inpainting and more controllable than GAN-based inpainting; latent-space approach enables semantic understanding of edits, though boundary artifacts require post-processing unlike some specialized inpainting models.

via “practical stable diffusion applications (inpainting, editing, upscaling)”

Python materials for the online course on diffusion models by [@huggingface](https://github.com/huggingface).

via “language-guided image editing with instruction following”

* ⏫ 07/2023: [Meta-Transformer: A Unified Framework for Multimodal Learning (Meta-Transformer)](https://arxiv.org/abs/2307.10802)

Unique: Performs language-guided editing within the unified decoder by conditioning on both image and text tokens, enabling instruction-based editing without separate mask inputs or specialized editing architectures

vs others: More intuitive than mask-based editing because it uses natural language instructions; more flexible than ControlNet because it doesn't require precise spatial control inputs

via “text-guided image editing with minimal denoising steps”

* ⭐ 10/2022: [LAION-5B: An open large-scale dataset for training next generation image-text models (LAION-5B)](https://arxiv.org/abs/2210.08402)

Unique: Achieves 2-4 step image editing by distilling guidance information, enabling interactive editing without separate guidance models. Preserves unedited regions through latent-space conditioning while reducing computational overhead.

vs others: 10-50× faster than standard diffusion-based editing (e.g., InstructPix2Pix with full steps), but may sacrifice fine-grained control and semantic accuracy compared to non-distilled approaches.

via “image-inpainting-via-conditional-diffusion”

* 🏆 2020: [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT)](https://arxiv.org/abs/2010.11929)

Unique: DDPM enables zero-shot inpainting by leveraging the forward process to compute noisy versions of known pixels at each timestep, then replacing unknown pixels with model predictions. This approach requires no special training and works with any trained diffusion model. The key insight is that the forward process provides a principled way to inject known information at each denoising step.

vs others: Requires no special training (unlike GAN-based inpainting), enables flexible mask shapes and sizes, and can be combined with text guidance for semantic inpainting.

via “instruction-guided image editing via diffusion”

instruct-pix2pix — AI demo on HuggingFace

Unique: Uses a dual-conditioning architecture combining CLIP text embeddings with image features in a single UNet, enabling instruction-guided edits without separate mask inputs or region selection — differs from traditional inpainting approaches that require explicit mask specification

vs others: More intuitive than mask-based editing tools and faster than training custom LoRA adapters, but less precise than pixel-level editing tools like Photoshop for geometric transformations

via “text-to-image generation with diffusion-based synthesis”

IF — AI demo on HuggingFace

Unique: Implements a cascaded multi-stage diffusion pipeline (base + super-resolution stages) rather than single-stage generation, enabling higher quality and resolution through progressive refinement. Uses frozen language model embeddings for text conditioning, reducing training complexity compared to end-to-end approaches like DALL-E.

vs others: Achieves higher image quality and finer detail than single-stage models (Stable Diffusion) through cascaded architecture, while maintaining faster inference than autoregressive approaches (DALL-E) by leveraging efficient diffusion sampling.

via “diffusion model inference with gpu acceleration”

IC-Light — AI demo on HuggingFace

Unique: Implements lighting-aware conditioning by injecting spatial maps into the diffusion model's cross-attention layers, rather than relying solely on text prompts or implicit context. This allows precise control over lighting direction without requiring complex prompt engineering.

vs others: Faster than CPU-based inference by 50-100x due to GPU parallelization of matrix operations, and produces higher-quality results than simpler inpainting methods (like content-aware fill) because it leverages learned generative priors from large-scale training.

via “image-generation-from-text-prompts-with-diffusion-models”

* ⭐ 03/2023: [Scaling up GANs for Text-to-Image Synthesis (GigaGAN)](https://arxiv.org/abs/2303.05511)

Unique: Integrates diffusion model inference into a conversational loop where the LLM can interpret user feedback ('make it more vibrant', 'add more detail') and translate it into updated prompts or adjusted diffusion parameters, rather than requiring users to manually re-engineer prompts.

vs others: Provides conversational refinement loop absent in standalone DALL-E or Midjourney APIs, and offers lower latency than some cloud-only solutions by supporting local inference.

via “instruction-conditioned image editing via diffusion models”

* ⭐ 12/2022: [Multi-Concept Customization of Text-to-Image Diffusion (Custom Diffusion)](https://arxiv.org/abs/2212.04488)

Unique: Pioneering approach to instruction-conditioned image editing using diffusion models with a two-stage training pipeline (semantic pre-training + instruction fine-tuning) that enables natural language control over pixel-level edits without explicit masks or selection tools. Concatenates image and text embeddings in the diffusion conditioning mechanism to jointly reason about source content and edit intent.

vs others: Outperforms prior mask-based editing methods (e.g., Inpainting) by eliminating the need for manual segmentation and enabling semantic understanding of edit intent, while being more controllable than pure text-to-image generation by anchoring edits to source image content.

via “text-to-image generation with diffusion model inference”

IllusionDiffusion — AI demo on HuggingFace

Unique: Integrates optical illusion conditioning into the standard Stable Diffusion pipeline via cross-attention fusion, rather than using simple prompt engineering or post-processing, enabling structural guidance that persists throughout the entire denoising process

vs others: Produces more coherent illusion-guided outputs than naive prompt-based approaches because the illusion pattern is embedded directly into the diffusion latent space, not just mentioned in text; faster than fine-tuning custom models because it uses pre-trained Stable Diffusion weights with conditioning injection