Decomposed Dual Branch Diffusion Inpainting With Masked Feature Separation

via “inpainting with masked region regeneration”

Open-source image generation — SD3, SDXL, massive ecosystem of LoRAs, ControlNets, runs locally.

Unique: Freezes unmasked latent regions during diffusion rather than post-processing or blending, ensuring the diffusion process respects spatial constraints throughout. This architectural approach produces better boundary coherence than naive masking-after-generation, though still requires careful mask preparation.

vs others: More flexible and cheaper than cloud-based inpainting APIs (Photoshop Generative Fill, DALL-E inpainting), but requires manual mask creation and produces less seamless blending than commercial tools optimized for this task.

via “inpainting and outpainting with mask-guided generation”

Widely adopted open image model with massive ecosystem.

Unique: Applies diffusion selectively to masked regions in latent space while preserving unmasked areas through masking operations in the UNet, enabling seamless blending without requiring separate inpainting-specific model weights or post-processing

vs others: Faster and more flexible than traditional content-aware fill algorithms, and produces more natural results than naive copy-paste or cloning approaches by understanding semantic context

via “image inpainting with masked region filling”

Implementation of Imagen, Google's Text-to-Image Neural Network, in Pytorch

Unique: Incorporates masks directly into diffusion process through concatenation with noisy images, enabling spatial awareness without separate mask encoder, and supports both training and inference with arbitrary mask patterns

vs others: Integrates masking into core diffusion loop rather than post-processing, enabling better boundary handling and semantic understanding of masked regions compared to naive blending approaches

via “inpainting and outpainting with mask-guided diffusion”

stable diffusion webui colab

Unique: Integrates inpainting directly into the WebUI's Gradio canvas interface, allowing users to draw masks interactively rather than preparing mask images externally — the notebook pre-loads inpainting model variants and exposes blend/feathering controls as UI sliders

vs others: More intuitive than command-line inpainting tools because users can draw masks directly in the browser and see results immediately, whereas standalone approaches require external mask preparation and manual parameter tuning

via “mask-aware latent concatenation for region-preserving inpainting”

text-to-image model by undefined. 2,97,544 downloads.

Unique: Concatenates the original latent directly to UNet input rather than using a separate masking network, reducing model complexity and enabling efficient reuse of the original latent across multiple inpainting runs. Mask blending occurs in latent space at each diffusion step, ensuring smooth transitions without post-processing.

vs others: Direct latent concatenation is simpler and faster than separate masking networks (e.g., used in some proprietary inpainting models), while producing comparable or better boundary quality because the original latent is preserved throughout the entire diffusion process rather than blended only at the end.

via “inpainting with mask-based region editing”

text-to-image model by undefined. 7,85,165 downloads.

Unique: Stable Diffusion v1.5 inpainting uses a separate VAE encoder for masked regions and blends generated content with original at each denoising step, enabling seamless region editing. The mask is applied in latent space, reducing artifacts compared to pixel-space blending.

vs others: More precise than image-to-image because mask enables region-specific control; more efficient than separate inpainting models because it reuses the diffusion process with mask conditioning

via “inpainting-selective-image-region-replacement”

Diffusion Bee is the easiest way to run Stable Diffusion locally on your M1 Mac. Comes with a one-click installer. No dependencies or technical knowledge needed.

Unique: Uses specialized inpainting model checkpoints that are trained with mask-aware conditioning, allowing the diffusion process to understand mask boundaries and blend seamlessly. The implementation encodes both image and mask through separate pathways in the latent space, enabling precise control over which regions are modified.

vs others: More precise than content-aware fill algorithms (which use statistical inpainting) and faster than manual Photoshop cloning, while requiring less training data than generative inpainting models that must learn from scratch.

via “decomposed dual-branch diffusion inpainting with masked feature separation”

[ECCV 2024] The official implementation of paper "BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion"

Unique: Uses decomposed dual-branch architecture with dense per-pixel control injected at multiple UNet resolution levels, enabling plug-and-play integration without modifying base model weights. Unlike naive masking approaches, separates masked feature processing from latent noise processing, reducing learning burden and improving boundary quality.

vs others: Achieves higher inpainting quality than simple mask-based approaches (e.g., Inpaint-LoRA) while maintaining compatibility with any pre-trained diffusion model, and requires significantly less training data than full model fine-tuning approaches.

via “masked image inpainting with diffusion-guided completion”

Kandinsky 2 — multilingual text2image latent diffusion model

Unique: Implements inpainting by zeroing latent features in masked regions rather than pixel-space masking, enabling coherent completion that respects both text guidance and unmasked image context. Supports soft masks (grayscale) for smooth boundary blending, reducing visible seams.

vs others: Produces fewer boundary artifacts than Stable Diffusion inpainting due to diffusion prior conditioning, and supports multilingual prompts for non-English inpainting instructions.

via “inpainting and image editing with diffusion-based content fill”

我的 ComfyUI 工作流合集 | My ComfyUI workflows collection

Unique: Provides Stable Cascade inpainting workflows with pre-tuned mask handling and feathering parameters, eliminating manual mask preprocessing that typically requires 3-5 iterations to achieve seamless blending

vs others: More flexible than Photoshop's content-aware fill because users can control the text prompt and model parameters; faster than traditional inpainting (Photoshop) because diffusion-based inpainting is GPU-accelerated

via “image-to-image generation with latent inpainting and mask-based conditioning”

State-of-the-art diffusion in PyTorch and JAX.

Unique: Implements mask-based latent blending where original latents are preserved in masked regions and only masked regions are denoised, enabling seamless inpainting without explicit boundary handling. Strength parameter controls the noise level of the initial latent, allowing fine-grained control over edit intensity.

vs others: More efficient than pixel-space inpainting and more controllable than GAN-based inpainting; latent-space approach enables semantic understanding of edits, though boundary artifacts require post-processing unlike some specialized inpainting models.

via “practical stable diffusion applications (inpainting, editing, upscaling)”

Python materials for the online course on diffusion models by [@huggingface](https://github.com/huggingface).

via “image-inpainting-via-conditional-diffusion”

* 🏆 2020: [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT)](https://arxiv.org/abs/2010.11929)

Unique: DDPM enables zero-shot inpainting by leveraging the forward process to compute noisy versions of known pixels at each timestep, then replacing unknown pixels with model predictions. This approach requires no special training and works with any trained diffusion model. The key insight is that the forward process provides a principled way to inject known information at each denoising step.

vs others: Requires no special training (unlike GAN-based inpainting), enables flexible mask shapes and sizes, and can be combined with text guidance for semantic inpainting.

via “high-quality inpainting with reduced computational cost”

* ⭐ 10/2022: [LAION-5B: An open large-scale dataset for training next generation image-text models (LAION-5B)](https://arxiv.org/abs/2210.08402)

Unique: Achieves 1-4 step inpainting by distilling guidance mechanisms, enabling semantic-aware region filling without separate guidance models. Latent-space implementation reduces computational cost while maintaining visual quality.

vs others: 10-100× faster than standard diffusion-based inpainting, but may produce visible artifacts or boundary inconsistencies at extreme step reduction compared to full-step approaches.

via “image inpainting and selective region editing”

Unique: Leverages the cascaded diffusion architecture to perform multi-stage inpainting where masked regions are regenerated at each super-resolution stage (64×64→256×256→1024×1024) while preserving unmasked content, enabling both coarse semantic changes and fine detail consistency

vs others: Inpainting quality benefits from the same frozen T5-XXL text encoder that enables superior language understanding in base generation, allowing more nuanced text-guided edits than models with weaker linguistic comprehension