ComfyUI

ComfyUI implements a directed acyclic graph (DAG) execution model where users visually connect nodes representing atomic operations. The backend parses the graph structure, topologically sorts nodes based on dependencies, and executes them in order while managing data flow between nodes. The execution engine (execution.py) handles dependency resolution, caching of intermediate results, and selective re-execution of only modified subgraphs, enabling efficient iterative refinement of complex pipelines.

ComfyUI's model management system (model_management.py, model_detection.py) automatically detects model architecture from weights files, loads appropriate model implementations, and manages GPU/CPU memory allocation. The system supports multiple model families (Stable Diffusion 1.5/2.x, SDXL, Flux, WAN, ControlNet) with automatic quantization selection (fp32, fp16, bf16) based on available VRAM. Memory is tracked per-model with automatic offloading to CPU when VRAM is exhausted, enabling inference on hardware with limited GPU memory.

ComfyUI's server (server.py) implements a queue-based execution model where workflows are submitted as JSON, queued, and executed asynchronously. The system supports batch processing of multiple workflows with priority queues, execution status tracking, and WebSocket-based progress updates. Workflows can be submitted programmatically via HTTP API, enabling integration with external applications. The execution engine processes queued workflows sequentially, with support for cancellation and pause/resume.

ComfyUI's quantization system (model_management.py) automatically selects appropriate data types (fp32, fp16, bf16, int8) based on available VRAM and hardware capabilities. The system supports per-layer quantization, allowing different layers to use different precisions for optimal memory/quality tradeoff. Mixed-precision inference runs different model components at different precisions (e.g., attention in fp16, output in fp32) to reduce memory while maintaining quality. Quantization is applied transparently during model loading without requiring separate quantized model files.

ComfyUI's blueprint system allows users to encapsulate workflow subgraphs as reusable components with defined inputs/outputs. Blueprints are stored as JSON and can be instantiated multiple times within larger workflows, enabling modular workflow design. The system supports blueprint nesting (blueprints containing other blueprints) and parameter passing, allowing complex workflows to be built from smaller reusable pieces. Blueprints can be shared as files or registered in a central library for team reuse.

ComfyUI's web-based frontend provides real-time preview of node outputs, allowing users to inspect intermediate results as workflows execute. The system supports interactive debugging by pausing execution, inspecting tensor shapes and values, and modifying node parameters without restarting. Preview nodes can be inserted anywhere in the workflow to visualize intermediate results (latents, embeddings, masks). The frontend communicates with the backend via WebSocket, enabling live updates as execution progresses.

ComfyUI enables composition of multiple models within a single workflow, allowing text from one model to condition another, or ensemble sampling where multiple models generate in parallel and results are blended. The system manages separate model instances in memory, handles conditioning tensor compatibility across models, and supports model-specific sampling strategies. Advanced users can implement ensemble techniques (e.g., averaging predictions from multiple models) by composing sampling nodes with custom logic.

ComfyUI provides a modular sampling system where users can compose different samplers (Euler, DPM++, LCM, etc.), schedulers (linear, karras, exponential), and guidance methods (CFG, DPM-Solver, Perturbed Attention Guidance) as separate nodes. The system implements the diffusion sampling loop with configurable noise schedules, step counts, and conditioning injection points. Custom sampler nodes allow advanced users to implement novel sampling strategies by directly accessing the noise schedule and model prediction tensors, enabling research-grade flexibility.

ComfyUI integrates ControlNet and T2I-Adapter models as specialized conditioning nodes that inject spatial control signals (edge maps, depth, pose, canny edges) into the diffusion process. These nodes load control model weights, process input images through the control encoder, and generate conditioning tensors that guide generation toward specific spatial layouts. The system supports multiple control methods simultaneously (e.g., pose + depth) by concatenating conditioning tensors, enabling complex multi-constraint generation.

ComfyUI's model patching system (model_patching.py) enables runtime modification of model weights through LoRA (Low-Rank Adaptation) merging and custom weight patches. LoRA weights are loaded as separate files, decomposed into rank-reduced matrices, and merged into base model weights during inference without modifying the original model files. The system supports multiple simultaneous LoRAs with per-LoRA strength scaling, enabling style transfer, subject-specific generation, and fine-tuned behavior without model retraining or storage overhead.

ComfyUI's text conditioning system (implemented via text encoder nodes) converts text prompts into embedding tensors using multiple encoder backends (CLIP, T5, Llama, custom transformers). The system supports prompt weighting syntax (e.g., '(word:1.5)' for emphasis, '[word:other:0.5]' for cross-fade) that modifies embedding magnitudes and interpolation. Multiple text encoders can be chained to produce multi-modal embeddings (e.g., CLIP + T5 for SDXL), and embeddings can be manipulated via tensor operations for advanced prompt engineering.

ComfyUI's VAE system (latent_formats.py, VAE nodes) encodes images into compressed latent representations and decodes latents back to pixel space. The system supports multiple latent formats (SD 1.5 VAE, SDXL VAE, Flux VAE, custom VAE) with automatic format detection and conversion. Latent tensors can be manipulated directly (scaling, interpolation, noise injection) before decoding, enabling advanced techniques like latent blending and inpainting. The VAE operates as separate encode/decode nodes, allowing latent inspection and manipulation between steps.

ComfyUI's extension system (nodes.py, extension registration) allows developers to create custom nodes by implementing Python classes with input/output type definitions and execution logic. Custom nodes are automatically discovered from designated directories, registered with type hints, and exposed in the UI without code changes. The system provides base classes and utilities for common patterns (model loading, tensor operations, image processing), and custom nodes can access the full Python ecosystem and GPU resources. Node type system uses Python type annotations to enable automatic UI generation and type validation.

ComfyUI supports video generation models (WAN, Flux Video, AnimateDiff) that generate multiple frames with temporal consistency constraints. The system handles frame batching, temporal noise scheduling, and frame interpolation. Video nodes manage frame sequences as tensor batches, support variable frame counts and frame rates, and can apply temporal guidance (motion vectors, optical flow) to constrain motion patterns. Generated frames can be post-processed with interpolation nodes to increase frame rate or smooth motion.

ComfyUI's inpainting system uses mask tensors to define editable regions, and the diffusion process is constrained to only modify masked areas while preserving unmasked regions. Mask nodes support various input formats (binary masks, grayscale masks, alpha channels) and operations (dilation, erosion, feathering). The inpainting process injects the original image into unmasked regions at each diffusion step, enabling seamless blending. Mask-guided sampling can be combined with ControlNet for precise spatial control of inpainted content.