ComfyUI CLI

ComfyUI represents image generation pipelines as directed acyclic graphs (DAGs) where nodes are atomic operations connected by edges representing data flow. The execution engine (execution.py) traverses this graph, executing only nodes whose inputs have changed since the last run, leveraging a smart caching layer that tracks node outputs and invalidates downstream dependents. This approach eliminates redundant computation—e.g., if only a prompt changes, the VAE encoding and diffusion sampling are re-executed while model loading is skipped.

ComfyUI provides a plugin architecture where custom nodes are Python classes that inherit from a base node interface and register themselves via a node registry (nodes.py 10881-10882). The system auto-discovers custom nodes from designated directories, introspects their input/output signatures using Python type hints (comfy_types/node_typing.py), and exposes them in the frontend without requiring code changes to the core. This enables third-party developers to add new operations (e.g., ControlNet, LoRA patching, custom samplers) as isolated, reusable components.

ComfyUI exposes a REST API (server.py) and WebSocket connection for remote workflow submission, execution monitoring, and real-time progress updates. Clients submit workflows as JSON, receive execution status via WebSocket events (node execution, progress, errors), and retrieve results via HTTP. The API supports batch processing, workflow queuing, and cancellation. WebSocket events include intermediate outputs (e.g., preview images during sampling), enabling real-time visualization of generation progress without waiting for completion.

ComfyUI's blueprint system (blueprints and subgraph system) allows users to encapsulate reusable workflow segments as blueprints, which can be instantiated multiple times with different parameters. Blueprints are stored as JSON and can be nested, enabling hierarchical workflow composition. Subgraphs are dynamically instantiated at runtime, allowing parameterized workflow templates. This enables code reuse without custom node development, and facilitates sharing of common patterns (e.g., 'upscale and enhance' subgraph) across teams.

ComfyUI's quantization system supports multiple precision levels (fp32, fp16, bf16, int8, int4) and mixed-precision inference, where different model components run at different precisions. The system automatically selects optimal precision based on hardware capabilities and available VRAM, with configurable fallback strategies. Quantization reduces model size and memory bandwidth, enabling inference on resource-constrained hardware. The system tracks memory usage and automatically switches between precision levels or enables offloading if VRAM is exhausted.

ComfyUI's CLI (cli_args.py, main.py) provides command-line arguments for configuring execution environment, model paths, GPU selection, and server settings. Arguments control device selection (CPU/GPU), precision (fp32/fp16/bf16), memory optimization (offload, sequential CPU offload), and server configuration (port, listen address). Configuration can be specified via command-line flags or environment variables, enabling easy deployment across different hardware configurations without code changes.

ComfyUI's model management system (model_detection.py, model_management.py) automatically detects model architecture from file metadata (safetensors headers, checkpoint keys) and routes models to appropriate loaders. The system supports Stable Diffusion 1.5/2.x, SDXL, Flux, Flow Matching models, video generation models (WAN), and specialized architectures (DiT, MMDiT). Models are loaded into GPU/CPU memory with configurable precision (fp32, fp16, bf16) and quantization strategies (int8, int4), with automatic offloading to manage VRAM constraints.

ComfyUI's model patching system allows runtime modification of model weights through LoRA (Low-Rank Adaptation) and other patching techniques. LoRA weights are loaded separately and composed with base model weights using low-rank matrix multiplication, enabling style transfer, concept injection, and fine-tuned adaptations without retraining. The patching system (model_patching.py) intercepts model forward passes, applies weight modifications on-the-fly, and supports stacking multiple LoRAs with configurable strength multipliers, all without modifying the original model checkpoint.

ComfyUI provides a comprehensive sampling framework supporting multiple diffusion algorithms (DDPM, DDIM, DPM++, Euler, Heun, LCM) and schedulers (linear, cosine, karras, exponential). The sampling system (diffusion and sampling nodes) allows users to compose custom samplers by chaining noise schedulers, guidance methods (CFG, GLIGEN, T5 conditioning), and sampler algorithms without writing code. Advanced features include custom noise injection, latent masking for inpainting, and support for flow-matching models (Flux) alongside traditional diffusion.

ComfyUI's conditioning system encodes text prompts into embeddings using multiple text encoders (CLIP, T5, BERT) and supports advanced prompt syntax for weighted token emphasis, negative prompts, and multi-stage conditioning. The system (text encoding nodes, conditioning nodes) tokenizes prompts, applies attention weights to specific tokens, and composes positive/negative conditioning pairs. Embeddings are cached and reused across sampling steps, reducing redundant computation. Support for LoRA-patched text encoders enables style-specific prompt interpretation.

ComfyUI integrates ControlNet and T2I-Adapter models as specialized nodes that inject spatial control signals (edge maps, depth, pose, canny edges) into the diffusion process. These control models are loaded separately and applied at specific timesteps with configurable strength, allowing users to guide generation toward specific compositions, poses, or structures. The system supports multiple control methods stacked in a single workflow, with independent strength multipliers and timestep ranges for fine-grained control over which denoising steps are influenced.

ComfyUI's VAE system (latent_formats.py) handles conversion between pixel space and latent space using Variational Autoencoders. The system supports multiple latent formats (standard VAE, VAE-FP32, VAE-Tiling for memory efficiency) and automatically selects appropriate encoding/decoding strategies based on available VRAM. Tiling mode processes large images in overlapping patches to avoid OOM errors, while maintaining output quality through careful patch blending. The system also supports custom VAE models and mixed-precision VAE inference.

ComfyUI provides a suite of image processing nodes for pixel manipulation, masking, and compositing. Operations include resizing, cropping, color space conversion, mask generation from images, and compositing with alpha blending. The system supports both raster operations (pixel-level) and mask-based operations (binary or soft masks) for inpainting and selective generation. Masks can be generated from images using thresholding, edge detection, or manual drawing, and applied to control which regions of an image are modified during generation.

ComfyUI supports video generation models (WAN, AnimateDiff) and frame interpolation techniques through specialized nodes. Video models generate multiple frames conditioned on text prompts and optional control signals, with support for motion control and temporal consistency. Frame interpolation nodes estimate intermediate frames between keyframes, enabling smooth motion and higher frame rates. The system handles video I/O (frame extraction, video encoding) and temporal batching for efficient multi-frame processing.