ComfyUI-LTXVideo

Generates video sequences from natural language prompts using the LTX-2 diffusion transformer model integrated into ComfyUI core. The system tokenizes text through a Gemma-based CLIP encoder, processes it through the DiT (Diffusion Transformer) architecture, and applies iterative denoising in latent space to produce video frames. Supports both base sampling and advanced guidance mechanisms (STG/APG) to control quality and semantic adherence during generation.

Converts a static image into a video sequence by encoding the image as the first frame and using the LTX-2 model to generate subsequent frames that maintain visual consistency and semantic coherence. The system loads the image through the VAE encoder, optionally applies IC-LoRA (in-context LoRA) for structural control, and uses specialized samplers (LTXVInContextSampler) to condition generation on the initial frame while allowing natural motion and scene evolution.

Implements a two-stage video upscaling pipeline that first generates low-resolution video with LTX-2, then applies specialized upscaling models to enhance resolution while preserving temporal coherence and semantic content. The system chains LTX-2 generation with external upscaling models (e.g., RealESRGAN, BSRGAN) through ComfyUI's node system, managing intermediate representations and quality metrics throughout the pipeline.

Enables precise control over camera movement and object motion in generated videos through in-context LoRA (IC-LoRA) conditioning. The system allows users to specify camera trajectories (pan, zoom, rotate) and object motion paths, which are encoded as conditioning signals and injected into the diffusion process. IC-LoRA weights are loaded through LTXVQ8LoraModelLoader and applied during sampling to guide motion generation without full model retraining.

Provides a plugin architecture that registers custom nodes with ComfyUI through a dual-registration system (static mappings in __init__.py and runtime-generated nodes from nodes_registry.py). The system enables users to compose complex video generation workflows by connecting nodes in ComfyUI's visual editor, with automatic type checking and data flow validation. NODE_CLASS_MAPPINGS and NODE_DISPLAY_NAME_MAPPINGS enable ComfyUI Manager compatibility and user-friendly node discovery.

Integrates Lightricks' Gemma-based CLIP text encoder for semantic understanding of prompts, with intelligent caching to avoid redundant encoding of identical prompts. The system implements LTXVGemmaCLIPModelLoader and LTXVGemmaCLIPModelLoaderMGPU that load the encoder, cache embeddings for repeated prompts, and manage encoder lifecycle across multiple generation calls. Supports both single-GPU and multi-GPU loading strategies.

Extends existing video sequences by generating additional frames that seamlessly blend with original footage. The system uses LTXVExtendSampler to process latent representations of video clips, applies temporal blending operations (LTXVBlendLatents) to smooth transitions between original and generated frames, and supports looping generation (LTXVLoopingSampler) for continuous video synthesis. Latent normalization (LTXVNormalizeLatents) ensures consistent quality across extended sequences.

Applies spatial and temporal guidance during video generation to improve quality and semantic adherence without retraining the model. The system implements two guidance mechanisms: STG (Spatial-Temporal Guidance) for general quality improvement and APG (Adaptive Prompt Guidance) for semantic control. Nodes (STGGuiderNode, STGGuiderAdvancedNode, MultimodalGuiderNode) inject guidance signals into the diffusion process at configurable timesteps, modulating the denoising direction toward desired outputs while maintaining diversity.

Reduces model memory footprint through 8-bit quantization, enabling LTX-2 inference on GPUs with limited VRAM (16GB or less). The system implements LTXVQ8LoraModelLoader and LowVRAMCheckpointLoader nodes that load model weights in quantized format, apply dynamic dequantization during inference, and optionally load LoRA adapters in quantized form. This approach trades minimal quality loss for significant memory savings (typically 40-50% reduction).

Distributes model components across multiple GPUs to enable larger batch sizes and longer video generation on multi-GPU systems. The system implements LTXVGemmaCLIPModelLoaderMGPU and memory optimization nodes that partition the text encoder, diffusion model, and VAE across available devices, managing inter-device communication and synchronization. Automatic memory profiling (LowVRAMCheckpointLoader) detects available VRAM and adjusts model placement accordingly.

Provides low-level operations on compressed video representations (latent tensors) to enable advanced workflows without decoding to pixel space. The system implements nodes (LTXVSelectLatents, LTXVBlendLatents, LTXVNormalizeLatents, LTXVConcatenateLatents) that manipulate latent dimensions, blend multiple latent sequences, normalize distributions, and concatenate temporal sequences. These operations work directly in compressed space, enabling efficient composition of video generation results.

Converts between pixel-space video frames and compressed latent representations using a variational autoencoder optimized for temporal coherence. The system provides VAE encoder/decoder nodes that process video sequences frame-by-frame or in temporal chunks, maintaining consistency across frames while achieving 8-16x compression. Supports both standard VAE decoding and tiled decoding for memory-constrained scenarios.

Generates high-resolution videos by dividing the spatial domain into overlapping tiles, sampling each tile independently, and blending results at tile boundaries. The system implements LTXVTiledSampler that manages tile generation, overlap regions, and boundary blending to produce seamless high-resolution output without requiring proportional VRAM increases. Tile size and overlap are configurable to balance quality and memory usage.

Augments user prompts with automatically generated enhancements and applies dynamic conditioning during generation. The system provides utility nodes that expand prompts with style descriptors, quality keywords, and temporal directives, then injects these enhanced prompts into the diffusion process at configurable timesteps. Supports both static prompt enhancement and dynamic prompt scheduling that varies conditioning over generation timesteps.