CogVideo

Generates videos from natural language prompts using a dual-framework architecture: HuggingFace Diffusers for production use and SwissArmyTransformer (SAT) for research. The system encodes text prompts into embeddings, then iteratively denoises latent video representations through diffusion steps, finally decoding to pixel space via a VAE decoder. Supports multiple model scales (2B, 5B, 5B-1.5) with configurable frame counts (8-81 frames) and resolutions (480p-768p).

Extends text-to-video by conditioning on an initial image frame, generating temporally coherent video continuations. Accepts an image and optional text prompt, encodes the image into the latent space as a keyframe, then applies diffusion-based temporal synthesis to generate subsequent frames. Maintains visual consistency with the input image while respecting motion cues from the text prompt. Implemented via CogVideoXImageToVideoPipeline in Diffusers and equivalent SAT pipeline.

Provides utilities for preparing video datasets for training, including video decoding, frame extraction, caption annotation, and data validation. Handles variable-resolution videos, aspect ratio preservation, and caption quality checking. Integrates with HuggingFace Datasets for efficient data loading during training. Supports both manual caption annotation and automatic caption generation via vision-language models.

Provides flexible model configuration system supporting multiple CogVideoX variants (2B, 5B, 5B-1.5) with different resolutions, frame counts, and precision levels. Configuration is specified via YAML or Python dicts, enabling easy switching between model sizes and architectures. Supports both Diffusers and SAT frameworks with unified config interface. Includes pre-defined configs for common use cases (lightweight inference, high-quality generation, variable-resolution).

Enables video editing by inverting existing videos into latent space using DDIM inversion, then applying diffusion-based refinement conditioned on new text prompts. The inversion process reconstructs the latent trajectory of an input video, allowing selective modification of content while preserving temporal structure. Implemented via inference/ddim_inversion.py with configurable inversion steps and guidance scales to balance fidelity vs. editability.

Provides bidirectional weight conversion between SAT (SwissArmyTransformer) and Diffusers frameworks via tools/convert_weight_sat2hf.py and tools/export_sat_lora_weight.py. Enables researchers to train models in SAT (with fine-grained control) and deploy in Diffusers (with production optimizations), or vice versa. Handles parameter mapping, precision conversion (BF16/FP16/INT8), and LoRA weight extraction for efficient fine-tuning.

Reduces GPU memory usage by 3x through sequential CPU offloading (pipe.enable_sequential_cpu_offload()) and VAE tiling (pipe.vae.enable_tiling()). Offloading moves model components to CPU between diffusion steps, keeping only the active component in VRAM. VAE tiling processes large latent maps in tiles, reducing peak memory during decoding. Supports INT8 quantization via TorchAO for additional 20-30% memory savings with minimal quality loss.

Implements Low-Rank Adaptation (LoRA) fine-tuning for video generation models, reducing trainable parameters from billions to millions while maintaining quality. LoRA adapters are applied to attention layers and linear projections, enabling efficient adaptation to custom datasets. Supports distributed training via SAT framework with multi-GPU synchronization, gradient accumulation, and mixed-precision training (BF16). Adapters can be exported and loaded independently via tools/export_sat_lora_weight.py.

Enables full supervised fine-tuning (SFT) of CogVideoX models on custom video datasets via SAT framework. Implements end-to-end training pipeline including dataset preparation (video preprocessing, caption generation/annotation), distributed training with gradient checkpointing, and checkpoint management. Supports variable-resolution training and mixed-precision (BF16) for efficient multi-GPU training on A100/H100 clusters.

Provides command-line interface (inference/cli_demo.py) for running text-to-video, image-to-video, and video-to-video generation without code. Exposes key parameters as CLI arguments: prompt, image_path, video_path, num_frames, guidance_scale, seed, output_path. Supports both Diffusers and SAT backends via --framework flag. Includes progress bars, memory monitoring, and error handling for batch processing.

Provides interactive web interface (inference/gradio_web_demo.py) for video generation using Gradio framework. Exposes text-to-video, image-to-video, and video-to-video modes via tabbed interface. Includes real-time parameter sliders (guidance_scale, num_frames, seed), file upload widgets, and live generation preview. Supports both Diffusers and SAT backends with automatic framework detection.

Supports INT8 and FP8 quantization via TorchAO library for reduced memory usage and faster inference. Quantizes model weights and activations to 8-bit precision while maintaining output quality through calibration on representative data. Integrated into inference pipeline via inference/cli_demo_quantization.py. Reduces memory footprint by 20-30% and inference latency by 10-20% with minimal quality degradation.