Hotshot-XL

Generates short video clips from natural language text prompts by extending Stable Diffusion XL's 2D UNet architecture to a 3D temporal UNet (UNet3DConditionModel). The system encodes text prompts via CLIP embeddings, generates random noise in latent space, then iteratively denoises across temporal dimensions using cross-attention mechanisms, finally decoding latents back to pixel space via VAE. This approach maintains frame-to-frame coherence by processing all frames jointly rather than independently.

Extends the base text-to-video pipeline with ControlNet integration (HotshotXLControlNetPipeline) to inject spatial guidance via control images (depth maps, canny edges, pose skeletons, etc.). Control images are processed through a ControlNet encoder that produces conditioning signals injected into the UNet3D's cross-attention layers at multiple scales, allowing precise spatial control over video generation while maintaining temporal coherence. The control signal is applied uniformly across all frames, ensuring consistent spatial structure throughout the video.

Uses residual blocks (ResNet-style) in the UNet3D encoder and decoder for efficient feature extraction and spatial/temporal upsampling/downsampling. ResNet blocks include skip connections that allow gradients to flow directly through the network, improving training stability and enabling deeper architectures. The encoder progressively downsamples spatial dimensions while increasing feature channels, and the decoder reverses this process. Skip connections from encoder to decoder preserve fine-grained spatial information, critical for maintaining video quality and temporal coherence.

Builds on the Diffusers library's DiffusionPipeline abstraction, inheriting model loading, scheduling, and inference utilities while implementing custom HotshotXLPipeline and HotshotXLControlNetPipeline classes. This integration provides standardized interfaces for model management, scheduler selection, and output handling, reducing boilerplate code and enabling compatibility with Diffusers ecosystem tools. The pipeline abstraction separates model logic from inference orchestration, making code modular and maintainable.

Encodes natural language text prompts into high-dimensional embeddings using pre-trained CLIP text encoders (typically OpenAI's CLIP-ViT-L or CLIP-ViT-G), then injects these embeddings into the UNet3D denoising process via cross-attention mechanisms. The text embeddings guide the diffusion process at each denoising step by computing attention weights between the latent features and text token embeddings, effectively steering the generation toward semantically relevant content. This approach reuses SDXL's proven text conditioning strategy, enabling natural language control over video content.

Encodes video frames into a compressed latent space using a pre-trained Variational Autoencoder (VAE) from Stable Diffusion XL, reducing computational cost and memory requirements for the diffusion process. The VAE encoder compresses each frame by a factor of 8 (spatial dimensions), allowing the UNet3D to operate on smaller tensors. After diffusion completes, the VAE decoder reconstructs pixel-space video frames from denoised latents. This two-stage approach (encode → diffuse in latent space → decode) is critical for making video generation tractable on consumer hardware.

Implements the core diffusion loop by iteratively denoising latent tensors over a configurable number of steps (typically 30-50 steps) using a noise scheduler (e.g., DDIM, Euler, DPM++) that controls the noise level at each step. At each denoising step, the UNet3D predicts the noise component in the current latent, which is subtracted to move toward the clean signal. The scheduler determines the noise schedule (how quickly noise is removed), enabling trade-offs between quality (more steps) and speed (fewer steps). Text embeddings and optional control signals guide the denoising via cross-attention at each step.

Provides a fine-tuning pipeline (fine_tune.py) that allows users to adapt the pre-trained Hotshot-XL model to domain-specific video generation tasks by training on custom video datasets. Fine-tuning updates the UNet3D weights (and optionally text encoders) on new data while leveraging pre-trained SDXL weights as initialization. The pipeline supports LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning, reducing VRAM and storage requirements. Users can fine-tune on custom video styles, objects, or concepts not well-represented in the base model's training data.

Implements memory optimization techniques (enable_attention_slicing, enable_vae_slicing, sequential attention computation) that reduce peak VRAM usage by trading off inference speed. When enabled, attention computations are split into smaller chunks processed sequentially rather than all at once, and VAE operations are similarly chunked. This allows inference on GPUs with 8GB VRAM (vs. 16GB+ for full resolution), making video generation accessible on consumer hardware. The optimization is transparent to users; quality is preserved while latency increases by ~20-30%.

Provides a user-friendly CLI (inference.py) for video generation with configurable parameters including prompt, output resolution, video length, number of denoising steps, guidance scale, scheduler type, and optional ControlNet conditioning. The CLI handles model loading, pipeline initialization, and output saving (MP4, GIF, or frame sequences) without requiring users to write Python code. Parameters are passed via command-line arguments or a configuration file, enabling easy experimentation and batch generation.

Implements a 3D UNet architecture (UNet3DConditionModel) that extends Stable Diffusion XL's 2D UNet by adding temporal attention layers between spatial attention blocks. Temporal attention operates across the time dimension, allowing the model to learn motion patterns and ensure consistency across frames. The architecture processes all frames jointly during denoising, with temporal attention computing relationships between latent features at different time steps. This joint processing is critical for generating coherent motion rather than independent, jittery frames.

Implements cross-attention mechanisms in the UNet3D that compute attention weights between spatial/temporal latent features and text token embeddings. At each denoising step, the model queries latent features against text embeddings, allowing the model to selectively attend to relevant text tokens and steer generation toward semantically aligned content. The cross-attention is applied at multiple scales (different spatial resolutions) and across all temporal frames, ensuring semantic consistency throughout the video. This approach is inherited from SDXL's proven conditioning strategy.