MagicTime

Generates time-lapse videos depicting physical transformations (plant growth, construction, melting) by conditioning a modified Stable Diffusion v1.5 base model with specialized Magic Adapters (spatial and temporal variants) and a Magic Text Encoder trained on metamorphic video datasets. The pipeline encodes text prompts through the Magic Text Encoder, guides diffusion-based frame generation with temporal coherence constraints via the Motion Module, and compiles output frames into coherent video sequences that maintain object identity across significant visual changes.

Applies visual style transfer to generated videos by composing DreamBooth fine-tuned models with the base diffusion pipeline, allowing users to select from pre-trained style variants that define aesthetic properties (e.g., oil painting, photorealistic, anime) without retraining the entire model. The system loads style-specific DreamBooth checkpoints and integrates them into the diffusion sampling process, enabling consistent stylistic rendering across all generated frames.

Combines Magic Adapter S (spatial detail focus) and Magic Adapter T (temporal coherence focus) during generation to provide fine-grained control over the balance between visual detail and temporal smoothness. The adapters operate on different aspects of the diffusion process—spatial adapter enhances object details and textures, temporal adapter constrains frame-to-frame consistency—allowing users to tune the trade-off between visual quality and temporal stability.

Ensures temporal consistency across generated video frames by integrating a dedicated Motion Module that operates on latent representations during the diffusion process. The Motion Module constrains frame-to-frame optical flow and appearance consistency, preventing temporal flickering and ensuring smooth transitions between frames depicting transformations. This component works in parallel with spatial diffusion, applying temporal constraints at each sampling step.

Encodes text prompts into embeddings optimized for metamorphic video generation by using a specialized encoder trained on time-lapse and transformation-focused datasets. Unlike standard CLIP encoders, the Magic Text Encoder learns to represent temporal transformation semantics (growth, melting, construction) and physical process descriptions, enabling the diffusion model to better understand and generate videos depicting meaningful changes over time.

Provides a web-based interface (app.py) for video generation with interactive controls for style selection, prompt input, and parameter tuning (dimensions, frame count, seed, sampling steps). The UI integrates the MagicTimeController class to handle model initialization, loading, and generation orchestration, enabling users to adjust parameters and preview results without command-line interaction or code modification.

Enables programmatic video generation through a command-line interface (inference_magictime.py) that accepts YAML configuration files specifying model components, generation parameters, and input/output paths. The CLI supports batch processing of multiple prompts from CSV, JSON, or TXT files, allowing users to define complex generation workflows, optimize settings, and automate video production pipelines without manual UI interaction.

Manages loading and composition of multiple model components (base model, Motion Module, Magic Adapters, DreamBooth models) through a checkpoint system that tracks model paths and versions. The system loads components on-demand, caches them in memory, and allows dynamic composition of different model variants without restarting the application, enabling efficient resource utilization and flexible model experimentation.

Provides data preprocessing utilities for creating metamorphic video datasets by extracting frames from source videos and generating captions using vision-language models. The system processes raw video files into frame sequences and associates them with text descriptions of the transformations, enabling the creation of training data for fine-tuning or evaluating metamorphic video generation models.

Provides documentation and examples for crafting effective prompts that describe metamorphic transformations, including guidance on temporal language, physical process descriptions, and transformation-specific keywords. The system helps users understand how to phrase prompts to maximize model understanding of growth, melting, construction, and other time-lapse phenomena, improving generation quality through better prompt semantics.

Enables users to customize video generation through YAML configuration files that specify model components, generation parameters (resolution, frame count, sampling steps, guidance scale), and style selections. The configuration system decouples user preferences from code, allowing non-technical users to modify generation behavior by editing configuration files without understanding the underlying implementation.