VQGAN-CLIP

Generates images from text prompts by iteratively optimizing a VQGAN latent vector using CLIP guidance. The system encodes text prompts into CLIP embeddings, then repeatedly decodes the latent vector through VQGAN, creates augmented cutouts of the resulting image, scores those cutouts against the text embedding using CLIP's contrastive loss, and backpropagates gradients to update the latent vector toward higher text-image alignment. This runtime optimization approach requires no model retraining and works with pre-trained VQGAN and CLIP models.

Applies artistic styles to existing images by encoding the source image into VQGAN's latent space, then iteratively optimizing that latent representation using CLIP guidance on style-related text prompts (e.g., 'oil painting', 'cyberpunk aesthetic'). The system preserves the original image structure through initialization while steering the optimization toward the desired style via CLIP embeddings, effectively performing style transfer without explicit style loss functions or paired training data.

Implements seed-based reproducibility by setting random number generator seeds for PyTorch and NumPy, ensuring identical results across runs with the same seed and hyperparameters. This enables deterministic generation workflows where the same prompt, seed, and hyperparameters always produce identical images, critical for reproducible research and production systems. Seed control extends to latent initialization, cutout augmentation, and optimization steps.

Implements gradient-based optimization of VQGAN's latent space using PyTorch's autograd system, with custom loss aggregation combining CLIP alignment scores, optional regularization terms, and multi-scale cutout evaluation. The system computes gradients of the aggregated loss with respect to the latent vector, applies gradient clipping and normalization, and updates the latent vector using configurable optimizers (Adam, SGD). This enables fine-grained control over the optimization trajectory and loss composition.

Processes video files by extracting frames, applying CLIP-guided style transfer to each frame sequentially using the previous frame's optimized latent vector as initialization for the next frame. This temporal coherence approach reduces flickering and maintains visual consistency across frames by leveraging frame-to-frame similarity, implemented via the video_styler.sh script that orchestrates frame extraction, per-frame optimization, and frame reassembly into output video.

Supports multiple text prompts with individual weighting factors and optional iteration-based scheduling, allowing users to blend multiple concepts or transition between prompts during generation. The system tokenizes and encodes each prompt separately using CLIP, computes weighted combinations of their embeddings, and optionally adjusts prompt weights across iterations to create smooth transitions or emphasis shifts. This enables complex creative directions like 'start with concept A, gradually shift to concept B' or 'blend three artistic styles with specific weights'.

Evaluates image-text alignment by creating multiple augmented crops (cutouts) of the generated image at different scales and positions, computing CLIP scores for each cutout independently, and aggregating these scores to guide latent optimization. This multi-scale evaluation approach helps the model learn diverse visual features and reduces overfitting to specific image regions, implemented via cutout augmentation pipelines that apply random crops, rotations, and perspective transforms before CLIP evaluation.

Provides flexible initialization of VQGAN's discrete latent space through random sampling, image encoding, or user-specified latent vectors, enabling control over the starting point for optimization. The system can encode existing images into VQGAN's latent space using the encoder, initialize from random noise, or load pre-computed latent vectors. This initialization flexibility enables inpainting-like workflows, seed-based reproducibility, and latent space interpolation experiments.

Exposes fine-grained control over the optimization process through configurable hyperparameters including learning rate, iteration count, step size, and gradient clipping thresholds. Users can adjust these parameters via command-line arguments or configuration files to balance convergence speed, image quality, and computational cost. The system implements standard gradient-based optimization with Adam or SGD solvers, allowing practitioners to tune the optimization trajectory for specific use cases.

Manages loading and caching of pre-trained VQGAN and CLIP model checkpoints from local disk or remote sources (e.g., Hugging Face Model Hub). The system automatically downloads missing models on first run, caches them locally for subsequent runs, and supports custom checkpoint paths for fine-tuned or alternative models. This abstraction enables users to swap models without code changes and supports reproducible model versioning.

Provides a Cog-based containerized inference interface (predict.py) that wraps the VQGAN-CLIP generation pipeline for deployment on Replicate or other container-based inference platforms. The interface exposes generation parameters as Cog input/output schemas, enabling remote API access and scalable cloud deployment without modifying core generation code. This abstraction separates the inference logic from deployment infrastructure.

Allows users to specify output image resolution and aspect ratio, with adaptive scaling of VQGAN's latent space dimensions to match the requested output size. The system computes appropriate latent dimensions based on VQGAN's decoder architecture and the requested resolution, enabling generation at various resolutions without retraining. Supports both square and rectangular aspect ratios with automatic padding or cropping.