CLIP-Interrogator

Converts images into natural language prompts by leveraging OpenAI's CLIP model to compute image embeddings, then uses a learned text encoder to map those embeddings into human-readable descriptions. The system processes uploaded images through CLIP's vision transformer backbone, extracts semantic embeddings, and generates descriptive text that captures visual content in a format suitable for text-to-image models. This enables reverse-engineering of image semantics into prompt form.

Provides a Gradio-based web UI deployed on Hugging Face Spaces that allows users to upload or paste image URLs and receive real-time prompt generation without authentication. The interface handles image preprocessing, manages concurrent requests on shared infrastructure, and streams results back to the browser. Built on Gradio's reactive component system, enabling instant feedback loops between image input and text output.

Implements a neural projection layer that maps CLIP's 512-dimensional image embeddings into a sequence of tokens that a language model can decode into natural language prompts. The architecture uses a learned linear or MLP projection followed by a text decoder (likely a small transformer or LSTM), trained to reconstruct human-written prompts from CLIP embeddings. This enables semantic-preserving conversion from vision embeddings to text without requiring image captioning models.

Accepts images in multiple formats (JPEG, PNG, WebP, GIF, BMP) and URLs, automatically detects format, resizes to CLIP's expected input dimensions (224x224 or 336x336), normalizes pixel values, and applies standard vision preprocessing (center cropping, normalization with ImageNet statistics). Handles edge cases like animated GIFs (extracts first frame), corrupted files (graceful error handling), and various aspect ratios through intelligent resizing strategies.

Executes CLIP forward passes and prompt decoding on Hugging Face Spaces' shared GPU infrastructure with automatic batching and request queuing. Implements inference caching to avoid redundant CLIP embedding computations for identical images, manages GPU memory efficiently by offloading models between requests, and streams results back to the Gradio UI with minimal latency. Leverages CUDA/GPU acceleration for both CLIP's vision transformer and the projection/decoding layers.

Analyzes the generated prompt text to extract key semantic concepts, visual attributes (colors, textures, composition), and style descriptors, then optionally refines the prompt by reweighting terms based on their visual salience in the CLIP embedding space. May implement secondary ranking of keywords by their contribution to the image embedding, enabling users to understand which visual features CLIP considers most important. Produces structured metadata alongside the natural language prompt.

Structures the image-to-prompt conversion as a composable pipeline (image preprocessing → CLIP embedding → projection → text decoding) that can be executed on single images through the web UI or adapted for batch processing through direct API calls or local scripts. The modular architecture separates concerns (vision, embedding, projection, language) enabling reuse of individual components. Supports both synchronous web requests and asynchronous batch jobs with result caching.