Director

Coordinates 25+ specialized agents (VideoGenerationAgent, TextToVideoAgent, AudioAgent, SearchAgent, etc.) through a reasoning engine that interprets natural language commands and routes them to appropriate agents based on task decomposition. Each agent inherits from BaseAgent, defines JSON schemas for inputs, implements business logic via run() methods, and communicates status through OutputMessage objects and WebSocket emissions. The reasoning engine (backend/director/core/reasoning.py) handles agent selection, parameter binding, and execution sequencing.

Translates natural language prompts into video generation requests by routing to 18+ integrated AI services (OpenAI, Anthropic, StabilityAI, ElevenLabs, etc.) through a unified tool interface. The VideoGenerationAgent and TextToVideoAgent classes implement provider-specific logic while abstracting differences via a common parameter schema. Requests flow through backend/director/tools/ai_service_tools.py which handles API calls, response parsing, and error handling. Generated videos are automatically stored in VideoDB infrastructure for indexing and retrieval.

Provides organizational primitives for managing video collections through VideoDB's collection system. Users can create collections, organize videos by tags/metadata, and perform bulk operations (search, edit, delete) across collections. Collections are persisted in VideoDB and accessible via the API. Supports hierarchical organization (nested collections) and sharing/permission controls.

Implements error handling at multiple levels: agent-level try-catch blocks, provider fallback logic, and user-facing error messages. When an agent fails, the system attempts fallback strategies (e.g., use alternative provider, retry with different parameters) before surfacing errors to the user. Error context (stack traces, provider responses, input parameters) is logged for debugging. Partial failures in multi-agent workflows are handled gracefully, allowing subsequent agents to proceed with available data.

Provides a plugin architecture for developers to create custom agents by extending BaseAgent (backend/director/agents/base.py). Custom agents define JSON parameter schemas, implement run() methods, and integrate with the existing tool ecosystem. The framework handles parameter validation, execution lifecycle, status communication, and WebSocket streaming. Documentation and examples guide developers through agent creation, testing, and deployment.

Supports asynchronous execution of long-running tasks (video generation, transcription, editing) through a job queue system. Jobs are submitted with parameters, assigned unique IDs, and processed asynchronously by backend workers. Users can poll job status or subscribe to WebSocket updates. Completed jobs are stored with results and metadata. Supports job cancellation, retry on failure, and priority queuing.

Enables searching video collections using natural language by leveraging VideoDB's native indexing and semantic understanding. The SearchAgent (backend/director/agents/) accepts natural language queries, translates them into VideoDB search parameters, and returns ranked results with relevance scores. Internally uses embeddings-based retrieval (memory-knowledge layer) combined with metadata filtering. Results are streamed back to the frontend via WebSocket with progressive refinement as more results are indexed.

Processes video audio to generate timestamped transcripts with speaker identification using the TranscriptionAgent (backend/director/agents/transcription.py). Internally routes to external speech-to-text providers (OpenAI Whisper, AssemblyAI, etc.) via the AI service tools layer. Transcripts are stored as metadata in VideoDB, enabling downstream search, dubbing, and content analysis. Supports multiple languages and automatic language detection.

Generates dubbed audio in target languages by combining transcription, translation, and text-to-speech synthesis. The AudioAgent and DubbingAgent classes orchestrate this pipeline: extract transcript → translate to target language → synthesize speech via ElevenLabs or similar providers → replace original audio track. Maintains speaker voice characteristics and emotional tone through provider-specific voice cloning parameters. Dubbed videos are stored back in VideoDB with language metadata.

Enables programmatic video editing through the FrameAgent (backend/director/agents/frame.py) which accepts natural language editing commands and translates them into frame-level operations. Supports trimming, concatenation, overlay insertion, effect application, and frame extraction. Internally uses FFmpeg or similar video processing libraries for codec-agnostic manipulation. Edited videos are re-encoded and stored in VideoDB with edit history metadata.

Maintains conversation state across multiple user interactions through the Session Management system (backend/director/core/session.py). Each session stores user context, previous agent outputs, video references, and conversation history. Sessions are persisted in a database (likely SQLite or PostgreSQL based on backend/requirements.txt) and retrieved on subsequent requests. WebSocket connections maintain real-time session updates, enabling progressive result streaming and live agent status updates.

Handles video file uploads from multiple sources (local files, URLs, YouTube links) and automatically ingests them into VideoDB infrastructure. The upload pipeline extracts metadata (duration, resolution, codec, frame rate), generates thumbnails, and initiates transcription and indexing. Supports resumable uploads for large files and progress tracking via WebSocket. Uploaded videos are immediately available for search, editing, and downstream processing.

Abstracts LLM interactions across multiple providers (OpenAI, Anthropic, Ollama, etc.) through a unified interface in the LLM integration layer. Agents call LLM methods without knowing which provider is active, enabling easy switching or fallback. Configuration is centralized in environment variables or config files. Supports streaming responses for real-time output, token counting for cost estimation, and provider-specific features (function calling, vision, etc.).

Provides real-time visibility into agent execution through WebSocket connections that stream progress updates, intermediate results, and status changes. The Handler Architecture (backend/director/handler.py) manages WebSocket lifecycle and message routing. Agents emit OutputMessage objects containing status, progress percentage, and partial results. Frontend receives updates in real-time, enabling live progress bars, streaming results, and early cancellation of long-running tasks.