VBench

Evaluates video generative models across 16-18 fine-grained dimensions (7 technical quality + 9 semantic understanding + 2 intrinsic faithfulness categories) rather than holistic scoring. Uses a modular evaluation pipeline where each dimension is computed independently via specialized pretrained models (CLIP, optical flow, scene detection, action recognition), then aggregated with human-preference-aligned weighting. The architecture separates concerns: quality metrics (resolution, motion smoothness, flicker) run through video processing pipelines, semantic metrics (object consistency, action fidelity) use vision-language models, and trustworthiness dimensions employ anomaly detection and human preference validation.

Maintains curated, balanced prompt datasets for text-to-video evaluation that ensure consistent, fair model comparison. The prompt suite is organized by semantic categories (objects, actions, scenes, attributes) with stratified sampling to cover diverse generation challenges. Prompts are validated against human preference annotations to ensure they discriminate between model quality levels. The system provides both the original VBench prompt set (used in CVPR 2024 leaderboard) and extended suites for I2V and long-video evaluation, with metadata mapping prompts to evaluation dimensions.

Maintains a public leaderboard for ranking video generation models based on VBench evaluation results. The leaderboard displays both overall scores and dimension-level breakdowns, enabling fine-grained model comparison. Implements score normalization and aggregation logic to ensure fair comparison across different model architectures and training approaches. Supports filtering and sorting by dimension, allowing users to identify models that excel in specific areas (e.g., motion quality vs. semantic consistency). The leaderboard infrastructure handles submission validation, duplicate detection, and result archival.

Implements a modular video processing pipeline that extracts features and metrics from video frames for evaluation. The pipeline includes optical flow computation (using pretrained optical flow networks) for motion analysis, frame-to-frame consistency detection for flicker/jitter measurement, and temporal sampling strategies for efficient processing of long videos. Uses configurable frame sampling (every Nth frame, adaptive sampling based on motion) to balance computational cost and temporal coverage. The pipeline is designed for reusability: computed features (optical flow, frame embeddings) are cached and reused across multiple evaluation dimensions.

Orchestrates evaluation of multiple videos across distributed compute resources by decomposing the pipeline into independent dimension-computation stages. Each dimension is computed via a specialized pretrained model (CLIP for semantic understanding, optical flow networks for motion metrics, action recognition models for temporal consistency). The pipeline uses a modular architecture where videos are processed sequentially through each dimension's computation graph, with intermediate results cached to avoid redundant model inference. Supports both local and distributed execution via configuration, with automatic GPU memory management and batch processing for efficiency.

Learns dimension-level aggregation weights from human preference annotations to ensure computed metrics correlate with human judgment. The system collects human preference labels for generated videos (e.g., 'video A is better than video B'), then uses these labels to calibrate how individual dimension scores (motion smoothness, semantic consistency, etc.) are weighted in the final aggregated score. This approach ensures that the benchmark's scoring aligns with human perception rather than arbitrary metric combinations. VBench-2.0 extends this with anomaly detection to identify videos that violate human preferences, enabling refinement of the metric suite.

Extends evaluation framework to image-to-video generation by adding I2V-specific dimensions that measure motion quality, temporal consistency, and adherence to input image constraints. Implements specialized metrics for evaluating how well generated videos maintain visual consistency with the input image while introducing plausible motion. Uses optical flow analysis to measure motion smoothness, frame-to-frame consistency metrics to detect flickering or jitter, and CLIP-based similarity to ensure the generated video remains faithful to the input image. The I2V evaluation pipeline is integrated into the VBench++ framework with separate prompt suites and dimension definitions.

Extends evaluation to long-form videos (>10 seconds) by adding dimensions that measure temporal coherence across longer sequences, scene consistency, and subject persistence. Implements specialized metrics for detecting temporal discontinuities (abrupt scene changes, subject disappearance), measuring motion consistency over extended durations, and evaluating semantic coherence across multiple scenes. Uses slow-fast network architectures for efficient long-video processing, with configurable temporal window sizes to balance computational cost and temporal coverage. The VBench-Long framework includes separate prompt suites and evaluation pipelines optimized for long-form content.

Evaluates trustworthiness dimensions of video generation models, including robustness to adversarial prompts, bias detection, and safety compliance. Implements metrics for detecting whether models generate harmful content (violence, explicit material), exhibit demographic biases, or produce anomalous outputs in response to edge-case prompts. Uses human anomaly detection to identify videos that violate safety guidelines, combined with automated classifiers for bias and harmful content detection. The VBench-Trustworthiness framework integrates these dimensions into the overall evaluation pipeline with separate scoring and aggregation logic.

Evaluates intrinsic faithfulness — the degree to which generated videos align with input prompts — across 18 fine-grained dimensions organized into 5 categories: object fidelity, spatial relationships, temporal dynamics, appearance consistency, and semantic understanding. Uses multimodal models (CLIP, action recognition, scene understanding) to measure alignment between prompt descriptions and generated video content. Implements specialized metrics for each faithfulness dimension (e.g., object presence detection, spatial relationship verification, action execution quality). VBench-2.0 extends the core framework with these intrinsic faithfulness dimensions, validated against human preference annotations.

Provides a unified CLI for running evaluations, managing configurations, and submitting results to the VBench leaderboard. The CLI supports both VBench (16 dimensions) and VBench-2.0 (18 dimensions) evaluation modes, with configuration-driven dimension selection, batch processing, and result aggregation. Implements subcommands for evaluation (vbench2 evaluate), leaderboard submission (vbench2 submit), and result visualization. The CLI handles model weight downloading, GPU memory management, and error recovery, abstracting away implementation details while exposing key parameters (batch size, dimension selection, output format).

Exposes a Python API for programmatic evaluation of video generation models, enabling integration into custom workflows and metric development. The API provides core classes (VBench, VBench2) that orchestrate evaluation pipelines, with methods for computing individual dimensions, aggregating scores, and retrieving detailed results. Supports custom metric registration via a plugin architecture, allowing researchers to add new dimensions without modifying core code. The API is designed for flexibility: users can evaluate single videos, batch process directories, or integrate evaluation into training loops.