PromptBench

Provides a factory-pattern-based Model System that abstracts heterogeneous LLM APIs (OpenAI, Anthropic, local models, etc.) behind a single LLMModel interface, enabling consistent model instantiation and inference regardless of underlying provider. Uses a registry-based approach where model names map to concrete implementations, eliminating boilerplate for API-specific authentication and request formatting.

Extends the Model System to support Vision-Language Models (VLMs) through a dedicated VLMModel factory class that handles image input preprocessing, multimodal tokenization, and provider-specific vision APIs (CLIP, GPT-4V, LLaVA, etc.). Abstracts away image encoding, resolution handling, and vision-specific parameters behind the same unified interface as text-only models.

Provides visualization utilities that generate charts, heatmaps, and interactive plots showing model performance across datasets, techniques, and perturbation levels. Includes analysis tools for understanding robustness degradation patterns, identifying failure modes, and comparing prompt engineering technique effectiveness. Visualizations support both static (matplotlib) and interactive (plotly) output formats.

Provides extension points and base classes that enable users to add custom models, datasets, attack methods, and evaluation metrics without modifying core framework code. Uses inheritance-based extension pattern where custom implementations extend base classes (LLMModel, Dataset, AttackMethod, Metric) and register themselves with the framework. Includes documentation and examples for implementing custom components.

Implements a hierarchical attack system that generates adversarial prompts at four granularity levels (character, word, sentence, semantic) using attack methods like DeepWordBug, TextFooler, BertAttack, CheckList, and StressTest. Each attack level uses different perturbation strategies: character-level attacks modify individual characters or introduce typos, word-level attacks substitute semantically similar words, sentence-level attacks restructure syntax, and semantic-level attacks use human-crafted adversarial examples. The system maintains semantic equivalence while degrading model performance to measure robustness.

Implements DyVal, a dynamic evaluation framework that generates evaluation samples on-the-fly with controlled complexity levels to mitigate test data contamination. Rather than using static benchmark datasets, DyVal generates samples for four reasoning types (Arithmetic, Boolean Logic, Deduction Logic, Reachability) with parameterized difficulty, ensuring models cannot memorize evaluation data. The system controls complexity through parameters like number of operations, variable counts, or graph sizes, enabling systematic evaluation of reasoning capabilities across difficulty ranges.

Implements PromptEval, an efficient evaluation method that predicts performance on large datasets using performance data from a small sample, reducing computational cost of evaluating multiple prompt variations. The system uses statistical inference from a small sample (e.g., 100 examples) to estimate performance on the full dataset (e.g., 10,000 examples), enabling rapid iteration over prompt engineering techniques without evaluating every prompt on every example. Maintains statistical validity through confidence intervals and sample size recommendations.

Implements a library of prompt engineering methods including Chain-of-Thought (CoT), Emotion Prompt, Expert Prompting, and other advanced techniques that modify prompts to improve model reasoning and performance. Each technique is implemented as a prompt transformation that injects reasoning patterns, emotional context, or role-based framing into the original prompt. The system allows composition of multiple techniques and systematic evaluation of their individual and combined effects on model performance.

Implements Meta Probing Agents (MPA), an automated system that discovers and characterizes model capabilities through systematic probing. The MPA framework uses agents to generate targeted probes (test cases) that explore model behavior boundaries, identify capability gaps, and characterize performance patterns across different input types and complexity levels. Agents iteratively refine probes based on model responses to discover what the model can and cannot do.

Implements a DatasetLoader class that provides unified access to diverse evaluation datasets (GLUE, MMLU, BIG-Bench Hard, etc.) with automatic downloading, caching, and preprocessing. The loader abstracts away dataset-specific formats, splits, and preprocessing requirements, enabling consistent dataset handling across different benchmarks. Supports both language datasets and vision-language datasets with automatic format normalization.

Implements a comprehensive metrics system (eval.py) that computes task-specific evaluation metrics including accuracy, F1, BLEU, ROUGE, and custom metrics for different task types (classification, generation, reasoning). The system automatically selects appropriate metrics based on task type and dataset, handles edge cases (empty predictions, mismatched lengths), and provides detailed metric breakdowns by example and category. Supports both exact-match and fuzzy matching for generated text.

Provides a leaderboard system that aggregates evaluation results across multiple models, datasets, and prompt engineering techniques, enabling comparative analysis and ranking. The leaderboard tracks model performance over time, supports filtering by dataset/technique/model, and generates visualizations of performance trends. Results are stored in a structured format that enables querying and statistical comparison across runs.