Promptmetheus

Enforces a compositional prompt structure decomposing prompts into discrete, reusable sections (Context → Task → Instructions → Samples → Primer) that can be independently authored, versioned, and substituted. Each section is treated as a modular building block allowing variant generation without rewriting entire prompts. The system maintains section-level metadata and enables LEGO-like recombination across prompt variants.

Executes a single prompt variant against multiple LLM providers and models simultaneously by injecting test datasets (context variables) into the prompt template, collecting completions from all models in parallel, and aggregating results for comparative analysis. The system dispatches API calls to 15 different provider endpoints, handles asynchronous completion collection, and correlates results by model and variant for statistical comparison.

Abstracts away provider-specific API differences by implementing unified interface supporting 15 LLM providers (Anthropic, OpenAI, DeepMind, Mistral, Perplexity, xAI, DeepSeek, Cohere, Groq, Fetch AI, OpenRouter, AI21 Labs, Venice, Moonshot AI, Deep Infra) and 150+ models. Credential management stores API keys securely (encryption mechanism unknown) and enables users to add/remove providers without code changes. Provider selection is decoupled from prompt definition, allowing same prompt to be tested against different providers.

Provides UI for configuring model-specific parameters (temperature, top_p, max_tokens, frequency_penalty, presence_penalty, etc.) for each model in batch tests. Parameter configurations are persisted and reusable across test runs, enabling systematic exploration of parameter space. The system maintains parameter presets (e.g., 'creative', 'precise', 'balanced') that can be applied to multiple models.

Maintains complete version history of prompt sections and variants with timestamped changelogs, enabling rollback to previous versions and tracking design decisions across iterations. Each version captures section content, variable definitions, and metadata. The system supports branching variants (testing different section combinations) while maintaining lineage to parent versions, allowing comparison of performance across versions.

Provides dual evaluation pathways: (1) manual quality assessment where users rate completions on custom scales (e.g., 1-5 stars, pass/fail), and (2) automated constraint validation via custom evaluators that programmatically assess completions against defined criteria. Custom evaluators execute against completion results (implementation language/format unknown) and produce pass/fail or scored outputs. Ratings are aggregated into statistical summaries by model and variant.

Supports two-tier variable scoping: project-level variables (shared across all prompts in a project, e.g., company name, API endpoint) and prompt-level variables (specific to individual prompts, e.g., user query, context). Variables are defined as key-value pairs and substituted into prompt templates using placeholder syntax (format unknown). During batch testing, dataset rows are injected as variable bindings, enabling dynamic context injection without prompt rewriting.

Automatically calculates API costs for each completion based on model pricing, input token count, and output token count. Costs are aggregated by model, variant, and dataset to provide per-completion and batch-level expense summaries. The system maintains pricing data for 150+ models across 15 providers and updates pricing as providers change rates. Cost estimates are displayed during batch test planning to enable cost-aware model selection.

Analyzes multi-step prompt chains (agents/workflows) to identify error propagation and compounding effects where failures in early steps cascade through downstream steps. The system models chain execution paths, calculates cumulative error probability, and provides optimization recommendations to reduce failure rates. Analysis includes identifying bottleneck steps with highest failure rates and suggesting prompt modifications to improve reliability.

Analyzes completion results across models, variants, and test cases to identify statistical patterns and correlations. The system computes metrics like success rate by model, quality distribution by variant, and correlations between input characteristics (e.g., prompt length, variable values) and output quality. Visualizations highlight patterns (e.g., 'Claude performs 20% better on reasoning tasks') enabling data-driven prompt optimization decisions.

Provides user-level workspace isolation where each user has a private workspace containing their prompts, datasets, and test results. Team accounts enable shared workspaces where multiple users can access and edit the same prompts and datasets. Real-time collaborative editing allows simultaneous edits (conflict resolution mechanism unknown). Shared prompt library enables teams to publish reusable prompts for organization-wide consumption.

Organizes prompts, datasets, and test results into projects as top-level containers. Each project has a dashboard displaying relevant statistics (e.g., number of prompts, recent test runs, average quality scores). Completion history maintains full metadata for each test execution including timestamp, model, variant, dataset, results, and ratings. Projects enable logical grouping of related prompts and datasets for organizational clarity.