Promptimize

Encapsulates individual prompts with associated evaluation logic by creating test cases that pair a prompt template with one or more scoring functions. Each prompt case returns a success rate between 0 and 1, enabling structured assessment of LLM responses against defined criteria. The framework uses a configuration-as-code approach where evaluation functions are first-class Python callables that process LLM responses deterministically.

Manages collections of prompt cases and orchestrates their execution across different LLM engines, models, and parameter configurations. The Suite component aggregates multiple prompt cases, handles execution flow, and tracks results. It supports weighted prompts (assigning importance to specific cases) and categorization for granular reporting. Execution is optimized to only reassess what has changed between iterations, minimizing API costs.

Tracks which prompt cases have changed between runs and only re-evaluates modified cases, minimizing API costs and execution time. The framework maintains execution history and compares current cases against previous runs to identify changes. Unchanged cases reuse cached results, while modified cases are re-executed. This capability is particularly valuable for iterative prompt development where only a few cases change per iteration.

Provides a CLI for executing prompt suites, generating reports, and managing evaluations without writing Python code. The CLI supports running suites, comparing results, exporting reports, and triggering human reviews. This capability enables non-developers (prompt engineers, product managers) to run evaluations and access results through a simple command-line interface.

Enables testing the same prompt suite across different LLM models (GPT-4, Claude, Llama) and inference engines (OpenAI, Anthropic, Ollama) without modifying the suite definition. The framework abstracts LLM interactions through a provider interface, allowing cases to be executed against any supported model. Results are aggregated by model, enabling comparison of how different models respond to the same prompts.

Provides a framework for defining evaluation functions that assess LLM responses and return normalized scores between 0 and 1. The evaluation system accepts arbitrary Python callables that can implement rule-based scoring, regex matching, semantic similarity, or custom business logic. Functions receive the LLM response as input and must return a float representing success rate. The system supports composing multiple evaluations per prompt case for multi-criteria assessment.

Systematically generates different prompt formulations from a base template by applying transformations, parameter substitutions, or AI-powered suggestions. The framework supports template-based prompting where variables are injected into prompt strings, enabling exploration of different phrasings, instruction styles, or context variations. Advanced features include AI-powered generation of additional test cases to expand the variation space.

Compiles and analyzes results from prompt suite executions, generating structured reports that compare performance across cases, categories, and models. Reports aggregate evaluation scores, track success rates, and enable side-by-side comparison of prompt variants. The reporting system supports categorization (grouping related prompts) and weighted scoring to reflect business priorities. Reports can be exported and analyzed programmatically or visualized for stakeholder review.

Enables manual review and override of automated evaluation results when human judgment is needed. The framework supports marking specific prompt cases for human review, storing reviewer feedback, and allowing manual score adjustments. This capability bridges the gap between automated evaluation (which may be incomplete or incorrect) and human expertise, enabling hybrid evaluation workflows where automated scoring is validated or corrected by domain experts.

Provides pre-run and post-run hooks that allow customization of execution behavior without modifying core framework code. Hooks enable injecting custom logic at key points in the execution lifecycle (before prompt execution, after response received, before evaluation, after scoring). This extensibility pattern allows teams to integrate custom preprocessing, response filtering, logging, or side effects into the evaluation pipeline.

Performs operations on LLM responses before evaluation, enabling normalization, extraction, or transformation of raw model outputs. Response processors can clean formatting, extract structured data (JSON, tables), apply regex transformations, or filter irrelevant content. This capability decouples response transformation from evaluation logic, allowing the same processor to be reused across multiple evaluation functions.

Assigns importance weights to individual prompt cases, allowing certain prompts to be prioritized in reporting and analysis. Weights influence how prompt performance is aggregated in reports and can reflect business priorities (e.g., 'this prompt is critical for user experience'). The weighting system enables teams to track performance of high-impact prompts separately from experimental variants, without requiring separate test suites.

Groups related prompt cases into logical categories, enabling granular performance tracking and reporting by category. Categories allow teams to analyze performance across dimensions (e.g., 'tone', 'length', 'structure') without creating separate suites. Reports can be filtered, aggregated, or compared by category, providing insights into which prompt characteristics drive performance.