Agenta

Interactive web-based interface for testing and iterating on prompts across multiple LLM providers (OpenAI, Anthropic, etc.) with full version history tracking. Uses a FastAPI backend to manage prompt variants as immutable configurations, storing each iteration in a database with metadata (model, temperature, max_tokens, etc.) and enabling rollback to any previous version. The playground executes prompts against live LLM APIs and caches results for comparison.

Executes parameterized evaluation functions (e.g., exact match, regex, semantic similarity, LLM-as-judge) against test cases in batch mode. The evaluation system uses a plugin-based architecture where evaluators are registered via Python decorators or JSON schema definitions, then executed in isolated processes or containers. Results are aggregated into a structured evaluation report with pass/fail counts, latency metrics, and cost breakdowns per evaluator.

Securely stores API keys and secrets (LLM provider credentials, database passwords, etc.) in an encrypted vault with workspace-scoped access. Secrets are never exposed in logs or UI; only referenced by name in configurations. The system supports secret rotation and audit logging for secret access. Secrets are injected into application code at runtime via dependency injection, preventing hardcoding of credentials.

Manages the deployment lifecycle of LLM applications, allowing teams to promote variants from development to production with traffic routing and rollback capabilities. The system tracks which variant is currently deployed, supports gradual rollout (canary deployment) by routing a percentage of traffic to a new variant, and enables instant rollback to a previous variant if issues are detected. Deployment history is fully audited with timestamps and user information.

Displays evaluation results in an interactive dashboard with side-by-side comparison of variants, metrics visualization (charts, tables), and drill-down capabilities to inspect individual test cases. The dashboard aggregates results from automated and human evaluations, showing pass/fail counts, score distributions, and statistical significance. Users can filter results by evaluator, test case tag, or variant to focus on specific aspects of performance.

Provides a production-ready Docker Compose configuration for self-hosted deployment of the entire Agenta stack (frontend, backend, database, services). The deployment includes environment variable templates for configuring LLM providers, database connections, and authentication. Supports both OSS (open-source) and EE (enterprise edition) deployments with feature flags. Includes migration scripts for upgrading between versions without data loss.

Provides a unified LLM API proxy (via LiteLLM) that abstracts differences between LLM providers (OpenAI, Anthropic, Cohere, etc.) into a single interface. The proxy handles authentication, rate limiting, retry logic, and cost tracking across providers. Applications can switch between providers by changing a configuration parameter without code changes. Supports streaming responses and function calling across different provider APIs.

Web-based interface for human annotators to label LLM outputs against test cases, with support for multiple annotation types (binary choice, multi-class, free-form feedback). The system manages annotator assignments, tracks inter-annotator agreement, and stores annotations in a database with full audit trails. Supports both single-annotator and consensus-based workflows where multiple annotators label the same output and results are aggregated.

Compares two or more prompt variants by running them against the same test set and computing statistical significance of performance differences. The system tracks which variant is deployed, routes test traffic to variants, and aggregates results with confidence intervals and p-values. Uses a Bayesian or frequentist approach (configurable) to determine if observed differences are statistically significant or due to random variation.

Automatically instruments LLM application code to collect traces (spans) for each API call, prompt execution, and evaluation step. Uses OpenTelemetry SDK to emit traces to a backend (Jaeger, Datadog, etc.) with support for custom attributes (model, prompt version, cost). Traces include latency, token counts, error information, and cost estimates per LLM call. The system provides a trace viewer in the web UI and exports traces for external analysis.

Provides a Python library (@app, @run, @evaluator decorators) for defining LLM applications as functions with automatic instrumentation, parameter management, and execution sandboxing. The SDK handles prompt templating, LLM API calls, and result caching. Applications defined with decorators are automatically registered with the backend and can be invoked via the web UI or API without code changes. Supports dependency injection for configuration and secrets management.

Stores and manages collections of test cases (input prompts + expected outputs) in a database with versioning and metadata. Test cases can be imported from CSV/JSON files, created manually via the UI, or generated programmatically. Each test case is immutable once created; new versions are tracked separately. The system supports filtering, searching, and organizing test cases by tags or categories. Test sets can be linked to evaluations and A/B tests for reproducible benchmarking.

Manages multiple configurations (variants) of the same LLM application, each with different prompts, models, or hyperparameters. Variants are stored as immutable configuration objects in the database with full lineage tracking. The system supports parameter templates (e.g., {{variable_name}}) in prompts, allowing dynamic substitution at runtime. Variants can be compared side-by-side in the UI and promoted to production via a deployment workflow.

Automatically tracks and aggregates costs for each LLM API call based on input/output token counts and model pricing. The system maintains a pricing database for common models (GPT-4, Claude, etc.) and allows custom pricing configuration. Costs are aggregated at multiple levels: per request, per variant, per evaluation run, and per organization. The dashboard displays cost breakdowns and trends over time, enabling cost optimization analysis.

Manages multi-tenant isolation via organizations and workspaces, with role-based access control (RBAC) for users. Each organization has multiple workspaces, and each workspace contains applications, test sets, and evaluations. Users are assigned roles (admin, editor, viewer) with corresponding permissions. The system enforces data isolation at the database level, ensuring users can only access resources in their assigned workspaces.