Opik

Captures hierarchical spans representing each step in agent execution (LLM calls, tool invocations, intermediate reasoning) and reconstructs them into an interactive timeline view. Uses a span-based tracing model where parent-child relationships preserve execution flow, enabling developers to inspect latency bottlenecks, token usage per step, and failure points across multi-step agent workflows. Supports async execution patterns and distributed agent systems.

Allows developers to define test suites with global rules and item-level assertions that validate LLM application outputs against expected behavior. Tests can be versioned alongside prompts and parameters, and executed against new traces to detect regressions. Assertions are defined declaratively (e.g., 'output must contain keyword X', 'latency < 500ms', 'cost < $0.01') and evaluated automatically when new traces are captured.

Abstracts away differences between LLM providers (OpenAI, Anthropic, Cohere, Ollama, etc.) through a unified SDK interface. Developers can switch models or providers without changing agent code, and Opik handles API differences, token counting, and cost calculation. Supports both cloud-hosted and self-hosted models.

Enables teams to collaboratively annotate failed traces with error categories, root causes, and remediation notes. Annotations are stored alongside traces and can be used to train automated fix generation (Ollie) or identify patterns in failures. Supports multi-user workflows with version history for annotations.

Analyzes failed traces and assertion violations to automatically generate code fixes that address root causes. Ollie (an embedded AI assistant) examines the execution flow, identifies where the agent deviated from expected behavior, and suggests or directly implements fixes (e.g., prompt rewrites, parameter adjustments, tool-calling logic corrections). Generated fixes can be version-controlled and tested against the regression suite before deployment.

Provides a web-based UI where non-technical stakeholders (product managers, QA) can test agents without writing code. Users configure agent parameters (model, temperature, system prompt), invoke the agent with test inputs, and view execution traces and outputs in real-time. Playground sessions are logged as traces and can be added to regression test suites, enabling non-developers to contribute test cases.

Continuously evaluates traces captured from production agents against defined quality metrics and assertion rules. When metrics deviate (e.g., latency spikes, cost increases, assertion failures), Opik triggers alerts via webhooks, email, or Slack. Dashboards display real-time KPIs (success rate, average latency, token usage) with drill-down into individual failing traces for root-cause analysis.

Automatically optimizes prompts by testing variations against defined quality metrics and selecting the best-performing version. Opik claims to use 'seven advanced prompt optimization algorithms' (specifics unknown) that explore the prompt space more efficiently than random search or grid search. Optimization runs are versioned and can be compared side-by-side to understand which prompt changes drove quality improvements.

Aggregates token usage, API costs, and latency metrics across all agent executions and surfaces them in customizable dashboards. Developers can define custom metrics (e.g., cost per successful interaction, latency percentiles) and drill down by model, tool, or time period. Dashboards support filtering, grouping, and export for cost analysis and capacity planning.

Automatically versions prompts, model parameters, and agent configurations alongside code changes. Each trace is tagged with the exact prompt/parameter version that produced it, enabling developers to compare quality across versions and understand the impact of changes. Versions are stored in Git (or compatible VCS) and can be rolled back or branched for experimentation.

Automatically scans agent inputs and outputs for personally identifiable information (PII) and applies content guardrails to prevent unsafe outputs. Detects patterns (email addresses, phone numbers, credit card numbers, etc.) and can redact or flag them in traces. Guardrails can be customized to enforce domain-specific policies (e.g., no medical advice, no financial recommendations).

Automatically generates immutable audit logs of all agent executions, configuration changes, and user actions within Opik. Logs include timestamps, user identities, actions taken (prompt updates, test runs, deployments), and outcomes. Audit logs can be exported for compliance audits and are retained according to configurable policies.