opik

Captures execution traces across LLM applications using language-specific SDKs (Python, TypeScript) that instrument framework-native hooks for LangChain, LlamaIndex, Claude SDK, Pydantic AI, and others. The SDK batches trace events and sends them asynchronously via HTTP to the backend, which persists them in a relational database with Redis Streams for async processing, enabling full visibility into multi-step agent and RAG workflows without code modification.

Executes evaluation metrics against trace data using a pluggable evaluation framework that supports LiteLLM for multi-provider LLM access (OpenAI, Anthropic, Ollama, etc.) and custom Python evaluators. The system runs evaluations asynchronously via a Python backend service, storing results as feedback scores linked to traces, enabling comparison of model outputs against ground truth or custom criteria without manual annotation.

Provides a web-based playground in the frontend that allows users to test prompts and model configurations against LLM providers (OpenAI, Anthropic, Ollama, etc.) in real-time. The playground supports variable substitution, message history, and cost estimation, with results automatically captured as traces for later analysis. Users can iterate on prompts without leaving the browser and save successful configurations as reusable prompts.

Provides a separate Python backend service that runs safety and content filtering checks on LLM inputs and outputs using configurable rules and external safety APIs. Guardrails can be applied at trace collection time or as a post-processing step, with results stored as feedback scores. The system supports custom guardrail definitions and integrates with popular safety frameworks.

Uses Redis Streams as a message queue for asynchronous processing of trace events, enabling decoupling of trace collection from persistence and evaluation. Trace events are published to Redis Streams, consumed by background workers, and processed (persisted, evaluated, guardrails checked) without blocking the SDK. This architecture supports high-throughput trace collection and enables scaling of evaluation and guardrails processing independently.

Manages datasets (collections of input-output pairs) and experiments (runs of an application against a dataset) with automatic comparison of results across runs. The system stores datasets in the relational database, executes applications against them, and computes aggregate metrics (accuracy, latency, cost) across experiment runs, enabling side-by-side comparison of different prompts, models, or configurations without manual result aggregation.

Provides a web-based frontend (React/TypeScript) that renders traces as interactive trees showing span relationships, inputs, outputs, and metadata. The frontend queries the REST API to fetch trace data, renders message content with syntax highlighting for code and JSON, and allows filtering/searching traces by project, tags, and metadata. Users can drill down into individual spans to inspect LLM calls, tool invocations, and intermediate results without leaving the browser.

Automatically extracts token counts from LLM provider responses (OpenAI, Anthropic, etc.) and computes costs using a pricing database that syncs daily with provider pricing data. The system aggregates costs at multiple levels (per trace, per project, per experiment) and stores them alongside trace data, enabling cost analysis without requiring manual token counting or external billing APIs.

Allows users to attach feedback scores and annotations to traces via the UI or API, supporting numeric scores (0-1 range), categorical labels, and free-form text comments. Feedback is stored in the database linked to specific traces and can be used as ground truth for evaluation, as training data for prompt optimization, or for manual quality assessment. The system supports batch feedback operations for bulk annotation of experiment results.

Stores and versions LLM prompts in a centralized registry with support for variables, metadata, and deployment tracking. Prompts can be retrieved by name and version, used in experiments to test prompt variations, and linked to traces for audit trails. The system supports semantic versioning and allows rollback to previous prompt versions without code changes.

Implements multi-tenancy at the database and API levels, with projects as the primary isolation boundary. Each project has its own traces, datasets, and experiments, with role-based access control (RBAC) supporting admin, editor, and viewer roles. Authentication is handled via API keys or OAuth, with audit logging of all data access and modifications for compliance.

Provides a BaseOptimizer framework that supports multiple optimization algorithms (e.g., Bayesian optimization, genetic algorithms) to automatically tune agent hyperparameters (temperature, top_p, system prompts, etc.) based on evaluation metrics. The system runs experiments with different hyperparameter combinations, evaluates results, and suggests optimal configurations without manual trial-and-error.

Exposes all Opik functionality via a REST API with a complete OpenAPI 3.0 specification, enabling automatic SDK generation for Python and TypeScript. The API supports CRUD operations on traces, datasets, experiments, prompts, and feedback, with pagination, filtering, and sorting built-in. The OpenAPI spec is versioned and published, allowing clients to generate type-safe SDKs automatically.