Langfuse

Captures end-to-end traces of LLM API calls, including latency, token usage, costs, and model parameters across multiple providers (OpenAI, Anthropic, Cohere, etc.). Works via SDK instrumentation that wraps LLM client libraries and automatically extracts request/response metadata without requiring manual logging code. Traces are structured hierarchically to capture nested calls within agents or chains.

Manages prompt templates as versioned artifacts with built-in support for A/B testing across variants. Prompts are stored in a centralized registry with metadata (model, temperature, max_tokens), and the system tracks which prompt version was used for each LLM call. Enables side-by-side comparison of prompt performance metrics (latency, cost, quality scores) across versions.

Provides language-specific SDKs (Python and Node.js) that integrate with LLM client libraries (OpenAI, Anthropic, LangChain, etc.) via automatic instrumentation. SDKs use library-specific hooks (e.g., monkey-patching, middleware) to intercept LLM calls and extract metadata without requiring code changes. Supports both synchronous and asynchronous execution.

Enables multiple team members to collaborate on prompt development with version control, comments, and approval workflows. Prompts are stored in a centralized registry with full history, and changes can be reviewed before deployment. Supports branching and merging of prompt variants, and integrates with CI/CD pipelines for automated testing and deployment.

Provides a framework for defining and executing evaluation functions against LLM outputs, including both automated scoring (via LLM-as-judge, regex, semantic similarity) and manual human feedback. Evaluation results are stored alongside traces and aggregated into dashboards. Supports custom evaluation logic via Python functions or LLM-based scoring with configurable rubrics.

Aggregates trace and evaluation data into real-time dashboards showing key metrics (latency, cost, token usage, error rates, quality scores) with filtering by model, prompt version, user, and custom tags. Uses time-series aggregation to compute metrics at configurable intervals (1min, 5min, 1hour) and supports custom metric definitions via SQL-like queries or pre-built templates.

Automatically calculates API costs for LLM calls by parsing provider pricing tables (OpenAI, Anthropic, Cohere, etc.) and token counts from responses. Costs are attributed to traces and aggregated by model, prompt version, user, or custom dimensions. Supports cost forecasting based on historical usage patterns.

Groups LLM traces into logical sessions or user interactions, enabling analysis of multi-turn conversations and user journeys. Traces within a session are linked via session_id metadata and can be filtered/aggregated together. Supports custom session definitions (e.g., conversation threads, user requests) and enables tracking of session-level metrics (total cost, total latency, success rate).

Provides utilities for extracting structured data from LLM outputs and validating against schemas (JSON Schema, Pydantic models). Includes automatic retry logic with prompt refinement if validation fails, and tracks validation success rates as a quality metric. Supports both synchronous validation and async batch processing.

Enables running LLM calls and evaluations over large datasets in batch mode, with progress tracking and result aggregation. Supports parallel execution with configurable concurrency limits, and integrates with the evaluation framework to compute aggregate metrics (accuracy, F1, BLEU) across the dataset. Results are stored and queryable alongside production traces.

Provides REST and GraphQL APIs for querying traces, evaluations, and metrics programmatically. Supports filtering by model, prompt version, user, tags, and time range, and enables exporting data for external analysis. APIs are paginated and support bulk operations (e.g., bulk evaluation, bulk tagging).

Supports self-hosted deployment via Docker containers, with optional PostgreSQL backend for data persistence. All trace and evaluation data can be stored on-premises, with no data sent to external services. Includes deployment guides for Kubernetes, Docker Compose, and cloud platforms (AWS, GCP, Azure).