Baserun

Automatically captures complete execution traces for LLM requests including prompts, model parameters, API calls, latency metrics, and token usage across the entire request lifecycle. Implements distributed tracing by instrumenting LLM SDK calls and HTTP interceptors to record request/response pairs with millisecond-precision timestamps, enabling developers to reconstruct exact execution paths and identify performance bottlenecks or failure points in multi-step LLM workflows.

Executes user-defined evaluation functions against LLM outputs to measure quality, correctness, and safety. Supports arbitrary Python/JavaScript functions that can access full request context (input, output, expected result) and return structured scores or pass/fail verdicts. Integrates with common evaluation patterns like BLEU scoring, semantic similarity, fact-checking, and custom business logic, enabling developers to define domain-specific quality metrics without leaving the platform.

Compares current LLM outputs against baseline results from previous runs to detect unintended behavior changes. Stores baseline traces and evaluation results, then runs new test suites against the same inputs and compares outputs using configurable diff strategies (exact match, semantic similarity, evaluation score deltas). Provides visual diffs and statistical summaries to highlight regressions, enabling developers to catch quality degradation before production deployment.

Integrates Baserun evaluations and regression tests directly into CI/CD workflows (GitHub Actions, GitLab CI, Jenkins) to automatically run test suites on code changes and block deployments if quality gates fail. Provides webhook-based triggers, status checks that report pass/fail to version control platforms, and configurable thresholds for blocking merges. Enables developers to define quality requirements (e.g., 'all evals must pass', 'no regressions detected') that are enforced automatically before production deployment.

Provides a repository for storing and organizing test cases (input prompts, expected outputs, evaluation criteria) with version control and metadata tagging. Supports grouping tests into suites, tagging with labels (e.g., 'critical', 'edge-case', 'regression'), and tracking test history across runs. Enables developers to maintain a curated set of test cases that represent important use cases, edge cases, and quality requirements without managing separate files or databases.

Continuously monitors LLM application performance by tracking request latency, token usage, API costs, and error rates across production traffic. Aggregates metrics over time windows (hourly, daily, weekly) and provides dashboards showing performance trends, cost breakdowns by model/endpoint, and anomaly detection for unusual latency or cost spikes. Enables developers to identify performance degradation, cost overruns, and optimization opportunities without manual log analysis.

Enables developers to version prompts and test multiple prompt variants against the same test cases to measure performance differences. Stores prompt history with metadata (author, timestamp, changes), supports side-by-side comparison of outputs from different prompt versions, and integrates with evaluation metrics to quantify which variant performs better. Allows developers to iterate on prompts safely by comparing new versions against baselines before deploying to production.

Enables developers to run the same test suite against multiple LLM models (OpenAI GPT-4, Claude, Cohere, etc.) to compare quality, latency, and cost. Provides side-by-side output comparisons, evaluation score aggregations, and cost-per-test metrics to help developers select the best model for their use case. Supports both commercial APIs and self-hosted models, allowing teams to benchmark proprietary models against public alternatives.

Automatically captures and categorizes errors from LLM requests (API failures, timeouts, invalid outputs, evaluation failures) and provides debugging context including full request traces, error messages, and stack traces. Groups similar errors together to identify patterns and enables developers to drill down into specific error instances to understand root causes. Integrates with issue tracking systems to automatically create tickets for recurring errors.

Provides shared dashboards, reports, and insights that teams can access to understand application quality, performance, and costs. Supports role-based access control (read-only, editor, admin) to manage permissions, enables team members to comment on test results and share findings, and generates automated reports (daily, weekly) summarizing key metrics. Enables non-technical stakeholders (product managers, executives) to understand LLM application health without direct access to traces or code.