LangSmith

Captures hierarchical execution traces across LLM calls, tool invocations, and chain steps by instrumenting LangChain runtime with automatic span creation. Uses OpenTelemetry-compatible tracing protocol to serialize traces with full context (inputs, outputs, latency, tokens, errors) and renders interactive flame graphs and dependency DAGs in the web UI. Traces are persisted server-side with queryable metadata for debugging multi-step agent executions.

Runs evaluation logic against captured traces by executing user-defined Python functions (evaluators) that score LLM outputs against ground truth or heuristics. Evaluators receive the full trace context (input, output, intermediate steps) and return numeric scores or categorical judgments. Results are aggregated across evaluation runs and compared against baseline traces to detect regressions in model behavior or output quality.

Monitors captured traces for anomalies (high latency, token count spikes, error rates, evaluation score drops) and triggers alerts via email, Slack, or webhooks. Supports custom alert rules based on trace metrics, evaluation results, or cost thresholds. Alerts include trace context and links to LangSmith UI for investigation. Integrates with incident management systems (PagerDuty, Opsgenie) for escalation.

Exposes REST and GraphQL APIs for querying traces, running evaluations, managing datasets, and accessing evaluation results programmatically. Enables building custom dashboards, integrating with external analysis tools, or automating evaluation workflows. APIs support filtering, pagination, and bulk operations. Authentication via API keys with role-based access control.

Stores and versions evaluation datasets (input-output pairs, test cases) with metadata tagging and split management. Datasets can be created by uploading CSV/JSON, importing from traces, or building interactively in the UI. Supports versioning with change tracking, enabling reproducible evaluation runs across dataset versions. Datasets are linked to evaluation runs for traceability.

Centralized registry for storing, versioning, and deploying prompt templates with metadata (model, temperature, system instructions). Prompts are versioned with change tracking and can be tagged (e.g., 'production', 'experimental'). Supports A/B testing by running evaluation against multiple prompt versions simultaneously and comparing results. Prompts can be fetched at runtime via API for dynamic prompt selection.

Provides a web UI for human annotators to review traces, provide feedback (ratings, corrections, labels), and flag problematic outputs. Annotation tasks are organized in queues with filtering and prioritization. Feedback is stored and linked back to traces for retraining or evaluation refinement. Supports custom annotation schemas (free-form text, multiple choice, ratings) and role-based access control.

Indexes trace inputs, outputs, and metadata for semantic search using embeddings. Enables finding similar traces or dataset examples by natural language query (e.g., 'traces where the model failed to answer math questions'). Search results are ranked by relevance and can be filtered by metadata tags, date range, or evaluation scores. Supports both keyword and semantic search modes.

Aggregates token counts and API costs across all captured traces, broken down by model, chain, date, and custom tags. Provides dashboards showing cost trends, per-chain cost breakdown, and token efficiency metrics. Integrates with LLM provider pricing (OpenAI, Anthropic, etc.) to calculate actual costs. Supports cost attribution by user, project, or custom dimension for chargeback or optimization.

Provides language-specific SDKs (Python, TypeScript/JavaScript) that automatically instrument LangChain chains and agents via decorators, context managers, or monkey-patching. Developers add a single import and API key configuration; trace capture happens automatically without modifying chain code. SDKs handle serialization, batching, and async submission of traces to LangSmith backend with configurable sampling and filtering.

Enables running the same evaluation dataset against multiple LLM models (GPT-4, Claude, Llama, etc.) and comparing results side-by-side. Supports batch evaluation across model variants with consistent evaluation metrics. Results are displayed in comparison tables showing performance deltas, cost differences, and latency metrics. Supports custom model configurations (temperature, system prompts) per model variant.

Captures user feedback on LLM outputs in production (thumbs up/down, corrections, ratings) and links it back to traces for analysis. Feedback is aggregated to identify patterns in model failures or user preferences. Supports exporting feedback-labeled traces as fine-tuning datasets or for retraining evaluation models. Enables closed-loop improvement by measuring whether model changes reduce negative feedback.