OpenLIT

Automatically intercepts and instruments calls to 30+ LLM providers (OpenAI, Anthropic, Google, Azure, local models) using the OpenTelemetry BaseInstrumentor pattern to patch third-party libraries at runtime. Captures prompts, completions, token usage, latency, costs, and model metadata without code changes, exporting structured traces and metrics via OTLP to any OpenTelemetry-compatible backend. Uses provider-specific wrapper implementations to normalize heterogeneous APIs into OpenTelemetry semantic conventions.

Auto-instruments vector database clients (Qdrant, Chroma, Pinecone, Milvus, Astra, Weaviate) to capture embedding operations, retrieval queries, and vector similarity metrics. Tracks embedding model usage, vector dimensions, retrieval latency, and result cardinality as OpenTelemetry spans and metrics. Integrates with the LLM instrumentation pipeline to correlate RAG retrieval steps with downstream LLM calls for end-to-end observability.

Defines and implements OpenTelemetry semantic conventions for AI operations (LLM calls, embeddings, vector database queries, agent steps) that standardize attribute names, span types, and metric definitions across all SDKs and providers. Semantic conventions enable consistent telemetry collection across heterogeneous LLM providers and frameworks, allowing downstream tools to understand and correlate AI telemetry without provider-specific logic. Conventions are documented in the OpenTelemetry specification and implemented in all SDKs.

Implements W3C Trace Context propagation to correlate traces across multiple services and languages in distributed AI applications. Automatically injects trace context (trace ID, span ID, trace flags) into outgoing requests (HTTP, gRPC) and extracts trace context from incoming requests to maintain trace continuity. Enables end-to-end tracing of requests that span multiple microservices, including LLM calls, vector database queries, and application logic.

Provides a real-time dashboard that streams telemetry data (traces, metrics, logs) from the OpenTelemetry Collector to web clients via WebSocket or Server-Sent Events (SSE). Displays live LLM calls, token usage, latency, and costs as they occur without requiring page refresh. Dashboard includes filtering, search, and drill-down capabilities to explore telemetry in real-time. Enables developers to monitor LLM applications during development and debugging.

Provides batch evaluation capabilities to analyze historical LLM traces stored in the platform, including cost analysis, performance trends, prompt effectiveness, and policy compliance. Supports SQL-like queries on trace data to aggregate metrics by model, provider, user, or custom dimensions. Enables teams to identify optimization opportunities, track performance over time, and audit LLM usage for compliance.

Auto-instruments AI frameworks (LangChain, LangGraph, AutoGen, CrewAI) to capture framework-level operations: chain execution, tool calls, agent reasoning steps, and memory interactions. Instruments at the framework abstraction layer (e.g., LangChain's Runnable interface, LangGraph's StateGraph) to create hierarchical spans that represent the logical flow of AI applications. Automatically correlates framework operations with underlying LLM and vector database calls.

Collects GPU metrics (utilization, memory usage, temperature, power consumption) from NVIDIA GPUs using the OpenTelemetry GPU Collector and exposes them as OpenTelemetry metrics. Integrates with the Python SDK to correlate GPU metrics with LLM inference operations, enabling visibility into hardware resource consumption during model serving. Supports Kubernetes environments via the OpenLIT Operator for automated GPU metric collection across clusters.

Provides a centralized Prompt Hub interface for storing, versioning, and managing prompts used in LLM applications. Enables teams to version prompts, track changes, and retrieve specific prompt versions at runtime via API. Integrates with the Python SDK to automatically capture prompts from LLM calls and store them in the Prompt Hub, enabling prompt-level observability and A/B testing of prompt variations.

Automatically calculates and tracks costs for LLM API calls, embedding operations, and vector database queries based on provider pricing models. Supports cost attribution by model, provider, user, application, and custom dimensions. Integrates with the instrumentation pipeline to capture token counts and model metadata, then applies provider-specific pricing rules to calculate per-call costs. Exports cost metrics as OpenTelemetry metrics for aggregation and analysis.

Provides a rule engine for defining and enforcing policies on LLM outputs, including content filtering, PII detection, toxicity detection, and custom validation rules. Evaluations are defined as rules that can be applied to captured prompts and completions in the OpenLIT platform. Integrates with the instrumentation pipeline to automatically evaluate LLM outputs against defined rules and flag violations. Supports both real-time evaluation during inference and batch evaluation of historical traces.

Provides language-specific SDKs (Python, TypeScript, Go) that implement the same instrumentation architecture and semantic conventions, enabling consistent telemetry collection across polyglot AI applications. Each SDK uses language-native instrumentation patterns (Python decorators/monkey-patching, TypeScript wrappers, Go middleware) to intercept LLM and framework calls. All SDKs export to the same OpenTelemetry backend, enabling unified observability across languages.

Provides a Kubernetes Operator that automates deployment of OpenLIT components (collector, platform) and injects instrumentation into Kubernetes workloads via webhooks. The operator watches for Kubernetes resources (Deployments, StatefulSets) and automatically injects OpenLIT SDKs and OTLP exporter configuration into pod specifications. Enables zero-touch instrumentation of AI applications running in Kubernetes without modifying application code or Helm charts.

Exports telemetry (traces, metrics, logs) to any OpenTelemetry-compatible backend via OTLP protocol, including Grafana Cloud, New Relic, SigNoz, Datadog, and self-hosted OpenTelemetry Collector. Configures OTLP exporter in the SDK initialization to specify backend endpoint, authentication credentials, and export batching parameters. Enables teams to use existing observability infrastructure instead of requiring a separate OpenLIT platform deployment.