Helicone

Helicone operates as a transparent HTTP/HTTPS proxy that intercepts all requests destined for external LLM providers (OpenAI, Anthropic, etc.) without requiring code changes to the application. Requests are routed through Helicone's infrastructure, logged with full request/response metadata, then forwarded to the target provider. The proxy pattern eliminates the need for SDK integration while capturing complete observability data including latency, tokens, costs, and custom properties.

Helicone automatically calculates and aggregates costs across all LLM provider requests by parsing response metadata (token counts, model pricing) and applying provider-specific pricing tables. Costs are tracked at request, user, session, and organization levels, with real-time cost dashboards and historical cost trends. The system supports custom pricing rules for enterprise contracts and volume discounts, enabling accurate chargeback and budget forecasting across heterogeneous provider usage.

Helicone provides a request search interface enabling users to filter logged requests by multiple dimensions (user, session, model, cost range, latency range, custom properties, error status). Filters can be combined using boolean logic and saved as reusable views. Advanced filtering uses HQL queries for complex conditions. Search results display request summaries with drill-down to full request/response details, enabling investigation of specific requests or cohorts.

Helicone supports SAML-based single sign-on (SSO) for enterprise authentication, enabling integration with corporate identity providers (Okta, Azure AD, etc.). The platform implements role-based access control (RBAC) with predefined roles (Admin, Member, Viewer) controlling permissions for dashboard access, configuration changes, and data export. Team management features enable organization of users into projects or teams with separate observability views and cost tracking.

Helicone offers on-premises deployment options for enterprise customers, enabling self-hosted observability infrastructure. Organizations can deploy Helicone on their own infrastructure (Kubernetes, Docker, etc.) with full control over data residency, security, and compliance. Self-hosted deployments support the same features as cloud version (request logging, cost tracking, caching, etc.) with additional customization options for enterprise requirements.

Helicone exposes REST APIs enabling applications to log LLM requests programmatically without using the proxy pattern. Applications can call Helicone APIs directly to log requests, responses, and custom metadata. The API supports batch logging for high-throughput scenarios and includes SDKs for popular languages (Python, JavaScript, etc.). API-based integration enables flexibility for applications that cannot use proxy pattern (e.g., serverless functions, edge computing).

Helicone implements a caching layer that stores LLM responses and matches incoming requests against cached responses using semantic similarity or exact matching. When a request matches a cached entry (same model, parameters, and prompt semantics), the cached response is returned immediately without calling the LLM provider, reducing latency and costs. The cache is provider-agnostic, allowing cached responses from one provider to serve requests intended for another provider if semantically equivalent.

Helicone enforces rate limits at multiple levels (per-user, per-session, per-organization) and automatically throttles requests that exceed configured thresholds. When rate limits are exceeded, Helicone can automatically fall back to alternative LLM providers or queue requests for later processing. The system supports configurable rate limit strategies (token bucket, sliding window) and provides real-time visibility into rate limit consumption and fallback events.

Helicone automatically groups LLM requests into user sessions and tracks user behavior across multiple interactions. Each request can be tagged with custom properties (user ID, feature flag, A/B test variant, etc.) enabling segmentation and cohort analysis. The analytics engine aggregates metrics (request count, total cost, average latency, token usage) by user, session, custom property, and time period, with drill-down capabilities to inspect individual requests within a cohort.

Helicone provides HQL (Helicone Query Language), a custom SQL-like query language enabling users to write custom analytics queries against logged request data. HQL supports filtering, aggregation, and joining across request, user, session, and cost dimensions. Query results can be exported in multiple formats (CSV, JSON) or visualized in custom dashboards. HQL is available on Pro+ tiers and enables advanced analytics without requiring direct database access.

Helicone includes a prompt management system for storing, versioning, and testing prompt templates. Prompts can be tagged with metadata (model, version, status) and organized into collections. The system tracks prompt versions and enables A/B testing different prompt variants against the same input. Prompts can be retrieved via API for use in applications, with version pinning to ensure consistent behavior across deployments.

Helicone enables users to create datasets of test inputs and expected outputs, then evaluate LLM responses against these datasets using custom scoring functions. The system supports multiple evaluation metrics (exact match, semantic similarity, custom scoring) and aggregates scores across dataset runs. Evaluation results are linked to request logs, enabling correlation between prompt/model changes and evaluation performance.

Helicone monitors LLM request metrics (latency, error rate, cost, token usage) and triggers alerts when values exceed configured thresholds or deviate from historical baselines. Alerts can be delivered via webhooks, enabling integration with external notification systems (Slack, PagerDuty, etc.). The system supports multiple alert conditions (threshold-based, anomaly-based, rate-of-change) and alert routing rules based on severity or metric type.

Helicone includes an interactive playground enabling users to test LLM requests in real-time without writing code. The playground supports prompt editing, parameter adjustment (temperature, max tokens, etc.), and model selection. Requests made in the playground are logged and linked to the observability dashboard, enabling debugging of specific requests. The playground supports prompt templating with variable substitution for testing parameterized prompts.