Keywords AI

Routes requests to 500+ LLM models across multiple providers (OpenAI, Anthropic, etc.) through a single API endpoint, abstracting provider-specific API differences and authentication. Implements request normalization to convert unified schema to provider-native formats, handling model selection, fallback routing, and cost tracking per request. Two-line integration replaces direct provider API calls with Keywords AI gateway URL.

Automatically captures every LLM request, response, tool call, and intermediate step from production applications via gateway or SDK integration, storing structured traces with full context (prompts, parameters, outputs, latency, cost, errors). Traces are queryable by content, latency, cost, quality scores, tags, and custom metadata. Enables reproduction of production issues by replaying exact request sequences with original parameters.

Accepts trace data in OpenTelemetry format (OTEL), enabling integration with existing observability infrastructure. Keywords AI acts as OTEL collector endpoint, ingesting traces from applications instrumented with OTEL SDKs. Supports OTEL semantic conventions for LLM spans (prompts, completions, tool calls). Traces are converted to Keywords AI format and stored alongside gateway traces. Enables teams to use existing OTEL instrumentation without rewriting code.

Integrates with PostHog analytics platform to track user behavior and correlate with LLM metrics. Sends user events (feature usage, conversions, errors) to PostHog, enabling analysis of how LLM quality/cost impacts user behavior. Supports custom event tracking and user property enrichment. Enables cohort analysis (e.g., 'users with high LLM latency have lower conversion rates').

Sends scheduled webhook payloads containing trace data, metrics, or evaluation results to external systems on a configurable schedule (daily, weekly, etc.). Webhooks can trigger external workflows (data pipelines, notifications, integrations). Payload format is JSON with full trace context. Supports filtering (e.g., 'only send traces with quality score < 0.7'). Webhook delivery guarantees not documented.

Offers self-hosted deployment option for Enterprise tier customers, allowing Keywords AI infrastructure to run on customer's own servers or cloud account. Enables data residency compliance (e.g., data must stay in EU for GDPR). Self-hosted deployment includes all Keywords AI features (gateway, tracing, evaluation, dashboards). Requires customer to manage infrastructure, updates, and security patches. Specific deployment options (Kubernetes, Docker, VMs) not documented.

Supports SAML 2.0 authentication for Enterprise tier customers, enabling integration with corporate identity providers (Okta, Azure AD, etc.). Allows centralized user management and access control through existing identity infrastructure. Supports role-based access control (RBAC) and single sign-on (SSO). SAML is available only on Enterprise tier; Pro/Team tiers use Google OAuth.

Stores prompts as versioned artifacts in Keywords AI UI, allowing teams to create, edit, test, and deploy prompt versions without modifying application code. Each version is immutable and tagged with metadata (author, timestamp, test results). Deployed versions are served through the API gateway, enabling instant rollback to previous versions or A/B testing between versions by routing traffic to different prompt versions.

Runs evaluations against LLM outputs using three judge types: LLM-as-judge (using any model from gateway), code-based judges (custom Python/JavaScript functions), and human review (manual scoring). Evaluations are executed against datasets (production traces or synthetic) and produce quality scores stored alongside traces. Supports batch evaluation of historical traces or real-time scoring of new requests. Evaluation results feed into dashboards and alerting.

Provides drag-and-drop dashboard builder allowing teams to create custom visualizations from trace data using 80+ graph types (line charts, histograms, heatmaps, etc.). Dashboards can display metrics like latency distribution, cost trends, quality scores over time, error rates, token usage, and custom business metrics. Dashboards are queryable (filter by date range, model, user, tags) and can be shared across team members. Real-time updates as new traces arrive.

Monitors trace metrics (quality scores, cost per request, latency percentiles, error rates) and triggers alerts when thresholds are exceeded. Alerts can be configured per metric (e.g., 'alert if p95 latency > 5s' or 'alert if average quality score < 0.7'). Supports multiple notification channels (Slack, webhooks) and automation triggers (UNKNOWN specifics) that can execute actions when alerts fire. Alerts are queryable and can be filtered by severity, metric type, or time range.

Enables A/B testing by splitting traffic between two prompt versions, models, or configurations at the gateway level. Specifies traffic split percentage (e.g., 90% control, 10% variant) and Keywords AI routes requests accordingly. Collects separate metrics (latency, cost, quality scores) for each variant, enabling statistical comparison. Results are displayed in dashboard with significance testing (UNKNOWN if implemented).

Automatically detects and masks personally identifiable information (PII) in traces before storage, replacing sensitive data with placeholder tokens. Supports selective log omission, allowing teams to exclude specific requests or data types from being logged (e.g., 'don't log requests from test users'). Masking rules are configurable per data type (email, phone, credit card, custom patterns). Masked data is not recoverable, enabling compliance with privacy regulations.

Stores LLM provider API keys (OpenAI, Anthropic, etc.) in encrypted vault within Keywords AI infrastructure, eliminating need for applications to manage keys directly. Keys are encrypted at rest and in transit, and access is logged for audit. Supports key rotation and revocation. Applications authenticate to Keywords AI with single API key, which grants access to all provider keys in vault. BYOK (Bring Your Own Keys) ensures provider credentials never leave Keywords AI infrastructure.

Stores evaluation datasets as collections of input-output pairs (prompts with expected outputs, or production traces). Datasets can be created from production traces (sampling real requests) or uploaded as synthetic examples. Datasets are versioned and can be used to run batch evaluations or as ground truth for quality scoring. Supports dataset export in JSONL/CSV format. Pro tier limited to 5 datasets; Team+ unlimited.