gateway

Routes incoming requests across 70+ AI providers (OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, Azure OpenAI, Cohere, etc.) using configurable strategies including fallback chains, load balancing, and conditional routing. Implements recursive target orchestration via tryTargetsRecursively() that attempts providers sequentially with exponential backoff retry logic (up to 5 attempts), automatically falling back to next provider on failure. Supports single-target, fallback, and load-balanced modes with provider-specific request/response transformation.

Abstracts provider-specific API differences by transforming incoming requests to provider-native formats and normalizing responses back to OpenAI-compatible schema. Each provider (OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, Azure OpenAI, Cohere) has dedicated transformation logic that maps request parameters (model, messages, temperature, etc.) to provider-specific payloads and transforms provider responses into unified format. Handles streaming responses, token counting, and function-calling schemas across heterogeneous provider APIs.

Built on Hono lightweight web framework supporting deployment across multiple runtime environments: Node.js, Cloudflare Workers, Bun, and Deno. Single codebase compiles to each runtime with minimal changes, enabling deployment flexibility. Runtime-specific features (e.g., real-time SSE log streaming) are conditionally available. Supports both HTTP server mode (Node.js, Bun) and serverless/edge function mode (Cloudflare Workers, Deno). Configuration and provider integrations are runtime-agnostic.

Routes requests to appropriate provider endpoints based on model identifier, abstracting provider-specific endpoint structures. Supports model aliasing so applications can reference models by friendly names (e.g., 'gpt-4') and gateway maps to provider-specific model IDs (e.g., 'gpt-4-turbo-preview'). Handles provider-specific endpoint variations (Azure endpoint structure, Bedrock model ARNs, etc.) transparently. Enables model switching without application code changes by updating configuration.

Normalizes function-calling schemas across providers with different function definition formats (OpenAI, Anthropic, Google, etc.). Transforms function definitions from OpenAI format to provider-native format before transmission, and transforms provider-native function calls back to OpenAI format in responses. Supports function calling for providers that implement it, with graceful degradation for providers without native function-calling support. Handles tool_choice parameter mapping and function execution context.

Routes requests to different providers based on conditional logic evaluating request parameters (model, message length, user metadata, etc.). Supports rule-based routing where conditions trigger provider selection, enabling sophisticated routing strategies beyond simple fallback or load balancing. Conditions can reference request fields, user context, and provider metadata. Enables A/B testing by routing subset of requests to experimental providers, cost optimization by routing expensive requests to cheaper providers, and capability-based routing by selecting providers supporting required features.

Implements dual-mode caching system supporting both simple (exact-match) and semantic (embedding-based similarity) caching with configurable TTL. Simple caching stores responses keyed by request hash, returning cached results for identical requests within TTL window. Semantic caching uses embeddings to match semantically similar requests and return cached responses, reducing redundant API calls for paraphrased queries. Caching decisions are configurable per request via headers or configuration, with cache invalidation and TTL management built-in.

Extensible plugin architecture with 22+ built-in guardrails and mutators that intercept requests and responses at defined lifecycle points. Hooks execute before request transmission (pre-request), after response receipt (post-response), and on errors, enabling validation, transformation, and security enforcement. Guardrails (validation hooks) reject requests/responses based on policies (PII detection, prompt injection, content filtering, etc.). Mutators transform requests/responses (e.g., prompt rewriting, response formatting). Custom hooks can be registered via plugin system with access to request context, provider info, and configuration.

Implements resilience patterns including automatic retries (up to 5 attempts) with exponential backoff for transient failures, and circuit breaker pattern to prevent cascading failures when providers are unhealthy. Retry logic distinguishes between retryable errors (rate limits, timeouts, 5xx) and permanent errors (4xx auth failures). Circuit breaker tracks provider health and temporarily stops sending requests to unhealthy providers, with configurable thresholds and recovery strategies. Integrates with timeout configuration to enforce maximum request duration.

Validates incoming requests against configuration schema (Options and Targets) before transmission to providers, enforcing required fields, parameter types, and value constraints. Implements Server-Side Request Forgery (SSRF) protection by validating provider URLs against allowlist and preventing requests to internal IP ranges (127.0.0.1, 10.0.0.0/8, etc.). Configuration inheritance and merging allows request-level overrides of global settings while maintaining security constraints. Schema validation uses strict type checking and format validation for model names, API keys, and endpoints.

Handles streaming responses from providers via Server-Sent Events (SSE) protocol, buffering and transforming provider-native streaming formats into OpenAI-compatible delta objects. Supports streaming for chat completions, text generation, and embeddings where applicable. Streaming responses are transmitted to client in real-time with proper SSE formatting, allowing applications to display responses incrementally. Integrates with hooks system to allow custom streaming transformations and monitoring.

Supports multi-source configuration with hierarchy: environment variables (lowest priority), configuration files/objects, and HTTP request headers (highest priority). Configuration schema defines Options (global settings like timeout, retries) and Targets (provider-specific settings like model, apiKey, endpoint). Configuration inheritance allows request-level settings to override defaults while maintaining constraints. Environment variables are loaded via src/utils/env.ts with support for .env files and runtime overrides. Headers can override any configuration parameter for per-request customization.

Provides comprehensive observability via request/response logging, usage analytics, and real-time log streaming. Logs capture request parameters, provider selection, response metadata (tokens, latency), and errors. Usage analytics track API costs, token consumption, and provider performance. Real-time SSE log streaming (Node.js only) allows clients to subscribe to gateway logs and monitor requests as they execute. Integrates with hooks system to allow custom logging and monitoring logic. Supports structured logging for easy parsing and analysis.

Enforces maximum request duration via configurable timeout settings, preventing requests from hanging indefinitely on slow or unresponsive providers. Timeout applies to entire request lifecycle including retries, so total duration is bounded. Supports per-provider timeout overrides and global defaults. Timeout errors are distinguished from other failures and trigger appropriate retry logic (timeouts are retryable). Integrates with circuit breaker to mark providers as unhealthy if they consistently timeout.