@auto-engineer/ai-gateway

Abstracts API differences across multiple LLM providers (OpenAI, Anthropic, etc.) behind a single standardized interface, translating provider-specific request/response formats into a normalized schema. Implements adapter pattern with provider-specific client wrappers that handle authentication, rate limiting, and protocol differences, allowing developers to swap providers without changing application code.

Translates MCP tool definitions (JSON schemas) into provider-native function calling formats (OpenAI function_calling, Anthropic tool_use, etc.), then routes tool execution results back through the LLM. Implements a schema normalization layer that maps between MCP's tool specification and each provider's function calling protocol, handling argument validation and result serialization.

Discovers and registers MCP servers and their tools, exposing them to LLM providers through the gateway. Implements MCP client protocol handling that connects to MCP servers, introspects available tools, and manages tool lifecycle (initialization, execution, cleanup), with automatic tool schema translation for function calling.

Handles streaming token responses from different providers (OpenAI streaming, Anthropic streaming, etc.) and normalizes them into a unified event stream. Implements a stream adapter that buffers partial tokens, detects stream completion, and emits normalized events (token, done, error) regardless of provider, enabling consistent streaming UX across backends.

Centralizes API key management and provider configuration (model selection, temperature, max tokens, etc.) with support for environment variables, config files, and runtime overrides. Implements a configuration hierarchy where runtime settings override file-based config, which overrides environment variables, with validation of required credentials before API calls.

Provides hooks for logging and monitoring all LLM API calls, including request payloads, response metadata, latency, and token usage. Implements a middleware pattern where developers can attach custom logging handlers (e.g., to send metrics to Datadog, write to files, or track costs) without modifying core gateway code.

Implements intelligent retry logic that handles provider-specific errors (rate limits, timeouts, API errors) with exponential backoff and optional fallback to alternative providers. Detects error types (transient vs permanent) and applies provider-specific retry strategies (e.g., longer backoff for Anthropic rate limits vs OpenAI).

Automatically detects which features each provider/model supports (vision, function calling, streaming, etc.) and negotiates feature availability at runtime. Implements a capability registry that maps model names to supported features and prevents unsupported feature requests (e.g., vision on text-only models) before sending to the API.

Groups multiple LLM requests into batches for providers that support batch APIs (e.g., OpenAI Batch API), reducing per-request costs. Implements a batching queue that accumulates requests up to a size/time threshold, then submits them as a single batch job, with result deduplication and callback routing.

Tracks token usage across requests and manages context windows to prevent exceeding model limits. Implements provider-specific token counters (using tokenizer libraries or provider APIs) and automatically truncates or summarizes context when approaching limits, with configurable truncation strategies (sliding window, summarization, etc.).

Parses LLM responses into structured formats (JSON, typed objects) with provider-agnostic handling of structured output modes (OpenAI JSON mode, Anthropic structured output, etc.). Implements schema validation and automatic fallback to regex/parsing if structured output fails, with error recovery for malformed responses.