kong

Kong routes LLM requests to multiple AI providers (OpenAI, Anthropic, Azure, Ollama, etc.) through a single standardized API endpoint, translating request/response formats between providers' native schemas. The gateway maintains a provider registry with format adapters that normalize chat completion, embedding, and streaming requests into provider-specific protocols, enabling seamless provider switching and fallback without client-side changes.

Kong intercepts LLM API requests and responses to apply transformations including prompt injection detection, token counting, cost calculation, response filtering, and header injection. The transformation pipeline uses Lua plugins that execute before requests reach the LLM provider and after responses return, enabling cost tracking, security scanning, and response normalization without modifying client or backend code.

Kong supports a hybrid architecture where a control plane (Admin API, configuration management) is separated from data planes (request processing) that connect to the control plane via RPC. The control plane manages configuration and pushes updates to data planes, which apply changes without restarting. Data planes can be deployed in different environments (on-prem, cloud, edge) and sync configuration from the control plane, enabling centralized management with distributed request processing.

Kong can automatically generate MCP servers from existing REST APIs by introspecting API schemas (OpenAPI/Swagger) and converting REST endpoints into MCP tools. The generated MCP server exposes REST endpoints as callable tools with parameter schemas derived from API specifications, enabling LLM agents to interact with REST APIs via MCP without manual MCP server implementation.

Kong is built on OpenResty (Nginx + Lua JIT), providing a high-performance reverse proxy foundation with Lua scripting for custom logic. The Nginx core handles connection management, TLS termination, and HTTP protocol processing, while Lua runs in the request processing pipeline for plugins, routing, and transformations. This architecture enables Kong to handle high request volumes (>10K req/sec per node) while remaining extensible via Lua without requiring C module compilation.

Kong Manager is a web-based UI that provides visual configuration of routes, services, plugins, and consumers without requiring Admin API calls or YAML editing. The UI displays real-time metrics (request count, latency, error rates), plugin status, and upstream health, enabling operators to manage Kong via a dashboard. The UI integrates with Kong's Admin API and supports role-based access control for multi-user environments.

Kong provides native MCP server support, routing MCP client requests to backend MCP servers with authentication, authorization, and observability. The gateway implements MCP protocol handling via Lua plugins that parse MCP JSON-RPC messages, enforce access control policies, and forward requests to configured MCP server upstreams, enabling centralized governance of agentic LLM-to-tool interactions.

Kong's router uses a tree-based matching algorithm that supports exact path matching, regex patterns, and semantic matching (e.g., matching by HTTP method, hostname, headers) to route requests to backend services. The router compiles routes into an optimized tree structure at startup, enabling O(1) lookup for exact matches and efficient regex evaluation for pattern-based routes, with support for route priorities and weighted load balancing across multiple upstreams.

Kong continuously monitors backend service health using active (periodic HTTP requests) and passive (request failure detection) health checks, automatically removing unhealthy upstreams from the load balancing pool and restoring them when health recovers. The health checker runs in a separate Lua coroutine, tracks health state per upstream, and integrates with the load balancer to skip unhealthy targets, enabling transparent failover without client-side retry logic.

Kong implements a plugin system where Lua-based plugins hook into the request/response lifecycle at multiple phases (init, access, header_filter, body_filter, log) and execute in a defined order. Plugins can read/modify requests and responses, access Kong context (route, service, consumer), and interact with external systems via HTTP or database calls. The Plugin Development Kit (PDK) provides a standardized API for common operations (authentication, rate limiting, logging), and plugins are loaded from the filesystem or database at startup.

Kong supports declarative configuration via YAML/JSON files that define routes, services, plugins, and consumers, with a schema system that validates configuration against defined types and constraints. The configuration can be loaded in DB-less mode (in-memory) or synced to a database (PostgreSQL, Cassandra) with automatic migrations that handle schema changes across Kong versions. The schema system uses Lua-based validators that check types, required fields, and custom constraints before configuration is applied.

Kong implements a consumer model where API clients (users, applications, services) are registered as consumers with associated credentials (API keys, OAuth2 tokens, JWT, mutual TLS certificates). Authentication plugins verify credentials against the consumer database, and authorization plugins check consumer attributes (groups, roles, custom metadata) to enforce access control. The consumer model integrates with Kong's plugin system, enabling plugins to apply different policies to different consumers.

Kong provides rate limiting plugins that enforce request quotas per consumer, API, or global level using sliding window or fixed window algorithms. The rate limiter tracks request counts in Redis or Kong's local memory, with distributed state coordination via Redis to ensure accurate limits across multiple Kong nodes. The rate limiting policy is configurable per route/service/consumer, and can enforce different limits for different consumers or APIs.

Kong provides logging plugins that capture request/response metadata (method, path, status, latency, consumer, upstream) and send logs to external systems (syslog, HTTP endpoints, files, Datadog, Splunk, etc.). The logging pipeline runs in the log phase after the response is sent, collecting metrics like request latency, upstream response time, and request/response sizes. Metrics can be exported to monitoring systems (Prometheus, StatsD) for real-time dashboards and alerting.