litellm

Abstracts 100+ LLM provider APIs (OpenAI, Anthropic, Azure, Bedrock, VertexAI, Cohere, HuggingFace, VLLM, NVIDIA NIM, Ollama) behind a single OpenAI-compatible interface. Uses provider detection logic that maps model names to their native providers and automatically translates request/response formats, handling provider-specific parameter mappings, authentication schemes, and response structures without requiring developers to write provider-specific code.

Routes requests across multiple LLM deployments using configurable strategies (round-robin, least-busy, cost-optimized, latency-based) with real-time health checks and fallback chains. The Router class maintains deployment metadata (model, provider, cost, latency), tracks request distribution, and automatically retries failed requests on alternate deployments while respecting cooldown periods to avoid cascading failures.

Manages model access control through model access groups that use wildcard patterns (e.g., 'gpt-4*', 'claude-*-v1') to grant users/teams access to sets of models. Evaluates patterns at request time to determine if a user can access a requested model, supporting hierarchical access (e.g., admin can access all models, team members can access team-specific models).

Enforces rate limits per API key, user, or team using token bucket or sliding window algorithms. Tracks rate limit state in Redis for distributed enforcement across multiple proxy instances, supporting different limit strategies (requests per minute, tokens per hour, cost per day). Returns HTTP 429 with retry-after headers when limits are exceeded, and integrates with cooldown management to prevent cascading failures.

Continuously monitors provider health by sending periodic test requests to each configured model, tracking response times and error rates. Marks providers as unhealthy when error rates exceed thresholds, automatically removing them from routing until they recover. Integrates with cooldown management to prevent repeated requests to failing providers, and exposes health status via /health endpoints for load balancer integration.

Provides OpenAI Assistants API compatibility by translating Assistants API requests to underlying LLM completion calls, managing conversation state, file uploads, and tool execution. Supports OpenAI-specific features (code interpreter, retrieval) through abstraction layers that map to provider-agnostic implementations, enabling applications built for OpenAI Assistants to work with alternative providers.

Supports provider-specific reasoning features (OpenAI o1 reasoning, Claude extended thinking) by translating reasoning parameters to provider-native formats and handling extended thinking responses. Manages longer processing times and higher costs associated with reasoning models, and provides access to reasoning traces for debugging and analysis.

Acts as an MCP (Model Context Protocol) server gateway, translating MCP tool definitions to LLM-compatible function schemas and vice versa. Enables LLMs to call MCP-compatible tools through a standardized interface, supporting tool discovery, execution, and result handling. Integrates with MCP servers for external tool access (file systems, databases, APIs).

Calculates per-request costs by parsing model pricing from a centralized registry, tracking input/output token counts, and aggregating costs across users, teams, and deployments. Integrates with the proxy database to store spend logs with timestamps, model names, and token counts, enabling cost analytics, budget enforcement, and FinOps reporting via FOCUS cost export format.

Manages API keys, user identities, and team memberships through a database-backed authentication system with role-based access control (RBAC). Supports multiple authentication methods (API keys, OAuth via SCIM/SSO), enforces per-key rate limits and budget caps, and tracks which users/teams can access which models via model access groups and wildcard patterns.

Handles streaming responses from diverse providers (OpenAI, Anthropic, Azure, etc.) by normalizing their different streaming formats (Server-Sent Events, JSON Lines, custom formats) into a unified stream of choice objects. Implements buffering, error handling, and graceful degradation when streaming fails, allowing clients to consume a consistent stream interface regardless of underlying provider.

Caches LLM responses using both exact-match caching (identical prompts) and semantic caching (similar prompts via embeddings). Stores cached responses in Redis with configurable TTL, supports cache invalidation strategies, and integrates with provider-native prompt caching (e.g., Claude's prompt caching) to reduce costs and latency for repeated or similar queries.

Enforces content safety policies by running requests and responses through configurable guardrails before reaching LLMs or returning to clients. Supports built-in guardrails (PII detection, prompt injection detection, toxicity filtering) and custom guardrails via a plugin architecture. Integrates with third-party safety services (e.g., Presidio for PII, custom ML models) and can block, redact, or flag requests based on policy violations.

Provides comprehensive observability through a callback system that hooks into request/response lifecycle events (pre-request, post-request, on-error). Logs all LLM interactions to configurable backends (Langfuse, Datadog, custom webhooks) with full context (model, tokens, cost, latency, user). Supports message redaction for privacy, custom logging logic via callback plugins, and integration with APM tools for distributed tracing.

Enables function calling by accepting tool/function definitions as JSON schemas, translating them to provider-specific formats (OpenAI function_calling, Anthropic tools, etc.), and parsing tool calls from responses. Validates tool schemas, handles tool execution orchestration, and supports automatic retry loops where the LLM can call tools and receive results until a final response is generated.

Deploys as a standalone HTTP proxy server that intercepts LLM API requests, applies routing, authentication, cost tracking, and guardrails before forwarding to providers. Implements OpenAI-compatible endpoints (/v1/chat/completions, /v1/embeddings, /v1/models) plus pass-through endpoints for provider-specific features. Supports Docker deployment, horizontal scaling with Redis state sharing, and management APIs for key/team/user administration.