LiteLLM

Provides a single litellm.completion() API that normalizes requests across 100+ LLM providers (OpenAI, Anthropic, Google, Azure, Ollama, etc.) by translating OpenAI message format into provider-specific request schemas. Uses provider detection logic in get_llm_provider_logic.py to route requests and a parameter mapping system (get_supported_openai_params.py) to handle capability differences across providers, enabling write-once code that works with any LLM backend.

The Router class (litellm/router.py) distributes requests across multiple model deployments using configurable routing strategies (round-robin, least-busy, cost-optimized, latency-optimized) with real-time health tracking and automatic failover. Maintains per-deployment metrics (latency, error rates, availability) and selects the next deployment based on strategy weights, enabling cost optimization and high availability without manual intervention.

Enables fine-grained model access control using model access groups (e.g., 'gpt-4-*' matches all GPT-4 variants) and wildcard patterns. Allows teams/users to be assigned to groups that grant access to specific model families without listing individual models. Supports dynamic model discovery where new models matching a wildcard pattern are automatically accessible.

Implements automatic fallback to alternative providers/models if the primary fails, with exponential backoff retry logic and cooldown periods to prevent thrashing. Tracks failure patterns per deployment and temporarily deprioritizes failed providers. Supports custom fallback chains (e.g., GPT-4 → Claude → Gemini) defined in router configuration.

Provides a standalone HTTP server (litellm/proxy/proxy_server.py) that acts as a centralized gateway for all LLM requests, implementing authentication, rate limiting, cost tracking, and observability. Exposes OpenAI-compatible REST API endpoints (/v1/chat/completions, /v1/embeddings, etc.) and management endpoints for key/team/user management. Supports deployment as Docker container or standalone Python service.

Provides a web-based dashboard (litellm/proxy/admin_ui/) for managing API keys, teams, users, and viewing spend analytics. Enables non-technical users to create/rotate keys, set rate limits, view cost breakdowns by model/team/user, and monitor API health. Supports role-based access (admin, team lead, viewer) with granular permissions.

Maintains a comprehensive database of model pricing and context windows (model_prices_and_context_window.json) covering 100+ models across all major providers. Automatically updates pricing for new models and provider price changes. Enables cost calculation, context window validation, and model selection based on budget/capability constraints.

Provides pass-through endpoints that forward requests directly to provider APIs without modification, enabling access to provider-specific features not yet supported by LiteLLM's unified interface. Useful for new provider features, experimental APIs, or edge cases. Maintains authentication and applies Proxy policies (rate limiting, cost tracking) even for pass-through requests.

Integrates with Model Context Protocol (MCP) servers to expose tools/resources as LLM-callable functions. Acts as a gateway between LLMs and MCP servers, translating tool definitions and handling tool invocations. Enables LLMs to access external tools (web search, code execution, database queries) via a standardized protocol.

Implements MCP (Model Context Protocol) server gateway that enables LLMs to interact with external tools and services via standardized protocol. Supports MCP clients connecting to LiteLLM proxy, which routes tool calls to registered MCP servers. Implements A2A (Agent-to-Agent) protocol for agent-to-agent communication. Provides tool registry and automatic tool discovery from MCP servers. Integrates with function calling to enable seamless tool use across providers.

Automatically calculates per-request costs using provider-specific pricing models stored in model_prices_and_context_window.json and litellm/llms/openai/cost_calculation.py. Tracks cumulative spend per user, team, organization, and tag via the Proxy's database layer (db_spend_update_writer.py) with Redis buffering for high-throughput scenarios. Supports budget enforcement at multiple levels (user, team, organization) with configurable alerts and hard limits.

Caches LLM responses using both exact-match (hash of messages + parameters) and semantic-match strategies via Redis integration (litellm/proxy/cache.py). Exact-match caching returns identical responses for identical requests; semantic caching uses embeddings to find similar past requests and return cached responses for semantically equivalent queries, reducing API calls and latency. Supports dynamic cache controls (TTL, cache-key customization) per request.

Enforces rate limits at multiple levels (per-user, per-team, per-organization, per-model) using token bucket algorithms stored in Redis or in-memory. Tracks request counts and token usage (input + output tokens) against configurable limits, returning 429 errors when limits are exceeded. Supports both hard limits (reject requests) and soft limits (log warnings) with customizable reset windows and burst allowances.

Normalizes streaming responses from 100+ providers into a unified Server-Sent Events (SSE) format compatible with OpenAI's streaming API. Handles provider-specific streaming formats (Anthropic's event stream, Google's chunked responses, Azure's streaming) and converts them to OpenAI's delta-based format. Supports both SDK streaming (Python generators) and Proxy streaming (HTTP SSE), with automatic error handling and graceful fallback to non-streaming if provider fails.

Normalizes function/tool calling across providers by translating OpenAI's function_call format to provider-specific schemas (Anthropic's tool_use, Google's function_calling, Ollama's tools). Accepts OpenAI-style tool definitions (name, description, parameters as JSON schema) and maps them to each provider's expected format, enabling write-once tool-calling code. Handles tool response routing and automatic re-invocation for multi-turn tool use.

Manages API keys, user permissions, and team hierarchies via the Proxy's database layer (schema.prisma) with role-based access control (RBAC). Supports key rotation, per-key rate limits, model access restrictions (which models a key can call), and audit logging. Integrates with SCIM and SSO for enterprise identity management, enabling centralized user/team provisioning.

Provides a callback system (litellm/integrations/custom_logger.py) that hooks into every LLM request/response for logging, monitoring, and analytics. Supports custom callbacks (user-defined functions) and pre-built integrations (Langfuse, Datadog, New Relic, Weights & Biases). Logs request metadata (model, tokens, latency, cost), responses, and errors with optional message redaction for privacy. Integrates with observability platforms for distributed tracing and analytics.

Leverages provider-native prompt caching (OpenAI's prompt_cache_control, Anthropic's cache_control) to reduce costs and latency for repeated context. Automatically detects provider support and applies caching headers to system prompts or long context blocks. Tracks cache hit rates and cost savings, enabling optimization of cached content.