workers-ai-provider

Executes LLM inference directly on Cloudflare Workers edge runtime without external API calls, leveraging Cloudflare's distributed GPU infrastructure. Routes requests through Cloudflare's proprietary model serving layer that optimizes for sub-100ms latency by executing models at edge locations closest to request origin. Integrates with Vercel AI SDK's standardized provider interface, allowing drop-in replacement of OpenAI/Anthropic providers with zero SDK code changes.

Implements the Vercel AI SDK's standardized LanguageModel interface, allowing Cloudflare Workers AI to be used as a drop-in provider replacement for OpenAI, Anthropic, or other LLM providers. Translates Vercel's unified message format (role/content pairs) into Cloudflare Workers AI API calls, handling response streaming, error mapping, and token counting transparently. Maintains API parity with other SDK providers so applications can switch providers with single configuration change.

Streams LLM responses token-by-token to clients while simultaneously tracking token consumption for billing/monitoring purposes. Implements Vercel AI SDK's streaming protocol which yields text chunks and metadata (finish_reason, usage) as they arrive from Cloudflare Workers AI backend. Handles backpressure and connection management to prevent memory leaks in long-running streams.

Supports routing requests to different Cloudflare Workers AI models (e.g., Llama 2, Mistral, GPT-4-equivalent) based on application logic, with automatic fallback to alternative models if primary model is unavailable. Implements model selection through configuration or runtime parameters, allowing A/B testing different models or graceful degradation when preferred models hit rate limits. Maintains model metadata (context window, cost, latency characteristics) for intelligent routing decisions.

Enables LLM-driven function calling by translating Vercel AI SDK's tool definitions into Cloudflare Workers AI's function calling format, then parsing model-generated tool calls back into structured JSON. Implements bidirectional schema translation between SDK tool format and Cloudflare's function calling API, handling type validation and error cases. Supports iterative tool use where model can call multiple functions and receive results for further reasoning.

Provides native integration with Cloudflare Workers runtime, including automatic credential management through environment variables, request context propagation (user IP, country, headers), and integration with Cloudflare's request/response lifecycle. Handles Workers-specific constraints like CPU time limits and memory bounds by optimizing for edge execution patterns. Supports both module and service worker formats for maximum compatibility.

Implements automatic retry logic for transient failures (rate limits, temporary unavailability) using exponential backoff with jitter to prevent thundering herd. Maps Cloudflare Workers AI error responses to standardized error types (RateLimitError, ModelNotFoundError, etc.) for consistent error handling across applications. Provides detailed error context including retry-after headers and remaining quota for intelligent client-side error recovery.