Fireworks AI

Serves 15+ open-source and proprietary LLMs (DeepSeek, Kimi, GLM, Qwen, MiniMax, Gemma) through a unified API with FireOptimizer engine for model-specific inference optimization. Routes requests to globally distributed GPU clusters with zero cold starts on serverless tier, achieving sub-100ms latency for typical completions through kernel-level optimizations and batched inference scheduling.

Implements tool-use capability via structured function calling that converts natural language requests into deterministic function invocations. Accepts JSON schema definitions for tools, validates model outputs against schemas, and returns structured function calls with arguments. Supports multi-step tool chains where model can call multiple functions sequentially with output from prior calls as context.

Provides dedicated GPU infrastructure for models with guaranteed resource allocation, lower latency, and higher rate limits than serverless. Customers specify GPU type and count, pay per GPU-second, and get isolated compute capacity. Supports custom model deployments (fine-tuned models, proprietary models) with minimal cold starts. Enables predictable performance for production workloads.

Caches frequently-used prompt prefixes (system prompts, context, documents) at 50% of standard input token price. Subsequent requests reusing cached prompts pay only for new tokens, reducing cost for multi-turn conversations, RAG systems, or repeated analysis tasks. Cache invalidation automatic on prompt changes; no manual cache management required.

Integrates Fireworks models with Claude Code through Model Context Protocol (MCP) server, enabling Claude to call Fireworks inference as a tool. Developers set up Fireworks MCP server, configure Claude to connect, and Claude can invoke Fireworks models for specific tasks within coding workflows. Enables hybrid workflows combining Claude's reasoning with Fireworks' model variety and cost efficiency.

Claims 'globally distributed virtual cloud infrastructure' with 'no cold starts' for serverless inference, implying models are pre-loaded across multiple geographic regions. Specific regions not documented. Cold-start elimination suggests persistent model loading or aggressive caching, but implementation details unknown. Latency claims ('industry-leading throughput and latency') unquantified. Distributed infrastructure presumably enables geographic load balancing and reduced latency for global users.

Enforces structured output formats through two mechanisms: JSON mode (guarantees valid JSON output matching schema) and grammar-based constraints (uses formal grammars like GBNF to restrict token generation to valid outputs). Grammar approach operates at token-level during generation, preventing invalid outputs before they're generated, rather than post-processing.

Processes images alongside text through vision-capable models (Kimi K2.5/K2.6, Qwen3 VL 30B, GLM-5.1, Gemma 4 variants) that accept image inputs in base64 or URL format. Models analyze document layouts, extract text via OCR, answer questions about image content, and generate descriptions. Multimodal context combines image understanding with text reasoning in single forward pass.

Transcribes audio to text using Whisper V3 Large and Whisper V3 Large Turbo models, billed per second of audio input. Supports optional speaker diarization (identifying who spoke when) with 40% cost surcharge. Batch API available at 40% discount for non-real-time transcription. Handles audio in various formats (WAV, MP3, M4A, etc.) with automatic format detection.

Generates images from text prompts using FLUX.1 family (dev, schnell, Kontext Pro/Max), SDXL, and Playground models. Pricing varies by model: step-based billing for iterative models (SDXL, Playground, FLUX dev/schnell at $0.00013-$0.00035/step) and flat-rate billing for Kontext variants ($0.04-$0.08/image). Kontext models support image context input for style/layout guidance.

Generates dense vector embeddings for text using Qwen3 8B and generic embedding models (<150M, 150M-350M parameter variants). Embeddings enable semantic search, similarity matching, and clustering. Priced per 1M tokens: Qwen3 8B at $0.1/1M, smaller models at $0.008-$0.016/1M. Embeddings can be stored in external vector databases (Pinecone, Weaviate, Milvus) for retrieval-augmented generation (RAG).

Processes multiple inference requests asynchronously in batches at 50% cost reduction on both input and output tokens. Batch API accepts job definitions with multiple prompts, executes them on lower-priority GPU capacity, and returns results when complete. Speech-to-text batch processing offers 40% discount. Ideal for non-real-time workloads (data processing, content generation, analysis).

Enables supervised fine-tuning (SFT) and preference-based fine-tuning (DPO) on open-source models using LoRA (parameter-efficient) or full-parameter training. Pricing scales by model size: LoRA SFT from $0.50/1M training tokens (≤16B) to $10/1M (>300B); full-parameter training 2x LoRA cost. Trained models served at same price as base models. Training tokens calculated as `dataset_tokens × epochs × (conversation_turns / 2)` for reasoning traces.

Implements OpenAI-compatible API interface allowing clients to switch from OpenAI to Fireworks by changing base URL only. Supports same request/response format, authentication headers, and parameter names as OpenAI API. Enables use of OpenAI SDKs (Python, Node.js, etc.) without code changes. Supports chat completions, embeddings, and image generation endpoints with OpenAI-compatible parameters.