vllm-mlx

Exposes a FastAPI server implementing OpenAI's /v1/completions and /v1/chat/completions endpoints, backed by a vLLM-style continuous batching scheduler that dynamically groups requests into batches and executes them on Apple Silicon MLX kernels. The scheduler maintains a request queue, allocates KV cache pages on-demand, and interleaves token generation across multiple requests to maximize GPU utilization without blocking on individual request completion.

Implements Anthropic's /v1/messages endpoint with native support for tool_use blocks, allowing models to request external tool execution via structured JSON schemas. The server parses tool definitions, validates model-generated tool calls against the schema, and integrates with the Model Context Protocol (MCP) to execute tools and return results back to the model in a multi-turn conversation loop.

Provides CLI and programmatic configuration for server startup, model selection, and quantization strategy. Automatically detects available GPU memory, selects appropriate quantization (4-bit, 8-bit, or full precision) based on model size and available memory, and loads models into MLX with optimized memory layout. Supports model discovery from HuggingFace Hub with automatic format conversion.

Implements Server-Sent Events (SSE) streaming for all generation endpoints, allowing clients to receive tokens as they are generated without waiting for completion. The server maintains per-request token buffers, flushes tokens at configurable intervals, and handles client disconnections gracefully. Supports both text and multimodal streaming with consistent message formatting.

Implements automatic error recovery for transient failures (OOM, timeout, model errors) with exponential backoff retry logic. Failed requests are queued for retry with configurable retry counts and backoff strategies. The scheduler tracks request state and can resume interrupted generations from checkpoints, reducing wasted computation.

Collects detailed performance metrics including tokens-per-second throughput, latency percentiles (p50/p95/p99), GPU memory utilization, and cache hit rates. Exposes metrics via Prometheus-compatible endpoint and provides CLI benchmarking tools for model comparison. Tracks per-request metrics and aggregates them for system-wide analysis.

Processes images and video frames through vision-language models (LLaVA, Qwen-VL) by encoding visual inputs into MLX tensors, caching vision embeddings to avoid redundant computation, and fusing visual tokens with text tokens in the model's input sequence. Supports batch processing of multiple images per request and video frame extraction with configurable sampling strategies to balance quality and latency.

Accepts audio streams or files, processes them through MLX-based speech recognition models (Whisper or similar), and returns transcriptions with optional timestamp alignment. Supports streaming input via chunked audio frames, allowing real-time transcription as audio arrives without waiting for the full file.

Converts text to natural-sounding speech using MLX-based TTS models, with optional voice cloning by conditioning on reference audio embeddings. Generates audio waveforms in streaming chunks, allowing playback to begin before synthesis completes. Supports multiple voices and speaking styles through model-specific parameters.

Implements a memory-efficient key-value cache using logical pages (fixed-size blocks) instead of contiguous tensors, allowing cache reuse across requests with shared prefixes (e.g., system prompts, conversation history). The scheduler tracks cache page allocation, deallocates pages when requests complete, and enables multiple requests to reference the same cached pages without duplication.

Integrates with the Model Context Protocol to discover, validate, and execute external tools defined in MCP servers. The server maintains a registry of available tools, translates model-generated tool calls into MCP requests, handles tool execution results, and feeds results back to the model for continued reasoning. Supports both synchronous and asynchronous tool execution with timeout handling.

Constrains model output to match a provided JSON schema by using guided generation (token masking) during decoding. The server validates the schema at request time, applies constraints to the model's token selection at each step, and returns only valid JSON matching the schema. Supports nested objects, arrays, and type constraints (string, number, boolean, enum).

Extracts and parses thinking/reasoning tokens from models like Qwen3 and DeepSeek-R1 that emit intermediate reasoning before final answers. The server identifies thinking block delimiters, separates reasoning from output, and optionally streams thinking tokens separately from final response tokens. Supports multiple reasoning formats and models with configurable parsing strategies.

Exposes /v1/embeddings endpoint compatible with OpenAI's embedding API, processing text inputs through MLX-based embedding models to generate dense vector representations. Supports batch processing of multiple texts in a single request, caching embeddings for identical inputs, and returning embeddings in OpenAI's format (array of floats with metadata).