Petals

Enables inference on large language models by distributing computation across a peer-to-peer network using BitTorrent-style protocols. Each peer runs a subset of model layers, and inference requests are routed through the network with automatic layer assignment and load balancing. Uses a DHT (Distributed Hash Table) for peer discovery and maintains connection pools to optimize throughput across heterogeneous hardware.

Dynamically assigns model layers to available peers based on real-time metrics including peer bandwidth, GPU utilization, latency, and VRAM availability. Uses a greedy routing algorithm that selects the optimal peer for each layer during inference, with fallback mechanisms for peer unavailability. Maintains a peer registry with periodic health checks and bandwidth estimation via probe requests.

Provides client libraries (Python, JavaScript) that handle inference orchestration, including prompt tokenization, layer routing, result decoding, and error handling. Manages inference context including conversation history, system prompts, and generation parameters. Implements client-side caching of tokenized prompts to avoid re-tokenization. Abstracts away network complexity, presenting a simple API similar to standard LLM inference libraries.

Supports any transformer-based model that can be split into layers, regardless of architecture (BERT, GPT, LLaMA, Mistral, etc.). Automatically detects model structure and layer boundaries from HuggingFace model configs. Handles different layer types (attention, feed-forward, embedding) transparently. Includes compatibility layer for models with non-standard architectures or custom layers. Supports both encoder-only and decoder-only models.

Caches model layers locally on peers to avoid re-downloading them for subsequent inferences. Implements LRU (Least Recently Used) eviction policy with configurable cache size based on available VRAM. Prefetches layers before inference begins based on predicted request patterns, reducing latency for common model paths. Uses content-addressable storage (hashing) to verify layer integrity and enable deduplication across peers.

Automatically selects appropriate numerical precision (FP32, FP16, INT8) for each layer based on peer hardware capabilities and model requirements. Handles mixed-precision inference where different layers run at different precisions on different peers. Includes quantization support for reducing VRAM requirements on resource-constrained peers. Detects hardware capabilities (GPU type, compute capability, available VRAM) and adapts layer execution accordingly.

Uses a Distributed Hash Table (DHT) similar to BitTorrent to discover peers offering specific model layers without requiring a central server. Peers register themselves in the DHT with their available layers, VRAM, and bandwidth. Clients query the DHT to find peers capable of serving requested layers. Includes bootstrap node mechanism for initial network entry and fallback peer lists for network resilience.

Streams generated tokens back to the client as they're produced rather than waiting for full sequence completion. Implements early stopping mechanisms allowing clients to terminate generation mid-sequence if desired (e.g., when reaching a stop token or max length). Uses token-by-token routing where each generated token is fed back through the network for the next iteration, with caching of intermediate states to reduce redundant computation.

Tracks peer reputation based on inference quality, availability, and response time. Implements incentive mechanisms (rewards, penalties) to encourage high-quality participation and discourage malicious or low-quality peers. Maintains reputation scores updated based on inference success/failure, latency measurements, and user feedback. Integrates with optional blockchain or token systems for monetizing peer contributions.

Detects inference failures (peer disconnection, timeout, corrupted output) and automatically retries with alternative peers. Implements exponential backoff for retries to avoid overwhelming peers. Maintains fallback peer lists for each layer, allowing seamless failover if primary peer becomes unavailable. Includes timeout detection and circuit breaker pattern to quickly identify failing peers and remove them from rotation.

Verifies model layer integrity using cryptographic hashing (SHA-256) to detect corrupted or tampered weights. Implements content-addressable storage where layers are identified by their hash, enabling deduplication and integrity verification across peers. Includes optional signature verification for layers signed by model authors, preventing unauthorized modifications. Detects bit-flip errors and network corruption during layer transfer.

Schedules layer transfers based on available bandwidth to minimize total inference time. Batches multiple inference requests to amortize network overhead and improve GPU utilization. Implements request queuing with priority scheduling (e.g., shorter sequences prioritized over longer ones). Predicts layer transfer time based on size and available bandwidth, allowing clients to make informed decisions about request batching.