In Flight Batching With Dynamic Request Scheduling

via “continuous batching with dynamic request scheduling”

High-throughput LLM serving engine — PagedAttention, continuous batching, OpenAI-compatible API.

Unique: Decouples batch formation from request boundaries by scheduling at token-generation granularity, allowing requests to join/exit mid-batch and enabling prefix caching across requests with shared prompt prefixes

vs others: Reduces TTFT by 50-70% vs static batching (HuggingFace) by allowing new requests to start generation immediately rather than waiting for batch completion

via “in-flight batching with dynamic request scheduling”

NVIDIA's LLM inference optimizer — quantization, kernel fusion, maximum GPU performance.

Unique: Implements token-level in-flight batching where requests can join ongoing batches at any token position, not just at batch boundaries. Uses a PyExecutor event loop that interleaves prefill and decode phases, allowing new requests to start prefill while other requests are in decode, maximizing GPU utilization.

vs others: More aggressive batching than vLLM's iteration-level batching; TensorRT-LLM's token-level scheduling reduces TTFT by 50-70% and increases throughput by 2-3x on latency-sensitive workloads by allowing requests to join mid-batch.

via “request scheduling with prefill-decode disaggregation”

Fast LLM/VLM serving — RadixAttention, prefix caching, structured output, automatic parallelism.

Unique: Separates prefill and decode scheduling with different batch sizes and priorities, enabling continuous batching where new requests are added to the decode queue without blocking prefill operations.

vs others: Achieves lower time-to-first-token than vLLM through prefill-decode disaggregation and continuous batching, with higher decode throughput by using larger decode batch sizes.

via “adaptive dynamic batching with configurable queue and timeout policies”

ML model serving framework — package models as Bentos, adaptive batching, GPU, distributed serving.

Unique: Implements task queue-based batching at the serving layer with per-endpoint configuration, allowing fine-grained control over batch size, timeout, and queue strategy without modifying model code — integrated directly into the request processing pipeline.

vs others: More efficient than application-level batching (e.g., in FastAPI middleware) because it operates at the worker process level with direct access to model execution, reducing context switching and enabling better GPU memory management.

via “dynamic request batching with configurable batch policies”

NVIDIA inference server — multi-framework, dynamic batching, model ensembles, GPU-optimized.

Unique: Implements a request-level batching scheduler that operates transparently to clients, accumulating requests in queues and executing them as batches without requiring clients to implement batching logic. Uses configurable timeout and size thresholds to balance latency vs throughput, with per-model tuning.

vs others: Automatic batching without client-side changes differs from frameworks like TensorFlow Serving which require clients to batch requests explicitly, reducing integration complexity for high-concurrency scenarios.

via “request batching and async inference for high-throughput workloads”

AI application platform — run models as APIs with auto GPU management and observability.

Unique: Implements dynamic batching that groups requests arriving within a time window (e.g., 100ms) into a single batch, maximizing throughput without requiring explicit batch submission. Uses priority queues to prevent starvation of high-priority requests.

vs others: More efficient than sequential inference (higher GPU utilization) and simpler than self-managed batch processing systems (no queue infrastructure needed)

via “dynamic batching with automatic request scheduling and padding”

Optimized quantized LLM inference for consumer GPUs — EXL2/GPTQ, flash attention, memory-efficient.

Unique: Uses a token-budget scheduler that accumulates requests until the total token count (sum of all sequence lengths) would exceed a threshold, then executes the batch. This is more efficient than fixed-size batching because it adapts to variable sequence lengths and maximizes GPU utilization without wasting compute on padding.

vs others: More efficient than naive fixed-size batching because it adapts to variable sequence lengths and doesn't waste GPU compute on padding, whereas fixed-size batching (e.g., batch_size=8) may underutilize the GPU if sequences are short or waste memory if sequences are long.

via “batch inference with dynamic batching for throughput optimization”

text-generation model by undefined. 92,07,977 downloads.

Unique: Enables dynamic batching through inference engine scheduling (vLLM's continuous batching) rather than static batch sizes, allowing requests to be added and removed from batches in-flight without waiting for batch completion — an architectural pattern that decouples request arrival from batch boundaries

vs others: More efficient than static batching (which requires waiting for full batches); more practical than per-request inference for production workloads with variable request patterns

via “batch inference with dynamic batching and request scheduling”

Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

Unique: Implements token-level continuous batching with dynamic padding and priority scheduling, allowing requests of varying lengths to be processed together without blocking

vs others: Achieves higher throughput than static batching (vLLM's approach) on heterogeneous request streams by adapting batch composition dynamically

via “batched token generation with continuous batching scheduler”

A high-throughput and memory-efficient inference and serving engine for LLMs

Unique: Uses a request-level continuous batching scheduler (not iteration-level) that tracks individual request state through InputBatch and RequestLifecycle objects, enabling dynamic batch composition without padding or request reordering overhead. Integrates with KV cache management to allocate/deallocate cache slots per-request rather than per-batch.

vs others: Achieves 2-4x higher throughput than static batching (e.g., TensorRT-LLM) by eliminating batch padding and idle GPU cycles when requests complete at different times.

via “batch inference with dynamic batching and scheduling”

Portable WASM embedding generation with SIMD and parallel workers - run text embeddings in browsers, Cloudflare Workers, Deno, and Node.js

Unique: Implements adaptive batch sizing based on request arrival rate and latency targets, automatically adjusting batch size and timeout to meet SLA constraints. Includes request prioritization with separate queues for latency-sensitive vs. throughput-focused requests.

vs others: More efficient than processing requests individually (1-5x throughput improvement via batching), and simpler than distributed inference services since batching runs in-process without network overhead.

via “continuous batching with dynamic request scheduling”

A high-throughput and memory-efficient inference and serving engine for LLMs

Unique: Decouples request lifecycle from GPU iteration cycles via iteration-level scheduling with per-request state tracking and configurable policies; most alternatives use static batching or simple FIFO queues that block on slowest request

vs others: Reduces time-to-first-token by 5-10x vs. static batching and achieves 2-3x higher throughput by eliminating idle GPU cycles waiting for request completion

via “adaptive batch processing with dynamic request grouping”

Gemini 2.5 Flash-Lite is a lightweight reasoning model in the Gemini 2.5 family, optimized for ultra-low latency and cost efficiency. It offers improved throughput, faster token generation, and better performance...

Unique: Dynamically adjusts batch sizes based on real-time system load and latency targets rather than using fixed batch sizes, enabling cost optimization that adapts to variable traffic patterns without manual reconfiguration

vs others: More cost-effective than static batching for variable-load systems because dynamic grouping optimizes batch sizes continuously, achieving 40-50% cost reduction compared to per-request processing while respecting latency SLAs

via “batch inference with dynamic batching and request scheduling”

Inference of Meta's LLaMA model (and others) in pure C/C++. #opensource

Unique: Implements dynamic batching with automatic request grouping based on context length and arrival time, rather than fixed batch sizes, reducing latency variance and improving utilization for heterogeneous request patterns

vs others: More efficient than static batching (adapts to request patterns) and simpler to deploy than vLLM's continuous batching (no complex state management)

via “bandwidth-aware layer scheduling and batching”

BitTorrent style platform for running AI models in a distributed way.

Unique: Implements layer-level scheduling with bandwidth awareness, allowing dynamic batching decisions based on available network capacity. Combines request prioritization with bandwidth prediction for optimal throughput.

vs others: More efficient than static batching by adapting batch size to available bandwidth; enables priority scheduling unlike FIFO queues.

via “batch inference with dynamic batching and padding optimization”

wan2-2-fp8da-aoti-faster — AI demo on HuggingFace

Unique: Implements dynamic batching within the Gradio/AOTI pipeline, automatically padding variable-length sequences and adjusting batch size based on GPU memory availability, without requiring external inference servers

vs others: Simpler than vLLM's continuous batching because it batches synchronously per Gradio request cycle, trading some latency variance for easier implementation and debugging