Multi Model Concurrent Serving With Memory Management

via “multi-model serving with dynamic model loading and unloading”

Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

Unique: Implements LRU-based memory eviction with pre-allocated memory pools and background unloading, avoiding fragmentation and GC pauses that plague naive model swapping approaches

vs others: Faster model switching than vLLM's multi-model support due to optimized memory pooling, though less sophisticated than Ansor-style learned scheduling

via “multi-model-concurrent-serving-with-memory-management”

Get up and running with large language models locally.

Unique: Implements transparent LRU model eviction with automatic VRAM-to-disk swapping, allowing users to work with 3-5 models simultaneously on 8GB VRAM by keeping only the active model loaded while others reside on disk

vs others: Simpler than vLLM's multi-model serving because Ollama handles memory swapping automatically without requiring explicit model scheduling, vs. manual model loading which requires application-level coordination

via “concurrent request handling for multi-model interactions”

MCP server: mm-sec-prototype

Unique: The server's non-blocking architecture allows for high throughput and low latency, making it suitable for demanding applications.

vs others: More efficient than traditional request handling systems that may block on I/O operations.

via “concurrent request handling for multiple models”

MCP server: mcpservers

Unique: Utilizes asynchronous programming to enable true concurrency, allowing for efficient processing of multiple requests, unlike synchronous models that can bottleneck under load.

vs others: Significantly faster than synchronous request handling systems, making it ideal for applications with high concurrency needs.

via “multi-threaded request handling for concurrent model calls”

MCP server: test_mcp_server

Unique: Utilizes a multi-threaded architecture to allow concurrent processing of requests, enhancing performance under load.

vs others: More efficient than single-threaded models, significantly improving response times in high-load scenarios.

via “concurrent request handling for model interactions”

MCP server: papers

Unique: Employs an event-driven architecture that allows for non-blocking I/O operations, which is more efficient than traditional multi-threaded approaches.

vs others: Handles more concurrent requests with lower latency compared to traditional multi-threaded servers.