Multi Tier Kv Cache Storage With Hicache And Storage Backends

via “multi-tier kv cache storage with hicache and storage backends”

Fast LLM/VLM serving — RadixAttention, prefix caching, structured output, automatic parallelism.

Unique: Implements a three-tier storage hierarchy (GPU VRAM → CPU RAM → NVMe) with predictive migration logic that monitors access patterns and proactively moves data between tiers. Includes configurable storage backends and transfer optimization for each tier boundary.

vs others: Enables serving sequences 2-4x longer than vLLM on the same hardware by intelligently spilling to CPU/NVMe, with prefetching logic that hides transfer latency for predictable access patterns.

via “query-aware-intelligent-caching”

Simple open-source embedding database — add docs, query by text, built-in embeddings, easy RAG.

Unique: Tiering is fully automatic and query-aware, learning access patterns over time and promoting/demoting data without user intervention. Eliminates manual cache management and tuning, reducing operational overhead compared to systems requiring explicit cache configuration.

vs others: More automatic than Redis-based caching (which requires manual key management) and more cost-effective than keeping all data in memory, but adds latency variability compared to all-in-memory systems and requires cloud storage integration.

via “caching layer with redis and kvrocks for session and job state management”

Open-source computer vision annotation tool.

Unique: Uses both Redis (for hot data) and Kvrocks (for persistent caching) in a tiered approach, balancing speed and durability. Cache invalidation is event-driven rather than time-based, reducing stale data issues.

vs others: More sophisticated than simple Redis caching (which lacks persistence) and more flexible than database-level caching (which is harder to control). Tiered approach (Redis + Kvrocks) provides both speed and durability.

via “cloudflare workers kv-based caching and storage layer”

Put an end to code hallucinations! GitMCP is a free, open-source, remote MCP server for any GitHub project

Unique: Leverages Cloudflare Workers KV as a native, zero-configuration cache layer integrated into the same serverless runtime, eliminating separate cache service dependencies and enabling global edge caching without additional infrastructure

vs others: Faster than external caches (Redis, Memcached) because data is stored at Cloudflare edge locations globally, providing sub-millisecond retrieval latency vs network round-trip times to centralized cache servers

via “multi-level kv cache management with prefix caching”

A high-throughput and memory-efficient inference and serving engine for LLMs

Unique: Implements block-level KV cache with prefix caching that tracks cache blocks as first-class objects with ownership and eviction policies, enabling cache reuse across requests without recomputation. Supports disaggregated serving via KV cache transfer protocol, allowing cache to be stored on dedicated cache servers separate from compute workers.

vs others: Reduces memory usage by 20-40% on multi-turn conversations vs. standard KV cache by reusing cached prefixes; disaggregated serving enables 10x larger batch sizes by decoupling cache capacity from compute capacity.