Response Time And Performance Metrics Collection

via “metrics collection and observability with performance tracking”

A high-throughput and memory-efficient inference and serving engine for LLMs

Unique: Implements multi-level metrics collection (request, batch, system) with automatic aggregation and Prometheus export, enabling real-time performance monitoring without external instrumentation. Tracks cache hit rates, expert utilization (for MoE), and attention backend performance.

vs others: Provides 10x more detailed metrics than alternatives like TensorRT-LLM; automatic Prometheus export enables integration with standard monitoring stacks without custom instrumentation code.