Efficient Batch Inference With Dynamic Expert Routing

via “batch embedding inference with dynamic expert routing”

sentence-similarity model by undefined. 22,72,861 downloads.

Unique: Implements sparse expert routing at the batch level, allowing different samples in a batch to activate different expert subsets simultaneously. This differs from dense models where all samples follow identical computation paths; the MoE design enables per-sample routing efficiency while maintaining batch-level parallelism, reducing total compute without sacrificing throughput.

vs others: Achieves 2-4x faster batch inference than dense multilingual transformers on typical hardware due to sparse expert activation, while maintaining competitive embedding quality and supporting larger batch sizes due to reduced per-sample memory footprint.