Distilled Transformer Inference With Knowledge Transfer

translation model by undefined. 13,09,929 downloads.

Unique: Applies knowledge distillation specifically to the M2M-100 architecture, preserving the multilingual shared embedding space while reducing parameters by 82%. Uses logit matching and intermediate layer alignment to transfer the teacher's translation knowledge, enabling competitive performance on 200 language pairs with a single 600M-parameter model.

vs others: Smaller than full NLLB-200 (600M vs 3.3B) with faster inference than uncompressed models, but slower and lower quality than language-specific models fine-tuned for single pairs; trade-off is worthwhile for multilingual coverage on resource-constrained devices.

via “distilled transformer inference with reduced parameter footprint”

zero-shot-classification model by undefined. 2,58,745 downloads.

Unique: Distilled from RoBERTa-Large specifically for NLI tasks using knowledge distillation, achieving 15x parameter reduction while maintaining >90% of teacher model accuracy on SNLI/MultiNLI benchmarks — most lightweight NLI alternatives either use non-distilled architectures or sacrifice accuracy more severely

vs others: Faster CPU inference than full-size cross-encoders (RoBERTa-Large, BERT-Large) by 3-5x; more accurate than simple bi-encoder baselines on entailment tasks due to cross-encoder architecture, despite smaller size

via “distilled transformer inference with reduced memory footprint”

question-answering model by undefined. 1,61,301 downloads.

Unique: Combines ELECTRA discriminator pre-training with knowledge distillation to achieve 40% parameter reduction while preserving KorQuAD performance; supports three inference backends (PyTorch, TensorFlow, TFLite) via unified transformers API, enabling deployment flexibility from cloud to mobile without retraining

vs others: Smaller than koelectra-base-v2-korquad (92M vs 110M parameters) with comparable accuracy; faster inference than full BERT-based Korean QA models; more flexible deployment than proprietary Korean QA APIs which require cloud connectivity

via “efficient transformer inference and optimization”


Unique: Combines algorithmic optimization techniques (sparse attention, linear attention approximations) with system-level considerations (batching strategies, KV-cache management, hardware acceleration), treating inference optimization as a holistic problem rather than isolated techniques

vs others: More comprehensive than individual optimization papers, but less practical than frameworks like vLLM or TensorRT that provide production-ready optimization implementations