Multimodal Knowledge Distillation And Compression

via “multimodal model compression with vision-language alignment”

Toolkit for LLM quantization, pruning, and distillation.

Unique: Implements multimodal compression by applying modality-specific compression strategies to vision encoders, text encoders, and fusion layers while validating cross-modal alignment, enabling efficient compression of vision-language models without degrading multimodal understanding

vs others: More suitable for multimodal models than generic compression because it preserves cross-modal alignment; more flexible than single-modality compression because it handles heterogeneous architectures; better integrated with multimodal inference engines than generic tools

via “knowledge distillation for model compression”

text-generation model by undefined. 1,60,37,172 downloads.

Unique: Enables knowledge transfer from larger teacher (GPT-2) to smaller student via soft target matching, preserving linguistic knowledge while reducing parameters — complementary to quantization for extreme compression

vs others: More effective than quantization alone for large compression ratios (5-10x), but requires training vs quantization's post-hoc approach — best combined with quantization for maximum compression

via “knowledge-distillation-from-roberta-base”

fill-mask model by undefined. 10,73,316 downloads.

Unique: Distilled from RoBERTa-base using standard knowledge distillation (MSE loss on hidden states + MLM loss) achieving 95-98% of teacher performance with 66% parameter reduction, representing a favorable compression-accuracy tradeoff compared to training smaller models from scratch

vs others: Maintains RoBERTa's superior pretraining procedure (dynamic masking, longer training) while achieving efficiency comparable to ALBERT or MobileBERT, and outperforms BERT-base distillations due to better teacher model quality

via “knowledge distillation-based model compression for transfer learning”

question-answering model by undefined. 32,657 downloads.

Unique: MobileBERT uses inverted bottleneck architecture (wide intermediate layers, narrow hidden states) combined with intermediate layer distillation, achieving superior compression compared to simple pruning or quantization. This architectural design is inherently distillation-friendly, enabling efficient knowledge transfer.

vs others: More effective knowledge transfer than DistilBERT (which uses only final layer distillation) due to intermediate layer matching; enables fine-tuning on custom datasets with better accuracy retention than training smaller models from scratch.

via “knowledge distillation for model compression”

Retrieval and Retrieval-augmented LLMs

Unique: FlagEmbedding provides retrieval-specific knowledge distillation framework that preserves embedding quality and ranking performance through teacher-student training with contrastive and ranking-aware losses.

vs others: Offers retrieval-optimized distillation compared to generic model compression, maintaining ranking quality while reducing model size.

via “cross-modal knowledge transfer (language-to-vision and vision-to-language)”

* ⭐ 03/2023: [PaLM-E: An Embodied Multimodal Language Model (PaLM-E)](https://arxiv.org/abs/2303.03378)

Unique: Achieves bidirectional knowledge transfer through a unified transformer architecture trained on mixed text-only and multimodal data, rather than using separate pre-trained vision and language models that are later aligned

vs others: More efficient than training separate vision and language models and then aligning them, because knowledge transfer happens during pretraining; likely produces more coherent multimodal representations

via “efficient inference with knowledge distillation from teacher models”

* ⭐ 02/2023: [Adding Conditional Control to Text-to-Image Diffusion Models (ControlNet)](https://arxiv.org/abs/2302.05543)

Unique: Combines multiple distillation strategies (response, feature, and relation-based) in a unified framework, enabling flexible compression where different layers can use different distillation targets. Uses attention pattern matching to preserve model interpretability while compressing.

vs others: Achieves 92-95% of teacher accuracy at 20% model size, compared to 85-90% for standard response-based distillation alone. Enables deployment of 1-2B parameter models with near-teacher performance, whereas pruning or quantization alone typically requires 30-40% accuracy sacrifice at equivalent compression ratios.

via “model distillation and compression for deployment”

Open Pretrained Transformers (OPT) by Facebook is a suite of decoder-only pre-trained transformers. [Announcement](https://ai.meta.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/).

via “multimodal-knowledge-distillation-and-compression”


Unique: Addresses the specific challenge of preserving cross-modal alignment and reasoning during compression, with concrete strategies for multimodal knowledge distillation (e.g., distilling attention patterns across modalities) — a critical concern absent from single-modality compression literature

vs others: Deeper treatment of multimodal-specific compression challenges (preserving cross-modal reasoning, handling modality imbalance during distillation) compared to generic model compression courses

via “multimodal-efficiency-and-inference-optimization”


Unique: Addresses efficiency as a multimodal-specific problem where modalities have different computational costs and compression sensitivity, requiring modality-aware optimization strategies

vs others: More practical than general model compression literature because it accounts for fusion-specific challenges and modality imbalances that generic compression misses