Reasoning Model Distillation To Smaller Parameter Scales

via “knowledge distillation and model compression for downstream tasks”

Hugging Face's small model family for on-device use.

Unique: SmolLM's curated training data provides a high-quality teacher signal for distillation — student models distilled from SmolLM achieve better generalization than those distilled from generic large models; supports both response-based and feature-based distillation strategies

vs others: Models distilled from SmolLM 1.7B outperform models distilled from Llama 2 7B at equivalent student size due to better data quality, and distilled SmolLM students are 2-3x smaller than TinyLlama while maintaining comparable performance

Open-source reasoning model matching OpenAI o1.

Unique: Applies distillation to reasoning models across 6 different scales (1.5B-70B), which is rare for frontier reasoning models. Most competitors only offer single-size deployment.

vs others: Provides multiple distilled sizes enabling flexible deployment, whereas o1 only offers cloud API access at fixed capability level.

via “model distillation and knowledge transfer to smaller models”

Largest open-weight model at 405B parameters.

Unique: 405B enables distillation at unprecedented scale in open source, allowing creation of smaller models that inherit 405B's capabilities through synthetic data generation and knowledge transfer, previously unavailable in open-source ecosystem

vs others: Larger model scale enables higher-quality synthetic data and more effective distillation than smaller open-source models; however, inference cost for distillation is higher than proprietary distillation services

via “parameter-efficient reasoning through rl scaling”

Alibaba's 32B reasoning model with chain-of-thought.

Unique: Achieves reasoning performance comparable to 671B-parameter models through RL scaling on robust foundation models with outcome-based verification, demonstrating parameter-efficient reasoning through training approach rather than architectural compression

vs others: Delivers reasoning capability at 32B parameters competitive with 671B+ parameter models through RL training efficiency, enabling cost-effective and resource-efficient reasoning deployment compared to larger models

via “compact reasoning model with stem optimization”

Latest compact reasoning model with native tool use.

Unique: Domain-specific distillation trained on curated STEM datasets rather than general reasoning; uses sparse attention and quantized embeddings to compress reasoning capability into a mini-class model, achieving 10-50x cost reduction vs. o1/o3 while maintaining domain-specific reasoning quality.

vs others: Cheaper and faster than o1/o3 for STEM workloads (estimated 5-10x cost reduction, 3-5x latency reduction) but with narrower reasoning scope; stronger than GPT-4o on math/physics but weaker on general reasoning tasks requiring cross-domain knowledge.

via “distilled transformer inference with reduced parameter footprint”

zero-shot-classification model by undefined. 2,58,745 downloads.

Unique: Distilled from RoBERTa-Large specifically for NLI tasks using knowledge distillation, achieving 15x parameter reduction while maintaining >90% of teacher model accuracy on SNLI/MultiNLI benchmarks — most lightweight NLI alternatives either use non-distilled architectures or sacrifice accuracy more severely

vs others: Faster CPU inference than full-size cross-encoders (RoBERTa-Large, BERT-Large) by 3-5x; more accurate than simple bi-encoder baselines on entailment tasks due to cross-encoder architecture, despite smaller size

via “inference-optimization-via-model-distillation-from-70b-to-49b”

Llama-3.3-Nemotron-Super-49B-v1.5 is a 49B-parameter, English-centric reasoning/chat model derived from Meta’s Llama-3.3-70B-Instruct with a 128K context. It’s post-trained for agentic workflows (RAG, tool calling) via SFT across math, code, science, and...

Unique: Knowledge distillation from 70B to 49B with agentic-specific post-training preserves tool-calling and RAG performance while reducing parameters by 30%, enabling faster inference than 70B without generic distillation quality loss

vs others: More efficient than running full 70B model while maintaining better reasoning than smaller models like Llama-3.1-8B, though with some capability trade-off vs full 70B

via “knowledge distillation-based reasoning compression”

Aion-1.0-Mini 32B parameter model is a distilled version of the DeepSeek-R1 model, designed for strong performance in reasoning domains such as mathematics, coding, and logic. It is a modified variant...

Unique: Applies knowledge distillation to compress DeepSeek-R1's reasoning capability into 32B parameters, enabling reasoning-based inference at lower cost and latency than full R1

vs others: More efficient than full R1 (32B vs 671B) while retaining reasoning capability, though with unknown performance trade-offs vs. non-distilled reasoning models

via “knowledge distillation-based reasoning transfer”

DeepSeek R1 Distill Qwen 32B is a distilled large language model based on [Qwen 2.5 32B](https://huggingface.co/Qwen/Qwen2.5-32B), using outputs from [DeepSeek R1](/deepseek/deepseek-r1). It outperforms OpenAI's o1-mini across various benchmarks, achieving new...

Unique: Uses knowledge distillation to transfer R1's reasoning capability to a 32B model, enabling R1-quality reasoning at 1/3 parameter count through supervised fine-tuning on R1 outputs

vs others: More efficient than full R1 while maintaining reasoning quality, and more transparent than black-box reasoning models like o1 through explicit reasoning traces

via “knowledge distillation for custom model training”

Amazon Nova Premier is the most capable of Amazon’s multimodal models for complex reasoning tasks and for use as the best teacher for distilling custom models.

Unique: Amazon positions Nova Premier specifically as a distillation teacher with optimized output formats and intermediate representations designed for knowledge transfer, rather than as a general-purpose model that happens to support distillation as an afterthought

vs others: Designed from the ground up for distillation workflows with better cost-to-quality ratio than using GPT-4 or Claude as a teacher, making it more economical for teams building custom models at scale

via “compact model inference with cost-efficiency optimization”

OpenAI o4-mini-high is the same model as [o4-mini](/openai/o4-mini) with reasoning_effort set to high. OpenAI o4-mini is a compact reasoning model in the o-series, optimized for fast, cost-efficient performance while retaining...

Unique: Achieves reasoning capability compression through architectural distillation rather than simple parameter reduction, maintaining reasoning quality while reducing inference cost by 60-80% compared to full o-series models. The mini variant preserves the two-stage reasoning pipeline but with optimized computational allocation.

vs others: Cheaper than full o-series reasoning models while maintaining reasoning capabilities; more cost-effective than running multiple standard model calls for complex problems, but slower and more expensive than non-reasoning models like GPT-4 Turbo.

via “model distillation and compression for deployment”

Open Pretrained Transformers (OPT) by Facebook is a suite of decoder-only pre-trained transformers. [Announcement](https://ai.meta.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/).

via “model distillation and knowledge transfer techniques”

A book about implementing DeepSeek-style LLM architecture, training, and distillation methods.

Unique: Focuses on distillation techniques specifically adapted for DeepSeek architectures rather than generic distillation tutorials; likely covers distillation patterns for DeepSeek's specific architectural features (e.g., distilling mixture-of-experts models, handling attention pattern transfer, preserving reasoning capabilities in student models)

vs others: More targeted than general distillation resources because it addresses the specific challenges of compressing DeepSeek-style models while maintaining their distinctive capabilities, rather than applying generic distillation to arbitrary architectures