Model Size Reduction Via Structured Pruning And Sparsity

Lightweight ML inference for mobile and edge devices.

Unique: Structured pruning removes entire filters/channels (not individual weights) to maintain hardware efficiency and avoid sparse tensor overhead. Uses magnitude-based or gradient-based importance scoring to identify prunable structures, then applies iterative fine-tuning to recover accuracy. Integrates with quantization pipeline for cumulative compression.

vs others: More hardware-efficient than unstructured pruning (which requires sparse tensor libraries) and more effective than simple weight decay regularization. Requires fine-tuning unlike quantization, but achieves higher compression ratios (30-50% vs. 4x from quantization alone).

via “model compression through pruning and distillation”

Microsoft's distributed training library — ZeRO optimizer, trillion-parameter scale, RLHF.

Unique: Combines structured pruning with knowledge distillation; supports both unstructured and structured sparsity patterns with automatic fine-tuning to recover accuracy

vs others: More integrated than separate pruning/distillation tools; automatic fine-tuning reduces manual tuning effort

via “structured and unstructured pruning with layer-wise sparsity patterns”

Toolkit for LLM quantization, pruning, and distillation.

Unique: Implements layer-wise pruning through a modifier system that applies sparsity masks to specific layer patterns, supporting both structured (channel/head removal) and unstructured (weight removal) pruning with automatic importance estimation from calibration data

vs others: More flexible than magnitude-based pruning because it supports learned importance scores; more practical than gradient-based pruning because it doesn't require training; better integrated with vLLM than generic sparse tensor libraries