peft

Injects trainable low-rank decomposition matrices (LoRA) into transformer model layers by wrapping linear modules with a parallel adapter path that computes A @ B^T additions to activations. Uses a registry-based dispatch mechanism (src/peft/mapping.py) to identify target layers by name pattern, then replaces them with LoRALinear wrappers that maintain frozen base weights while training only the rank-r adapter matrices, achieving 0.1-2% parameter overhead per adapter.

AdaLoRA extends LoRA by maintaining per-layer importance scores that guide automatic rank allocation during training. The implementation computes Hadamard products of adapter gradients to estimate parameter importance, then dynamically increases ranks for high-importance layers and decreases ranks for low-importance ones, achieving 40-50% parameter reduction vs fixed-rank LoRA while maintaining task performance.

Provides merge_adapter() and unmerge_adapter() methods that fuse adapter weights into base model weights or extract them back out. For LoRA, merging computes (W + alpha/r * A @ B^T) to create a single set of weights, reducing inference latency by eliminating the adapter computation path. Unmerging recovers the original base weights and adapter weights from the merged state, enabling reversible adapter composition. Implemented through method-specific merge logic in each tuner class.

Validates PEFT configurations against model architecture and detects incompatibilities before training begins. The system checks that target_modules exist in the model, that adapter ranks are compatible with layer dimensions, and that method-specific constraints are satisfied. Implemented through PeftConfig validation methods and pre-training checks in get_peft_model() that raise informative errors for common misconfiguration patterns.

Enables fine-tuning of 4-bit and 8-bit quantized models by freezing the quantized base weights and training only adapter parameters, implemented through integration with bitsandbytes quantization library. The system detects quantized layers (Linear4bit, Linear8bit) and injects adapters in the forward pass without dequantizing base weights, reducing memory footprint by 75-90% compared to full-precision training while maintaining numerical stability through careful gradient flow management.

Enables loading and composing multiple adapters on a single base model through add_adapter(), set_adapter(), and delete_adapter() methods that manage an adapter registry. Supports sequential composition (stacking adapters), parallel composition (weighted averaging), and task-specific routing where different adapters activate based on input characteristics. Implemented via the PeftModel wrapper maintaining a dictionary of adapter states and switching between them without reloading the base model.

Implements prefix tuning and prompt tuning methods that prepend learnable soft prompt tokens to input sequences, optimizing only the prompt embeddings while freezing all model weights. The implementation maintains a learnable embedding matrix that is concatenated to input embeddings before the first transformer layer, enabling task adaptation through prompt optimization rather than weight updates. Supports both prefix (prepended to all layers) and prompt (prepended to input only) variants.

Provides save_pretrained() and from_pretrained() methods that serialize only adapter weights and configurations to disk, enabling efficient checkpoint storage and loading. The system saves adapter parameters as .safetensors or .bin files alongside adapter_config.json containing method-specific hyperparameters, supporting both local filesystem and HuggingFace Hub uploads. Implemented through a unified serialization interface (src/peft/utils/save_and_load.py) that abstracts method-specific serialization logic.

Enables distributed training across multiple GPUs/TPUs by synchronizing adapter gradients using standard PyTorch DistributedDataParallel (DDP) or DeepSpeed integration. The implementation treats adapters as regular parameters in the distributed training graph, with gradient accumulation and all-reduce operations handled by the distributed backend. Supports both data parallelism (same adapters across devices) and model parallelism (adapters sharded across devices) through integration with transformers' distributed training utilities.

Extends PEFT methods (LoRA, prefix tuning, etc.) to vision transformers (ViT, DeiT) and diffusion models (Stable Diffusion, DDPM) by identifying and wrapping attention/linear layers in these architectures. The implementation uses the same adapter injection mechanism as language models but adapts layer identification patterns for vision-specific architectures. Supports fine-tuning image generation, classification, and segmentation tasks with minimal parameter overhead.

Provides a plugin architecture for implementing new PEFT methods by extending BaseTuner and registering them in the method registry (src/peft/mapping.py). New methods define a configuration class (inheriting from PeftConfig), a tuner class (inheriting from BaseTuner), and register themselves via PEFT_TYPE_TO_CONFIG_MAPPING. The system automatically handles adapter lifecycle (initialization, forward pass injection, serialization) through the base class, enabling new methods to integrate with all PEFT infrastructure without reimplementation.

Enables fine-grained control over adapter training through layer-wise learning rate schedules and gradient clipping strategies. The implementation integrates with PyTorch optimizers to apply different learning rates to different adapter layers, and supports gradient accumulation patterns specific to adapter training. Implemented through integration with transformers' Trainer API and custom callback hooks that modify optimizer parameter groups per training step.