lora

Decomposes model weight updates into low-rank matrix products (W' = W + ΔW where ΔW = A×B^T) using trainable matrices A and B with rank d << min(n,m), reducing trainable parameters by 10-100× compared to full fine-tuning. Implements LoraInjectedLinear and LoraInjectedConv2d layer classes that wrap original weights and apply low-rank updates during forward passes without modifying base model weights.

Implements subject-specific fine-tuning by training on a small set of target images (3-5) while using class-prior images to prevent overfitting and catastrophic forgetting. The training loop alternates between updating the model on target images and regularizing with class images, using a weighted loss that balances concept learning against generalization. Integrates with LoRA to make this process memory-efficient.

Enables combining multiple trained LoRA adapters by stacking their low-rank updates (ΔW_total = α₁·ΔW₁ + α₂·ΔW₂ + ...) with learnable or fixed weights. Supports linear interpolation between LoRA models in weight space, enabling smooth transitions between different concepts or styles. Implements composition without retraining by directly manipulating weight matrices.

Enables applying multiple LoRA adapters during inference with per-step or per-layer weight scheduling. Supports dynamic adjustment of LoRA influence across diffusion timesteps, allowing different concepts to dominate at different denoising stages. Implements efficient inference by caching composed weights and avoiding redundant computation.

Provides CLI tool lora_ppim for automated preprocessing of training datasets including image resizing, cropping, augmentation, and caption generation. Handles batch operations on image directories, validates image quality, and generates metadata files required for training. Supports multiple preprocessing strategies (center crop, random crop, aspect-ratio preservation).

Learns new token embeddings in the CLIP text encoder's vocabulary space by optimizing a learnable embedding vector [V] that captures a concept's visual characteristics. During training, the model freezes all diffusion weights and only updates the embedding vector via backpropagation through the text encoder and UNet, allowing the model to bind arbitrary concepts to new tokens without modifying model weights.

Combines DreamBooth and Textual Inversion by jointly optimizing both LoRA weights and learned token embeddings during training. The method alternates between updating LoRA parameters on target images and refining the learned embedding, allowing the model to capture both structural adaptations (via LoRA) and semantic concept binding (via embeddings) simultaneously.

Extracts trained LoRA matrices (A and B) from fine-tuned models via extract_lora_ups_down function, enabling separation of adaptation weights from base model. Supports merging LoRA weights back into the original model (collapse_lora) to create standalone checkpoints, or composing multiple LoRA adapters by stacking their low-rank updates. Handles both safetensors and CKPT formats.

Extends LoRA training to inpainting tasks by conditioning the diffusion model on binary masks that specify regions to regenerate. During training, the model receives concatenated input (image + mask) and learns to denoise only masked regions while preserving unmasked content. Implements mask-aware loss weighting to focus gradient updates on inpainted areas.

Implements face-aware fine-tuning by integrating face detection and embedding extraction (via face recognition models) to preserve identity consistency across generated images. During training, the model receives face embeddings as additional conditioning signals, and the loss function includes face similarity terms to ensure generated faces maintain identity characteristics of the training set.

Integrates XFormers library's optimized attention implementations (flash attention, memory-efficient attention) into the diffusion model's forward pass to reduce peak memory usage and accelerate training. Automatically replaces standard PyTorch attention with XFormers kernels when available, reducing attention complexity from O(n²) to O(n) in sequence length.

Provides bidirectional conversion between CKPT (PyTorch pickle format) and safetensors (safe tensor format) while preserving LoRA weights and metadata. Handles format detection, weight mapping, and safe serialization to prevent data corruption. Supports loading models in either format and saving to preferred format without manual intervention.

Provides CLI tools (lora_pti, lora_add, lora_distill, lora_ppim) that abstract training complexity into simple command-line interfaces with YAML/JSON configuration files. Handles argument parsing, configuration validation, checkpoint management, and training loop orchestration without requiring code modification. Supports distributed training setup and experiment tracking.