Axolotl

Declarative configuration system that translates YAML training recipes into executable fine-tuning pipelines. Uses a schema-driven approach to validate and parse training parameters (model architecture, learning rates, batch sizes, optimization strategies) into Python objects that drive the training loop. Eliminates boilerplate by centralizing all hyperparameters, data paths, and training strategies in a single human-readable file that can be version-controlled and shared across teams.

Abstraction layer that handles fine-tuning across diverse model architectures (LLaMA, Mistral, Phi, Qwen, etc.) through a single training pipeline. Internally detects model architecture from HuggingFace model cards, applies architecture-specific tokenization and attention patterns, and routes training through the appropriate PyTorch modules. Supports both base models and instruction-tuned variants without requiring separate training scripts per architecture.

Integrated validation loop that evaluates model performance on held-out data at configurable intervals during training. Supports custom evaluation metrics (perplexity, BLEU, exact match, F1) and early stopping based on validation performance. Automatically saves best-performing checkpoints and logs validation metrics to WandB. Handles metric computation across distributed training setups with proper synchronization.

Specialized data formatting system for instruction-tuning workflows that converts raw user/assistant conversation data into model-compatible prompt sequences. Supports multiple prompt templates (Alpaca, ChatML, Llama2, Mistral, etc.) with automatic template selection based on model architecture. Handles multi-turn conversations, system prompts, and special token insertion. Validates prompt formatting and provides debugging output for malformed data.

Automatically calculates effective batch size based on per-device batch size, number of GPUs, and gradient accumulation steps. Axolotl handles gradient accumulation logic transparently, allowing users to specify desired effective batch size in YAML and automatically computing accumulation steps. This enables training with large effective batch sizes on limited GPU memory.

Applies architecture-specific optimizations automatically: Flash Attention v2 for faster attention computation, RoPE (Rotary Position Embedding) scaling for longer context windows, and other model-specific tweaks. Axolotl detects model architecture and applies relevant optimizations via transformers library integrations. Flash Attention reduces attention complexity from O(n²) to O(n) with minimal accuracy loss.

Implements Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA) through integration with the PEFT (Parameter-Efficient Fine-Tuning) library. Automatically injects trainable low-rank decomposition matrices into model attention and linear layers while freezing base model weights. For QLoRA, additionally quantizes base model weights to 4-bit precision using bitsandbytes, reducing memory footprint by 75%+ while maintaining training quality. Configuration-driven rank selection, alpha scaling, and target module specification allow fine-grained control over adapter architecture.

Abstracts distributed training complexity through automatic detection of available GPUs and configuration of PyTorch Distributed Data Parallel (DDP) or DeepSpeed backends. Handles gradient accumulation, mixed-precision training (FP16/BF16), and synchronization across devices without requiring manual distributed training code. Supports both single-node multi-GPU and multi-node setups through environment variable detection and automatic rank/world-size configuration.

Automated data loading and preprocessing system that handles multiple input formats (JSON, CSV, Parquet, HuggingFace datasets) and applies architecture-specific tokenization. Supports dataset concatenation, filtering, and sampling through configuration. Implements prompt templating for instruction-tuning datasets, automatically formatting user/assistant exchanges into model-compatible sequences. Handles special tokens, padding, and truncation with configurable strategies (e.g., 'right' padding for causal LMs, 'max_length' truncation).

Integrates post-training quantization through GPTQ (Generative Pre-trained Transformer Quantization) and GGUF (GPT-Generated Unified Format) export pipelines. Supports 4-bit and 8-bit quantization with configurable group sizes and dynamic activation quantization. After fine-tuning, automatically exports models to GGUF format for CPU inference or GPTQ format for GPU inference with minimal accuracy loss. Quantization parameters are configuration-driven, allowing experimentation without code changes.

Built-in integration with Weights & Biases (WandB) for real-time training metrics visualization, hyperparameter logging, and experiment comparison. Automatically logs loss curves, learning rates, gradient norms, and custom metrics to WandB dashboards. Supports local logging fallback and configuration-driven metric selection. Enables reproducibility through automatic logging of training configuration, model architecture, and dataset metadata to experiment records.

Automated checkpoint saving, loading, and resumption system that persists model state, optimizer state, and training metadata at configurable intervals. Supports resuming training from any checkpoint without data reprocessing. Includes model merging utilities for combining LoRA adapters back into base models, converting between formats (SafeTensors, PyTorch, HuggingFace), and creating inference-ready artifacts. Handles checkpoint cleanup to manage disk space on long training runs.

Extensible training objective system supporting standard supervised fine-tuning (SFT), DPO (Direct Preference Optimization), and custom loss functions. Allows configuration-driven selection of training objectives without code changes. Supports weighted loss combinations for multi-task training and custom loss implementations through Python function registration. Handles special token masking (e.g., ignoring padding tokens in loss calculation) automatically based on model configuration.

Post-training export pipeline that prepares fine-tuned models for inference deployment. Automatically converts models to optimized formats (SafeTensors, ONNX, TensorRT), generates inference configs, and bundles tokenizers with model weights. Supports exporting both full models and LoRA adapters, with optional quantization during export. Generates deployment-ready artifacts including model cards, usage examples, and configuration files for popular inference frameworks (vLLM, TGI, llama.cpp).