transformers

Automatically detects model architecture from a model identifier string and instantiates the correct model class for PyTorch, TensorFlow, or JAX without explicit class specification. Uses a registry-based Auto* class system (AutoModel, AutoModelForCausalLM, etc.) that maps model names to their corresponding PreTrainedModel subclasses, enabling framework-agnostic model loading via a single unified API that queries the Hugging Face Hub's model card metadata.

Provides a framework-agnostic tokenization system that automatically selects the correct tokenizer (BPE, WordPiece, SentencePiece, etc.) based on model architecture and applies model-specific preprocessing rules (special tokens, padding, truncation). The AutoTokenizer class wraps 50+ tokenizer implementations and integrates with the Hub to download and cache tokenizer artifacts (vocab files, merge files, configs), while the Tokenizer base class enforces a consistent encode/decode interface across all implementations.

Provides an agents framework that enables language models to use external tools via structured function calling. The system automatically converts tool definitions into model-specific function schemas, manages tool execution and result handling, and supports agentic loops where models decide which tools to call based on task requirements. Integration with model-specific function-calling APIs (OpenAI, Anthropic, Ollama) enables seamless tool use across different model providers.

Integrates with Hugging Face Hub to enable seamless model discovery, downloading, and caching with support for remote code execution. Models can include custom modeling code that is automatically downloaded and executed when loading the model, enabling community contributions of novel architectures without requiring library updates. The caching system automatically manages model versions, handles network failures with retry logic, and supports offline mode for cached models.

Provides optimized implementations of attention mechanisms (scaled dot-product, multi-head, grouped-query, flash attention) with automatic selection of the fastest variant based on hardware and model configuration. Supports both dense and sparse attention patterns, enables flash attention for faster inference on compatible GPUs, and provides fallback implementations for unsupported hardware without requiring model changes.

Provides multiple positional embedding implementations (absolute, relative, rotary, ALiBi) with automatic selection based on model architecture and support for extrapolation beyond training sequence length. Enables models to generalize to longer sequences than seen during training through techniques like position interpolation and dynamic scaling, without requiring retraining.

Provides implementations of Mixture-of-Experts models with sparse routing mechanisms that selectively activate expert subsets based on input, reducing computation while maintaining model capacity. Supports different routing strategies (top-k, expert choice, load balancing) and integrates with distributed training to shard experts across devices, enabling efficient training and inference of large sparse models.

Provides a unified preprocessing pipeline for images, audio, and video that automatically selects the correct feature extractor (ImageProcessor, AudioProcessor, VideoProcessor) based on model architecture and applies model-specific normalization, resizing, and augmentation. The AutoProcessor class wraps feature extractors and tokenizers together, enabling end-to-end preprocessing of multimodal inputs (e.g., image + text for vision-language models) with a single call that handles alignment and batching across modalities.

Provides high-level task-specific pipelines (Pipeline class) that wrap model loading, preprocessing, inference, and postprocessing into a single callable interface for common NLP/vision tasks (text-generation, question-answering, image-classification, etc.). Each pipeline automatically selects the correct model and preprocessor, handles batching and device placement, and applies task-specific postprocessing (e.g., softmax for classification, beam search for generation) without requiring users to write boilerplate inference code.

Provides a Trainer class that orchestrates distributed training across multiple GPUs/TPUs with automatic gradient accumulation, mixed-precision training (FP16/BF16), learning rate scheduling, and checkpoint management. The Trainer integrates with PyTorch's DistributedDataParallel (DDP) and DeepSpeed for distributed training, automatically handles device placement and gradient synchronization, and supports custom training loops via callbacks without requiring users to write distributed training boilerplate.

Provides a flexible text generation system that supports multiple decoding strategies (greedy, beam search, sampling, constrained decoding) with fine-grained control over generation behavior via GenerationConfig and LogitsProcessor chains. The generation system automatically manages KV-cache for efficient autoregressive decoding, applies model-specific constraints (e.g., forced token sequences, vocabulary restrictions), and supports advanced features like assisted decoding and speculative decoding for faster inference without sacrificing quality.

Provides quantization support for reducing model size and accelerating inference through multiple precision formats (INT8, INT4, FP8, NF4) with automatic calibration and weight conversion. Integrates with bitsandbytes for 8-bit and 4-bit quantization, GPTQ for post-training quantization, and AWQ for activation-aware quantization, enabling users to load quantized models with a single config parameter without manual quantization code.

Integrates with PEFT (Parameter-Efficient Fine-Tuning) library to enable low-rank adaptation (LoRA), prefix tuning, and other adapter-based fine-tuning methods that update only a small fraction of model parameters while maintaining full model capacity. The integration automatically wraps pretrained models with adapter layers, manages adapter state during training and inference, and supports composing multiple adapters for multi-task learning without requiring full model retraining.

Provides utilities for converting model weights between different formats (PyTorch, TensorFlow, JAX, ONNX, SafeTensors) and frameworks without retraining. The conversion system automatically maps layer names across frameworks, handles dtype conversions (FP32, FP16, BF16), and validates weight integrity during conversion, enabling seamless model portability across the ML ecosystem.

Provides a standardized chat template system that automatically formats conversation history into model-specific prompt formats without manual string concatenation. The system supports role-based message formatting (user, assistant, system), automatic special token insertion, and model-specific prompt engineering patterns, enabling consistent multi-turn conversation handling across different chat models (Llama, Mistral, GPT, etc.).