Finetuning Large Language Models - DeepLearning.AI

Teaches LLMs to follow specific instructions and output formats by training on curated examples of input-output pairs. Uses standard supervised learning with cross-entropy loss on the model's next-token prediction, where the model learns to replicate desired behaviors from labeled examples rather than relying solely on base model pretraining. The course covers dataset preparation, loss computation strategies, and validation approaches to ensure the model generalizes beyond memorization.

Reduces fine-tuning computational cost and memory requirements by training only small adapter modules (LoRA, QLoRA) instead of all model parameters. Uses low-rank decomposition to approximate weight updates as A × B^T where A and B are small matrices, reducing trainable parameters from millions to thousands while maintaining performance. The course covers how to integrate adapters into transformer architectures, merge them with base weights, and stack multiple adapters for multi-task learning.

Provides frameworks for collecting, cleaning, and validating training data to ensure fine-tuning effectiveness. Covers techniques like data augmentation, deduplication, filtering for quality, and stratification to create balanced datasets. The course teaches how to identify and remove low-quality examples, detect distribution shifts between training and validation data, and measure dataset quality metrics that correlate with fine-tuned model performance.

Establishes frameworks for measuring fine-tuned model performance beyond simple loss metrics, including task-specific evaluation, human evaluation protocols, and detecting overfitting. Covers techniques like hold-out validation sets, cross-validation, benchmark datasets, and defining success metrics aligned with business objectives. The course teaches how to compare fine-tuned models against baselines and identify when a model has overfit to training data.

Covers advanced fine-tuning approaches for scenarios with multiple tasks or domains, including multi-task learning, continual learning, and domain adaptation. Teaches how to structure training data and loss functions to prevent catastrophic forgetting when fine-tuning on new tasks, and how to leverage shared representations across domains. Includes techniques like task-specific adapters, weighted loss combinations, and curriculum learning.

Covers techniques for deploying fine-tuned models efficiently in production, including quantization, batching, caching, and serving infrastructure. Teaches how to integrate fine-tuned models with inference frameworks (vLLM, TensorRT, ONNX) to reduce latency and memory footprint. Includes strategies for A/B testing fine-tuned models against baselines and monitoring performance in production.

Provides practical tutorials for fine-tuning using managed fine-tuning services from OpenAI (GPT-3.5, GPT-4) and Anthropic (Claude). Covers API-based fine-tuning workflows without requiring local GPU infrastructure, including data formatting, job submission, monitoring, and evaluation. Teaches when to use API-based fine-tuning vs. open-source models, and how to manage costs and quotas.

Specializes fine-tuning techniques for code-related tasks, including code completion, bug fixing, code review, and test generation. Covers code-specific data preparation (handling multiple programming languages, code formatting), evaluation metrics (pass@k, compilation success), and preventing the model from generating syntactically invalid code. Includes techniques like in-context examples and chain-of-thought prompting for code tasks.

Teaches fine-tuning techniques for specialized domains like legal, medical, scientific, or financial text, where domain vocabulary and conventions are critical. Covers domain-specific data preparation, handling technical terminology, and preventing hallucinations on domain-specific facts. Includes techniques for incorporating domain knowledge (ontologies, knowledge graphs) into fine-tuning and evaluating factual accuracy.