Build a Large Language Model (From Scratch)

Teaches the implementation of byte-pair encoding (BPE) tokenization from first principles, covering vocabulary construction, token merging algorithms, and handling special tokens. The guide walks through building a custom tokenizer that converts raw text into token IDs suitable for LLM input, including edge cases like unknown tokens and subword handling.

Covers the design and implementation of embedding layers that map discrete token IDs to continuous vector representations. Explains positional encoding schemes (absolute and relative), embedding initialization strategies, and the mathematical foundations of how embeddings enable the model to learn semantic relationships between tokens.

Covers the implementation of text generation by sampling tokens autoregressively: computing logits for the next token, applying temperature scaling and top-k/top-p filtering, sampling the next token, and repeating until a stop token or max length. Explains decoding strategies (greedy, beam search, sampling) and their tradeoffs.

Covers evaluation metrics for language models including perplexity (measuring prediction accuracy on held-out data), loss on validation sets, and task-specific metrics (BLEU for translation, ROUGE for summarization). Explains how to structure evaluation datasets, compute metrics efficiently, and interpret results to diagnose model issues.

Covers efficient data loading for training, including reading text files, tokenizing data, creating batches of appropriate size, and handling variable-length sequences. Explains padding strategies, batch construction for efficient GPU utilization, and how to structure data pipelines for fast training.

Covers saving model state (weights, optimizer state, training step) to disk and resuming training from checkpoints. Explains how to implement checkpointing strategies (periodic saves, best model tracking), handle distributed training checkpoints, and verify checkpoint integrity.

Covers the basics of distributed training across multiple GPUs or TPUs, including data parallelism (splitting batches across devices), gradient synchronization, and how to scale training to larger models. Explains communication patterns and synchronization points that affect training speed.

Provides detailed implementation of the multi-head self-attention mechanism, including query-key-value projections, scaled dot-product attention, and attention head concatenation. Covers the computational flow from input embeddings through attention weights to output representations, with explanations of why attention enables the model to focus on relevant tokens.

Covers the implementation of position-wise feedforward networks (FFN) that process each token independently through two linear transformations with a non-linearity (typically ReLU or GELU). Explains the role of the hidden dimension expansion factor and how FFN layers contribute to model capacity and non-linearity.

Teaches the implementation of layer normalization (normalizing across feature dimensions) and residual connections (skip connections that add input to output). Explains how these components stabilize training, enable deeper networks, and improve gradient flow through the model during backpropagation.

Combines attention, feedforward, normalization, and residual connections into a complete transformer block. Shows how to stack multiple blocks to build the full transformer encoder/decoder, including proper ordering of components (pre-norm vs post-norm architectures) and how information flows through the stack.

Explains the training objective for decoder-only LLMs: predicting the next token given previous tokens. Covers the implementation of causal masking (preventing attention to future tokens), loss computation (cross-entropy on predicted token logits), and how this objective enables autoregressive generation. Shows how to structure training data and compute per-token loss.

Covers the implementation of backpropagation through the transformer architecture, including gradient computation for each component (attention, FFN, embeddings) and how gradients flow backward through the network. Explains numerical stability considerations and how to debug gradient issues (vanishing/exploding gradients).

Covers initialization schemes for transformer weights (embeddings, attention projections, FFN layers) that affect training stability and convergence speed. Explains why random initialization matters, common schemes (Xavier/Glorot, He initialization), and how to initialize different layer types appropriately to maintain stable activation distributions.

Covers the implementation of optimization algorithms (SGD, Adam, AdamW) that update model parameters based on gradients. Explains momentum, adaptive learning rates, weight decay, and how these techniques improve convergence. Shows how to implement learning rate schedules and warmup strategies that improve training stability.