Neural Networks: Zero to Hero - Andrej Karpathy

Delivers structured video lectures that progressively build neural network understanding from mathematical foundations through implementation, using a pedagogical approach that alternates between conceptual explanation and live coding demonstrations. Each lecture combines whiteboard derivations of backpropagation, gradient descent, and activation functions with real-time implementation in Python/PyTorch, enabling learners to see theory-to-code mapping directly.

Provides a complete walkthrough of building a minimal automatic differentiation engine (micrograd) from scratch in Python, demonstrating how computational graphs track operations, how backpropagation traverses these graphs to compute gradients, and how gradient descent updates parameters. The implementation uses a directed acyclic graph (DAG) structure where each operation node stores references to its inputs and a backward function, enabling reverse-mode autodiff.

Introduces convolutional neural networks by explaining how convolution operations extract spatial features, how pooling reduces dimensionality, and how stacking these layers builds hierarchical feature representations. The implementation shows how to implement convolution as a sliding window operation, how to compute gradients through convolution, and how to design CNN architectures for image tasks.

Explains recurrent neural networks by showing how they maintain hidden state across time steps, how unrolling creates a computation graph through time, and how backpropagation through time (BPTT) computes gradients. Demonstrates the RNN equations (hidden state update, output computation) and discusses challenges like vanishing/exploding gradients that arise from long sequences.

Walks through building a complete training loop that orchestrates forward passes, loss computation, backward passes, and parameter updates, demonstrating how these components interact in sequence. The implementation shows explicit gradient zeroing, loss calculation, backpropagation invocation, and optimizer steps, revealing the control flow and state management required for iterative training.

Demonstrates how to design and implement fully-connected neural networks with multiple hidden layers, including decisions about layer sizes, activation functions, and weight initialization. The implementation shows how to compose layers sequentially, how activation functions introduce non-linearity, and how network depth affects expressiveness and training dynamics.

Provides a complete mathematical derivation of the backpropagation algorithm using the chain rule, showing how gradients flow backward through a network from loss to parameters. The implementation demonstrates both the mathematical formulation (partial derivatives, Jacobians) and the computational implementation (storing intermediate activations, computing gradients layer-by-layer), revealing how the algorithm achieves efficiency through dynamic programming.

Analyzes different activation functions (ReLU, sigmoid, tanh, etc.) by examining their mathematical properties, derivatives, and effects on network training. The analysis includes visualization of activation curves, gradient flow properties, and empirical comparison of how different activations affect convergence speed and final accuracy on benchmark problems.

Covers how to design and implement loss functions for different ML tasks (classification, regression, etc.), including mathematical formulation, gradient computation, and implementation in code. Demonstrates how loss function choice affects what the network learns and how to debug loss computation issues.

Explains different optimization algorithms (SGD, momentum, Adam, etc.) by deriving their update rules, analyzing their convergence properties, and comparing their empirical performance on training tasks. Demonstrates how each algorithm modifies the basic gradient descent update and what problems each solves (e.g., momentum for accelerating convergence, adaptive learning rates for handling different gradient scales).

Explains batch normalization by deriving how it normalizes activations across a batch, reducing internal covariate shift and enabling higher learning rates. The implementation shows the forward pass (computing batch statistics, normalizing, scaling/shifting), the backward pass (computing gradients through normalization), and how batch statistics differ between training and inference.

Covers regularization methods (L1/L2 weight decay, dropout, early stopping, data augmentation) by explaining their mathematical basis and empirical effects on generalization. Demonstrates how each technique modifies the training objective or data distribution to reduce overfitting and improve test performance.