Neural Networks: Zero to Hero Video Lecture Curriculum & Repository
The "Neural Networks: Zero to Hero" repository represents Andrej Karpathy's comprehensive pedagogical curriculum that systematically guides developers from raw mathematical basics to building full-scale modern GPT transformers. Spanning seven landmark lectures and companion code repositories, the series has educated hundreds of thousands of AI researchers and practitioners globally.
The curriculum begins with micrograd, deriving backpropagation from first principles. It then progresses through bigram language modeling, Bengio et al.'s 2003 MLP character model, batch normalization dynamics, WaveNet residual architectures, byte-pair encoding (tiktoken), and culminates in building GPT-2 entirely from scratch in PyTorch.
What distinguishes Zero to Hero from traditional university courses is its uncompromising hands-on approach: Karpathy writes every line of code live in Jupyter notebooks, demonstrates common failure modes (exploding/vanishing gradients, improper tensor initialization, broadcasting bugs), and teaches the intuitive debugging techniques used by top research teams.
The accompanying repository contains verified exercises, reference solutions, and Google Colab links that allow learners to reproduce every milestone independently on free cloud compute tiers.
Progresses systematically from scalar autograd to full GPT-2 transformer pre-training.
Derives every neural network component (attention, backprop, batchnorm) from raw mathematical equations.
Provides deep insights into weight initialization, activation histograms, and gradient health diagnostics.
Widely cited across frontier labs as the canonical onboarding curriculum for deep learning engineers.
Autograd Foundations
Derives backpropagation and gradient descent from scratch using the micrograd scalar engine.
Statistical & MLP Models
Builds bigram statistical baselines and Bengio 2003 neural language models with embedding layers.
Training Dynamics & BatchNorm
Analyzes tanh saturation, dead neurons, Kaiming weight initialization, and batch normalization mechanics.
Transformer & GPT Synthesis
Implements multi-head causal self-attention, residual connections, LayerNorm, and generative text sampling.
Pure PyTorch and custom Python micrograd engine; runnable on consumer GPUs or Google Colab.
Pedagogical and reference implementation. Optimized for code clarity, educational inspection, and conceptual clarity rather than maximum distributed hardware throughput.
What prerequisites are needed to follow Neural Networks: Zero to Hero?
Learners need comfortable familiarity with Python programming and basic high school calculus (derivatives) and linear algebra (vectors and matrices). Prior experience with PyTorch or machine learning is not required.
How does the curriculum handle transformer attention mechanics?
The series builds self-attention incrementally: starting from simple spatial averaging, introducing weighted keys and queries, adding value vectors, scaling by square root of head dimension, applying causal lower-triangular masks, and finally wrapping multiple heads into parallel projections.
This proof of work artifact was source-checked on Sep 20, 2026 by the AI Experts Directory editorial team. Our source review confirms that public code repositories, research papers, and technical artifacts directly corroborate Andrej Karpathy's active contributions. For full verification criteria, read our editorial methodology.
Inspect original artifact sources
Review raw code repositories, benchmark datasets, and technical citations directly on github.com.