nanoGPT: Minimal, Readable, and Fast PyTorch Framework for Training Medium GPTs
nanoGPT stands as one of the most influential educational and engineering artifacts in the modern generative AI landscape. Created by Andrej Karpathy, the repository fundamentally demystified transformer pre-training by stripping away hundreds of thousands of lines of enterprise library boilerplate, reducing the core training loop to two compact files (train.py and model.py) that can be read, understood, and modified in an afternoon.
By intentionally avoiding opaque abstraction layers, nanoGPT offers pure PyTorch mechanics: raw tensor operations, multi-head causal self-attention projections, rotary or learned positional embeddings, and PyTorch 2.0 torch.compile integration with FlashAttention-2. This design enables individual researchers and students to reproduce GPT-2 (124M parameters) on a single 8x A100 node in under four days for ~$100 in cloud compute, or train a character-level Shakespeare model on a MacBook CPU in minutes.
Beyond education, nanoGPT serves as the baseline testbed across leading AI research labs for benchmarking novel optimizer proposals (such as Sophia, Muon, or Lion), custom CUDA kernels, and memory-efficient attention variants. Its uncompromising clarity has trained a generation of frontier AI engineers who build today's leading LLM infrastructure.
The architectural footprint relies on minimal external dependencies—solely requiring PyTorch, tiktoken for BPE encoding, numpy, and optional wandb logging. By avoiding distributed runtime frameworks like Ray or DeepSpeed for single-node setups, it illuminates the fundamental mathematics of backpropagation and gradient descent in modern autoregressive sequence models.
Complete GPT-2 training in ~300 lines of clean, readable PyTorch (train.py and model.py) without opaque abstraction layers.
Native integration with torch.compile, FlashAttention-2, and automatic mixed precision (BF16/FP16) yielding near-cuDNN peak FLOPS.
Validates GPT-2 (124M, 350M, 774M, 1558M) pre-training convergence on OpenWebText to target validation loss within 0.01 cross-entropy.
Universally adopted as the de facto testbed for experimentation with new optimizers (Muon, Lion) and CUDA kernels.
Token Ingestion & Sharding
BPE tokenization using tiktoken (gpt2 vocabulary of 50,257 tokens), sharded into raw uint16 binary memmaps for zero-copy memory loading.
Causal Transformer Core
Standard decoder-only transformer with LayerNorm, GELU activations, residual stream connections, and fused scaled dot-product self-attention.
Distributed Optimization
AdamW optimizer with decoupled weight decay, cosine learning rate schedule with linear warmup, and DistributedDataParallel (DDP) gradient synchronization.
Validation & Checkpoint Export
Periodic validation loss calculation over unseen tokens, generation sampling with top-k filtering, and SafeTensors/PyTorch checkpoint state preservation.
PyTorch 2.0, CUDA, Python, FlashAttention, and OpenWebText.
Engineered for educational transparency and small-to-medium parameter models; multi-node distributed trillion-token training requires complex pipeline parallel frameworks like Megatron.
How does nanoGPT compare to Hugging Face Transformers or Megatron-LM?
nanoGPT prioritizes extreme readability and hackability over enterprise multi-model generality. While Hugging Face wraps models in deep class hierarchies and Megatron-LM targets 10,000+ GPU tensor-parallel clusters, nanoGPT keeps the entire architecture in 300 lines of clear PyTorch, making it ideal for rapid prototyping, learning, and algorithmic research on 1 to 8 GPUs.
Can nanoGPT be used to fine-tune modern LLMs like Llama 3 or Mistral?
While nanoGPT is primarily designed for pre-training and fine-tuning GPT-2 class architectures, its clean codebase is frequently used as an educational stepping stone for building custom fine-tuning scripts. For modern Llama 3 or Mistral models with RoPE, GQA, and SwiGLU, practitioners often transition to Karpathy's companion repositories like llm.c or community forks adapting nanoGPT to Llama architectures.
What are the minimal hardware requirements to run nanoGPT?
To train a toy character-level language model, any consumer laptop (even a CPU or Apple Silicon Mac) can train in 5-10 minutes. For training the full 124M parameter GPT-2 on OpenWebText, a modern GPU with at least 8GB to 12GB of VRAM (e.g. RTX 3060/4070) is sufficient for small batch sizes, while an 8x A100 (40GB/80GB) rig completes the full run in ~4 days.
How does torch.compile accelerate nanoGPT?
PyTorch 2.0's torch.compile inspects the computational graph of the model, fuses elementwise operations (such as bias addition and GELU activations), and eliminates Python interpreter roundtrips between GPU kernel launches. In nanoGPT, this produces a ~25-35% speedup without altering numerical convergence.
This proof of work artifact was source-checked on Sep 20, 2026 by the AI Experts Directory editorial team. Our source review confirms that public code repositories, research papers, and technical artifacts directly corroborate Andrej Karpathy's active contributions. For full verification criteria, read our editorial methodology.
Inspect original artifact sources
Review raw code repositories, benchmark datasets, and technical citations directly on github.com.