Barret Zoph
Member of Technical Staff, OpenAI | Co-Inventor of Neural Architecture Search (NAS) & ST-MoE
Member of Technical Staff at OpenAI and former Staff Research Scientist at Google Brain. Co-inventor of Neural Architecture Search (NAS) using reinforcement learning, co-creator of AutoAugment, and lead architect of ST-MoE (Sparse Mixture-of-Experts) and PaLM-2 MoE architectures. Key contributor to ChatGPT and GPT-4 post-training, RLHF alignment, and reasoning architectures.
Areas of focus
Professional niches
Proof of Work
ST-MoE: Designing Stable and Transferable Sparse Mixture-of-Experts Models
Authored the foundational engineering playbook for training multi-hundred-billion parameter Sparse Mixture-of-Experts (MoE) models, introducing the router z-loss to eliminate numerical instability and routing collapse.
Sparse routing creates communication-heavy all-to-all cross-accelerator collective operations that require high-bandwidth interconnects (NVLink/InfiniBand).
Context: TPU v4 pods, Megatron/Jax, multi-billion parameter foundation language models.
View missionNeural Architecture Search with Reinforcement Learning
Introduced Neural Architecture Search (NAS), employing a recurrent neural network controller trained via reinforcement learning to generate model descriptions and automate neural network architecture design, winning seminal citations.
Original NAS required massive compute expenditures (thousands of GPU hours) before differentiable architecture search (DARTS) was invented.
Context: CIFAR-10, Penn Treebank, distributed TPU clusters, policy gradients.
View mission