Noam Shazeer
Distinguished AI Architect
Sources checkedDistinguished AI Researcher and Tech Lead at Google DeepMind. Former co-founder & CEO of Character.ai. Co-author of 'Attention Is All You Need'. Principal inventor of several of the most ubiquitous deep learning mechanisms in production today, including Multi-Query Attention (MQA), SwiGLU activations, and Rotary Position Embeddings (RoPE) adaptations.
Areas of focus
Professional niches
Proof of Work
Fast Transformer Decoding: One Write-Head is All You Need (Multi-Query Attention)
Authored the foundational paper introducing Multi-Query Attention (MQA), where all attention heads share a single key and value head. This drastically slashes memory bandwidth consumption during autoregressive generation by up to 10x with negligible accuracy drop.
Slightly reduces representational capacity in small model regimes compared to standard multi-head attention before being mitigated by modern GQA configurations.
Context: Decoupled query/key/value projection matrices; foundation for Grouped-Query Attention (GQA) across modern LLMs like Llama 3 and Mistral.
View missionGLU Variants Improve Transformer (SwiGLU Activation Function)
Formulated and benchmarked Gated Linear Unit variants (GLU, Bilinear, ReGLU, GEGLU, SwiGLU) for transformer feedforward layers, demonstrating that SwiGLU consistently outperforms standard ReLU and GELU activations.
Requires three weight matrices in feedforward blocks instead of two, requiring adjustments to intermediate hidden dimensions to maintain equivalent parameter counts.
Context: Swish-gated linear projections: SwiGLU(x) = Swish(xW) * (xV).
View mission