Noam Shazeer
Distinguished AI Architect
Verified Proof of Work Artifacts
2 items catalogedEach artifact below represents an authenticated research publication, production code repository, or technical architectural framework directly authored or co-created by Noam Shazeer. Every entry undergoes editorial source verification.
Fast Transformer Decoding: One Write-Head is All You Need (Multi-Query Attention)
Authored the foundational paper introducing Multi-Query Attention (MQA), where all attention heads share a single key and value head. This drastically slashes memory bandwidth consumption during autoregressive generation by up to 10x with negligible accuracy drop.
Slightly reduces representational capacity in small model regimes compared to standard multi-head attention before being mitigated by modern GQA configurations.
GLU Variants Improve Transformer (SwiGLU Activation Function)
Formulated and benchmarked Gated Linear Unit variants (GLU, Bilinear, ReGLU, GEGLU, SwiGLU) for transformer feedforward layers, demonstrating that SwiGLU consistently outperforms standard ReLU and GELU activations.
Requires three weight matrices in feedforward blocks instead of two, requiring adjustments to intermediate hidden dimensions to maintain equivalent parameter counts.