Dr. Sasha Rush
Cornell Tech / Hugging Face
Associate Professor at Cornell Tech and Researcher at Hugging Face. Former President of ICLR. Creator of 'The Annotated Transformer', the definitive line-by-line PyTorch guide that educated an entire generation of deep learning practitioners. Pioneer of Flash-Decoding and open structured generation engines.
Areas of focus
Professional niches
Proof of Work
Flash-Decoding: Accelerating Long-Context LLM Inference via Partitioned Attention
Co-developed Flash-Decoding, an inference parallelization algorithm that partitions key-value caches across thread blocks during token generation, delivering up to 8x speedup on 64k+ context sequences.
Requires reduction step across partitions that introduces minor overhead for very short sequence lengths.
Context: CUDA kernels, split-KV cache reduction, integrated into vLLM, TensorRT-LLM, and PyTorch 2.2.
View missionThe Annotated Transformer: A Line-by-Line PyTorch Implementation
Authored the universally referenced educational codebase providing an exact, executable line-by-line PyTorch implementation of 'Attention Is All You Need', bridging theoretical tensor math with production code.
Presents vanilla quadratic multi-head attention without later FlashAttention or grouped-query optimizations.
Context: PyTorch tensor operations, multi-head attention, positional encoding, and beam search decoding.
View mission