AI research and projects
The research, projects, and explanations behind the people. Read the source, inspect verified code and models, and find a collaborator.
A source check confirms that a source supports the summary. It is not an independent replication or endorsement. Our methodology ↗
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (MAML)
Pieter Abbeel · UC Berkeley / CovariantFormulated the MAML meta-learning algorithm that optimizes model parameters such that a small number of gradient steps on a tiny amount of new data produces rapid adaptation across vision, regression, and robotics reinforcement learning tasks.
Second-order gradient computations through inner-loop optimization paths impose significant memory and compute overhead.
Context: Deep policy gradients and meta-optimization on Omniglot, MiniImageNet, and simulated robotic locomotion.
View missionMultimodal Datasets: Misogyny, Pornography, and Malign Content in LAION-400M
Abeba Birhane · Mozilla / Trinity College DublinConducted the rigorous empirical safety audit of the LAION-400M multimodal dataset used to train models like Stable Diffusion, exposing widespread presence of non-consensual imagery, racist caricatures, and toxic associations in web-scraped training pairs.
Automated keyword and similarity filtering misses nuanced contextual harms without intensive manual human review.
Context: Multimodal image-text pairs indexed via Common Crawl and CLIP cosine similarity filters.
View missionNeural Machine Translation by Jointly Learning to Align and Translate (Additive Attention)
Yoshua Bengio · Mila - Quebec AI Institute / Université de MontréalCo-authored the milestone paper introducing the attention mechanism into sequence-to-sequence translation, allowing decoders to dynamically attend to relevant input positions and directly inspiring the Transformer architecture.
Recurrent sequential processing precluded massive cluster-scale parallelization.
Context: Bidirectional RNN encoder with Bahdanau additive attention decoder on WMT14.
View missionPractical LoRA & QLoRA Fine-Tuning Parameter Analysis & Benchmarks
Sebastian Raschka · Lightning AIPublished empirical studies benchmarking the impact of rank (r), alpha scaling, target weight modules (Q, K, V vs all linear layers), and 4-bit NormalFloat quantization on model performance across Llama and Mistral.
Aggressive 4-bit quantization can induce subtle degradation on specialized domain syntax without rank compensation.
Context: 7B to 70B parameter models fine-tuned across consumer GPU hardware with LitGPT.
View missionProgress Measures for Grokking via Mechanistic Interpretability
Neel Nanda · Google DeepMindReverse-engineered the exact circular Fourier representation circuits formed when transformers 'grok' modular addition long after overfitting, proving that continuous generalization progresses silently before discontinuous performance jumps.
Toy model arithmetic circuits do not fully capture superposition and polysemanticity present in production-scale language models.
Context: One-layer transformers trained on modular arithmetic tasks with weight decay.
View missionRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Richard Socher · You.comIntroduced Recursive Neural Tensor Networks (RNTN) and the Stanford Sentiment Treebank, establishing how compositional semantic representations can be computed over grammatical parse trees to capture negation and complex sarcasm.
Tree-structured recursive models require syntactic parse trees as inputs, creating parsing error propagation bottlenecks.
Context: Recursive neural tensor networks evaluated on 11,855 parsed sentences from Rotten Tomatoes movie reviews.
View missionReinforcement Learning for Personalized Dynamic Treatment Regimes
Joelle Pineau · Meta FAIR / McGill UniversityPioneered offline batch reinforcement learning formulations optimizing multi-stage clinical interventions for chronic diseases including epilepsy and pediatric cancer, adapting dosages over time while minimizing toxic side effects.
High clinical safety thresholds require provable off-policy evaluation bounds before live patient deployment.
Context: POMDP algorithms evaluated against clinical patient trial trajectories.
View missionSafeTensors: Fast, Secure Tensor Storage for Deep Learning
Julien Chaumond · Hugging FaceSpearheaded the creation and adoption of SafeTensors, replacing insecure Python pickle files with zero-copy memory-mapped deserialization that prevents arbitrary code execution and boosts model loading speeds 2-10x.
Requires strict contiguous memory layout incompatible with raw arbitrary python object serialization.
Context: Rust-based tensor serialization format adopted universally across Hugging Face, PyTorch, and vLLM.
View missionScale AI Data Engine for Supervised Fine-Tuning & RLHF at Scale
Alexandr Wang · Scale AIEngineered human-in-the-loop expert annotation pipelines generating multi-million token pairs of complex instruction following, code review, and preference datasets powering the training of leading open and commercial models.
Human expert verification costs scale super-linearly with problem complexity in post-doctoral mathematics and legal reasoning.
Context: Distributed annotation platforms with automated inter-annotator agreement metrics and synthetic validation loops.
View missionScaling Instruction-Finetuned Language Models (Flan-T5 & Flan-PaLM)
Hyung Won Chung · OpenAISpearheaded the research demonstrating that instruction fine-tuning scaled across 1,836 diverse tasks and 540 billion parameters enhances task-general reasoning, chain-of-thought problem solving, and zero-shot performance across every benchmark.
Instruction tuning without careful task balance can degrade pure memorization capacity on open-domain fact retrieval.
Context: Flan-T5 and Flan-PaLM models evaluated on MMLU, Big-Bench, and HumanEval.
View missionScaling Laws for Neural Language Models
Dario Amodei · AnthropicDiscovered empirical power-law relationships between transformer performance (cross-entropy loss) and model parameter count, dataset size, and compute budget spanning over six orders of magnitude, providing the blueprint for modern frontier foundation model investment.
Scaling power laws hold primarily for pretraining loss; downstream task sample efficiency and multi-step reasoning require structured inference-time compute scaling.
Context: Autoregressive language models scaled from 10^3 to 10^9 non-embedding parameters across WebText.
View missionSEAL Leaderboards: Standardized Private Evals for Frontier LLM Benchmarking
Alexandr Wang · Scale AICreated the SEAL evaluation platform providing contamination-free, expert-curated benchmark suites testing advanced programming (Coding-GSM), mathematical rigor, and adversarial safety to combat public leaderboard gaming.
Requires ongoing prompt refresh cycles to prevent indirect test data leakage into web-crawled pretraining corpora.
Context: Private held-out dynamic evaluation testbeds continuously updated by human domain experts.
View missionSequence to Sequence Learning with Neural Networks
Ilya Sutskever · Safe Superintelligence Inc. (SSI)Pioneered the multi-layered Long Short-Term Memory (LSTM) encoder-decoder architecture mapping arbitrary sequence lengths to variable output sequences. Revolutionized machine translation, conversational AI, and established the direct conceptual architecture upon which transformers and autoregressive LLMs were built.
Recurrent execution bottlenecked parallelization across large cluster topologies before self-attention mechanisms emerged.
Context: Deep LSTM networks, WMT14 English-to-French benchmark, 160,000 input vocabulary.
View missionShow and Tell: A Neural Image Caption Generator
Oriol Vinyals · Google DeepMindPioneered the integration of deep convolutional neural networks (CNNs) for vision encoding with recurrent language models (RNNs) for text generation, producing the first end-to-end neural image captioning pipeline.
Susceptible to hallucinating objects based on strong linguistic prior frequencies rather than verified visual presence.
Context: GoogLeNet CNN feature extractor combined with LSTM language decoder on MSCOCO dataset.
View missionSparks of Artificial General Intelligence: Early experiments with GPT-4
Sebastien Bubeck · Microsoft ResearchAuthored the 154-page seminal investigation demonstrating that early non-multimodal GPT-4 exhibited emergent general intelligence across mathematics, coding, law, vision-via-code, psychology, and medicine without domain-specific fine-tuning.
Highlighted critical remaining flaws including lack of long-term planning, working memory degradation, and blind confidence in false premises.
Context: Unconstrained early GPT-4 base model checkpoint evaluated across novel out-of-distribution reasoning probes.
View missionStanford CS224N: Natural Language Processing with Deep Learning
Christopher Manning · Stanford UniversityCreated and delivered the globally benchmarked Stanford university curriculum educating over one million students online on word vectors, recurrent networks, attention mechanisms, transformers, and modern large language models.
Curriculum must be overhauled annually to keep pace with rapid shifts in foundation model post-training and inference techniques.
Context: PyTorch deep learning pipelines spanning pretraining, fine-tuning, and self-attention interpretability.
View mission