AI research and projects
The research, projects, and explanations behind the people. Read the source, inspect verified code and models, and find a collaborator.
A source check confirms that a source supports the summary. It is not an independent replication or endorsement. Our methodology ↗
Scaling Laws for Neural Language Models
Dario Amodei · AnthropicDiscovered empirical power-law relationships between transformer performance (cross-entropy loss) and model parameter count, dataset size, and compute budget spanning over six orders of magnitude, providing the blueprint for modern frontier foundation model investment.
Scaling power laws hold primarily for pretraining loss; downstream task sample efficiency and multi-step reasoning require structured inference-time compute scaling.
Context: Autoregressive language models scaled from 10^3 to 10^9 non-embedding parameters across WebText.
View missionSequence to Sequence Learning with Neural Networks
Ilya Sutskever · Safe Superintelligence Inc. (SSI)Pioneered the multi-layered Long Short-Term Memory (LSTM) encoder-decoder architecture mapping arbitrary sequence lengths to variable output sequences. Revolutionized machine translation, conversational AI, and established the direct conceptual architecture upon which transformers and autoregressive LLMs were built.
Recurrent execution bottlenecked parallelization across large cluster topologies before self-attention mechanisms emerged.
Context: Deep LSTM networks, WMT14 English-to-French benchmark, 160,000 input vocabulary.
View missionShow and Tell: A Neural Image Caption Generator
Oriol Vinyals · Google DeepMindPioneered the integration of deep convolutional neural networks (CNNs) for vision encoding with recurrent language models (RNNs) for text generation, producing the first end-to-end neural image captioning pipeline.
Susceptible to hallucinating objects based on strong linguistic prior frequencies rather than verified visual presence.
Context: GoogLeNet CNN feature extractor combined with LSTM language decoder on MSCOCO dataset.
View missionSparks of Artificial General Intelligence: Early experiments with GPT-4
Sebastien Bubeck · Microsoft ResearchAuthored the 154-page seminal investigation demonstrating that early non-multimodal GPT-4 exhibited emergent general intelligence across mathematics, coding, law, vision-via-code, psychology, and medicine without domain-specific fine-tuning.
Highlighted critical remaining flaws including lack of long-term planning, working memory degradation, and blind confidence in false premises.
Context: Unconstrained early GPT-4 base model checkpoint evaluated across novel out-of-distribution reasoning probes.
View missionTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Jeff Dean · Google / Google DeepMindCo-architected TensorFlow, the open-source computation graph framework that unified distributed deep learning execution across mobile devices, CPU servers, GPU clusters, and TPU pods, catalyzing the deep learning revolution.
Static computational graph compilation initially hindered dynamic Pythonic debugging compared to eager execution paradigms.
Context: Heterogeneous distributed computation graphs supporting thousands of distributed workers.
View missionTextbooks Are All You Need (The Phi Model Series)
Sebastien Bubeck · Microsoft ResearchDemonstrated that a 1.3-billion parameter transformer trained on 6 billion tokens of curated synthetic 'textbook quality' data achieved 50.6% on HumanEval, dramatically outperforming models 10x its size trained on hundreds of billions of uncurated web tokens.
Synthetic data diversity must be aggressively managed to prevent model collapse and repetition loops on out-of-distribution prompts.
Context: Phi-1 and Phi-2 autoregressive models trained on synthetic Python exercises and educational textbooks curated by GPT-3.5.
View missionThe Representation of Truth & World Models in Large Language Models
Max Tegmark · MIT / Future of Life InstituteDiscovered that large language models build linearly readable internal representations of truth and physical space/time, demonstrating that models do not merely emulate shallow stochastic parrots but construct coherent world models.
Probes capture model belief states which can diverge from actual external physical reality.
Context: Llama-2 and Pythia checkpoints probed via principal component analysis and logistic probes across factual statements.
View missionA Foundation Model for Clinician-Centered Drug Repurposing with TxGNN
Dr. Kexin Huang · Stanford University / Harvard UniversityFirst author of the Nature Medicine 2024 paper introducing TxGNN, a geometric foundation model trained on a medical knowledge graph of 17,080 diseases to identify drug repurposing opportunities for rare diseases with clinician-interpretable rationales.
Predictions require in vitro wet-lab assay validation and clinical trials before therapeutic efficacy can be established.
Context: Graph Neural Networks, 17k disease knowledge graph, clinician explanatory subgraphs.
View missionAggregated Residual Transformations for Deep Neural Networks (ResNeXt)
Dr. Saining Xie · New York University (NYU)Co-authored the seminal CVPR 2017 paper introducing ResNeXt, which introduced 'cardinality' (the size of the set of transformations) as a concrete, essential dimension that outclasses depth and width for computer vision accuracy.
Grouped convolution memory access patterns can yield lower tensor-core utilization compared to standard dense GEMMs.
Context: Grouped convolutions, split-transform-merge topology, ImageNet-1k classification.
View missionAWQ: Activation-aware Weight Quantization for On-Device LLM Compression
Dr. Ji Lin · NVIDIA / MITAuthored the MLSys 2024 Best Paper introducing AWQ, demonstrating that protecting the top 1% salient weight channels based on activation magnitudes avoids backpropagation retraining and achieves lossless 4-bit inference.
Extreme 2-bit quantization still requires second-order error compensation beyond magnitude scaling.
Context: W4A16 GEMM kernels, LLaMA, Mistral, CodeLlama, NVIDIA TensorRT-LLM and vLLM runtimes.
View missionBenchmarking Vision-Language Models on Live Web Interaction
Gregor Zunic · Browser-UseConducted empirical benchmarks assessing vision versus accessibility-tree DOM representation for autonomous agents, proving multimodal screenshot parsing reduces failure rates by 35% on checkout flows.
High resolution screenshot ingestion increases token consumption significantly per navigation step.
Context: Claude 3.5 Sonnet vision, WebVoyager benchmark dataset, automated browser sessions.
View missionBenchmarking Web Extraction Quality for Large Language Models
Minh Do · Crawl4AIConducted empirical benchmarks demonstrating a 6x speedup and 70% reduction in context token waste by stripping non-semantic DOM elements prior to RAG embedding.
Complex nested web tables with multiple header levels require custom parsing logic.
Context: DOM pruning algorithms, cosine similarity RAG retrieval, token economics.
View missionBLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Dr. Thomas Wolf · Hugging FaceInitiated and directed the BigScience open scientific collaboration of 1,000+ researchers, pre-training BLOOM on the Jean Zay supercomputer across 46 natural languages and 13 programming languages.
Massive parameter size requires distributed multi-node hardware (8x A100 80GB minimum) for unquantized inference.
Context: 176B parameters, Megatron-DeepSpeed 3D parallelism, ALiBi positional embeddings.
View missionColorful Image Colorization with Deep Convolutional Neural Networks
Dr. Alexei Efros · UC BerkeleyPioneered automated black-and-white image colorization by formulating the task as multinomial classification over quantized Lab color space with class rebalancing, tricking human observers into believing synthesized colors are real.
Ambiguous objects (e.g. clothing, cars) can display desaturated or sepia tints if the model averages multimodal color distributions.
Context: CIE Lab color space, class-rebalanced cross-entropy loss, 1.3M ImageNet images.
View missionConnectionist Temporal Classification: Labelling Unsegmented Sequence Data
Dr. Alex Graves · NNAISENSE / former Google DeepMindAuthored the monumental ICML 2006 paper introducing CTC loss, allowing RNNs to be trained directly on unsegmented, unaligned sequence pairs by integrating over all possible alignments with a blank token, with over 15,000 citations.
Assumes conditional independence between label outputs given the input sequence, necessitating external language models for grammar smoothing.
Context: Dynamic programming forward-backward algorithm, RNN sequence alignment.
View missionContrastive Multiview Coding and Self-Supervised Representation Learning
Dr. Phillip Isola · MITIntroduced Contrastive Multiview Coding (CMC), showing that maximizing mutual information across multiple views or sensory modalities learns invariant visual representations competitive with full ImageNet supervision.
Requires view-generation or multiple aligned cameras/modalities during pre-training to compute contrastive pairs.
Context: Multi-sensory embedding spaces, contrastive InfoNCE loss.
View mission