AI research and projects
The research, projects, and explanations behind the people. Read the source, inspect verified code and models, and find a collaborator.
A source check confirms that a source supports the summary. It is not an independent replication or endorsement. Our methodology ↗
LangChain and LLM Agent Notebook Implementations
Sam Witteveen · Red Dragon AIAuthored dozens of open-source Jupyter notebooks with thousands of stars demonstrating tool integration, multi-agent debate, and local RAG retrieval pipelines.
Notebook workflows require refactoring into stateless services for production high-concurrency environments.
Context: LangChain, ChromaDB, Hugging Face Transformers, Ollama.
View missionLangGraph and CrewAI Production Agent Blueprints
Kris (All About AI) · All About AIAuthored and open-sourced production agent starter repositories with thousands of stars, implementing tool-calling loops, document retrieval, and structured JSON output schemas.
Asynchronous task execution requires careful error-handling proxies around flaky external web scrapers.
Context: Python, FastAPI, LangGraph, Ollama, OpenAI Assistant API.
View missionLarge-Scale Distributed Pre-Training and Sharded Tensor Parallelism
Timothée Lacroix · Mistral AIArchitected the cluster infrastructure utilizing Megatron-LM and custom NCCL collective communication topologies that pre-trained the Mixtral family across thousands of H100 GPUs with near-linear scaling efficiency.
Inter-node all-to-all communications in large MoE routing can bottleneck throughput on non-InfiniBand cloud interconnects.
Context: H100 NVLink clusters, InfiniBand HDR/NDR fabric, FP8 mixed precision.
View missionLearning to Learn with Generative Models of Neural Network Checkpoints
William Peebles · OpenAIAuthored the NeurIPS 2022 paper exploring generative modeling of neural network parameter checkpoints, demonstrating that diffusion models can synthesize functioning weights of performant downstream classifiers.
Permutation symmetry in high-dimensional weight spaces makes scaling to multi-billion parameter checkpoints difficult.
Context: Graph hypernetworks, weight-space diffusion, permutation invariance.
View missionLearning Transferable Visual Models From Natural Language Supervision (CLIP)
Alec Radford · OpenAIAuthored the foundational ICML 2021 paper introducing CLIP, which trained dual vision and text encoders via symmetric cross-entropy contrastive loss on 400M image-text pairs, unlocking robust zero-shot image classification and powering Stable Diffusion.
Fine-grained spatial reasoning, counting, and typographic reading struggle without explicit spatial bounding box supervision.
Context: Vision Transformers (ViT-B/16, ViT-L/14) and ResNet baselines, 400M image-text dataset.
View missionLLaMA: Open and Efficient Foundation Language Models
Guillaume Lample · Mistral AICore co-author of the original Meta LLaMA 7B-65B release, proving that smaller models trained on significantly more tokens than Chinchilla recommendations surpass larger architectures and enable democratized research.
Base pre-trained checkpoints require safety and instruction fine-tuning to prevent conversational misalignment.
Context: SwiGLU activations, RMSNorm, Rotary Embeddings, 1.4T tokens.
View missionLoRA: Low-Rank Adaptation of Large Language Models
Edward Hu · Microsoft ResearchAuthored the foundational ICLR 2022 paper introducing LoRA, which freezes pre-trained model weights and injects trainable rank decomposition matrices into Transformer attention layers, matching full fine-tuning quality with 3x less GPU VRAM.
Adapting non-attention feed-forward layers yields diminishing returns unless higher rank dimensions are budgeted.
Context: GPT-3 175B, RoBERTa, DeBERTa, rank 1-8 adapters with zero additional inference latency after matrix folding.
View missionLoRA: Official Parameter-Efficient Open-Source PyTorch Implementation
Edward Hu · Microsoft ResearchCreated and open-sourced the original PyTorch LoRA package providing drop-in `nn.Linear`, `nn.Embedding`, and `nn.Conv2d` layers with automated weight merging, forming the foundation of modern Hugging Face PEFT.
Dynamic multi-tenant serving requires runtime adapter switching kernels (e.g., S-LoRA/Punica) to avoid memory fragmentation.
Context: PyTorch 1.8+, Hugging Face integration, scalable rank scaling factor alpha.
View missionMaking Deep Learning Go Brrrr: Performance Engineering and GPU Kernel Optimization
Horace He · Thinking Systems / former PyTorch CoreAuthored the definitive technical guide on GPU architecture, memory hierarchy, arithmetic intensity, and operator fusion, read by hundreds of thousands of AI infrastructure practitioners worldwide.
High-level theoretical roofline models do not account for tail-latency jitter from asynchronous PCIe bus transfers.
Context: NVIDIA Ampere/Hopper architectures, SRAM bandwidth, Tensor Cores, roofline analysis.
View missionMCUNet: Tiny Deep Learning on IoT Devices with TinyEngine
Dr. Ji Lin · NVIDIA / MITEngineered TinyEngine, a specialized neural network runtime that co-designs model architecture (TinyNAS) and memory scheduling to run deep learning vision and speech models on microcontrollers with only 256KB SRAM.
Static memory layout requires exact pre-compilation; dynamic tensor shapes are not supported on microcontrollers.
Context: ARM Cortex-M7 microcontrollers, integer arithmetic, bare-metal C runtime.
View missionMedical AI Governance and Patient Privacy Compliance Protocols
Dr. Harvey Castro · Healthcare AI AdvisoryCreated compliance frameworks for hospital CIOs to evaluate de-identification pipelines, zero data retention BAA contracts, and clinical liability safeguards.
EHR vendor legacy APIs (Epic, Cerner) impose strict integration and certification barriers.
Context: HIPAA compliance, BAA agreements, de-identification algorithms, EHR integration.
View missionmistral-src: Reference Implementation and MoE Inference Engine
Timothée Lacroix · Mistral AIBuilt the official open-source reference codebase for Mistral 7B and Mixtral 8x7B, implementing sliding window attention (SWA), dynamic chunking, and Megatron-style tensor parallel execution for multi-GPU serving.
Sliding window attention truncates attention cache for earlier tokens outside the local window unless global tokens are designated.
Context: Sliding window attention (W=4096), grouped-query attention (GQA), PyTorch.
View missionMixtral of Experts: Sparse Mixture of Experts with Open Weights
Guillaume Lample · Mistral AICo-authored the paper introducing Mixtral 8x7B, a decoder-only model where each layer is composed of 8 feedforward blocks with a router network that selects 2 experts per token, outperforming Llama 2 70B while maintaining the inference speed of a 12B model.
MoE models require substantial VRAM for weight storage despite sparse compute activations during generation.
Context: Sparse MoE, 47B total parameters, 13B active per token, 32k context with RoPE.
View missionMLX: An Efficient Machine Learning Framework for Apple Silicon
Dr. Awni Hannun · Apple Machine Learning ResearchConceived, engineered, and open-sourced MLX, a NumPy-like array framework with composable function transformations (grad, vmap) that exploits Apple Silicon's unified memory architecture for zero-copy CPU/GPU execution.
Exclusively optimized for Apple Silicon hardware; non-macOS platforms are not natively accelerated.
Context: Apple Silicon Metal, unified memory up to 192GB, C++ core with Python bindings.
View missionMPT-7B and MPT-30B: Commercial-Grade Open-Source Foundation Models
Dr. Jonathan Frankle · Databricks Mosaic AI / HarvardLed the engineering and open-weights release of MPT models trained with FlashAttention, ALiBi positional embeddings, and fast streaming data loaders, providing reproducible blueprints for sub-$200k foundation model pre-training.
ALiBi positional embeddings exhibit slight performance degradation compared to RoPE when scaled past 128k context windows.
Context: MosaicML Composer, ALiBi 65k context length, 1T token pre-training.
View missionMulti-Agent Automation Frameworks and Swarm Blueprints
David Ondrej · Prompt Engineering / Ondrej AIMaintains widely used open-source developer blueprints for building voice-driven desktop agents, automated research teams, and local rag pipelines.
Desktop UI automation relies on OS-level accessibility permissions and screen coordinates.
Context: Python, Streamlit, ChromaDB, Whisper API, Ollama.
View mission