Arthur Mensch
AI Founder
Sources checkedCo-founder and CEO of Mistral AI. Previously a research scientist at DeepMind, working on Retro, Chinchilla, and Flametree architectures. Led Mistral to develop open-weights models that match and exceed proprietary frontier models with efficient grouped-query attention and sparse mixture-of-experts architectures.
Areas of focus
Professional niches
Proof of Work
Mistral 7B: Grouped-Query and Sliding Window Attention Foundation Model
Designed and published Mistral 7B, an open-weights autoregressive transformer utilizing Grouped-query Attention (GQA) and Sliding Window Attention (SWA) to achieve faster inference and handle 8k context sequences while beating Llama 2 13B across all standard benchmarks.
Requires high cache memory bandwidth on commodity GPUs; sliding window attention drops exact attention history beyond context horizon without caching adaptations.
Context: 7.3B parameters, 32 layers, hidden dimension 4096, 32 attention heads, 8 key-value heads (GQA), sliding window attention of 4096 tokens.
View missionMixtral of Experts: Sparse Mixture-of-Experts Open Weight Model
Published the architectural blueprint for Mixtral 8x7B, a sparse mixture-of-experts model where a router chooses 2 out of 8 feedforward experts per token, activating only 13B parameters out of 47B during forward pass and outperforming Llama 2 70B with 6x faster throughput.
High RAM/VRAM memory footprint for hosting all 47B parameters despite activating only 13B during computation; requires specialized multi-GPU tensor or pipeline parallelism.
Context: 46.7B total parameters, 12.9B active parameters per token, 32k token context window, Top-2 softmax gating router.
View mission