Dr. Matei Zaharia
Chief Technologist
Sources checkedCo-founder and Chief Technologist at Databricks and Associate Professor of Computer Science at UC Berkeley. Original creator of Apache Spark, MLflow, and Delta Lake. Co-creator of DSPy (Declarative Self-improving Python modules for foundation models). ACM Doctoral Dissertation Award recipient and leading expert in scalable machine learning infrastructure.
Areas of focus
Professional niches
Proof of Work
DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines
Co-created DSPy, a programming framework that abstracts brittle prompt engineering into typed declarative modules (Predict, ChainOfThought, ProgramOfThought) and uses optimizers (BootstrapFewShot, MIPRO) to compile optimal few-shot prompts and weight updates automatically.
Compiling complex multi-step pipelines requires running multiple optimization iterations, incurring upfront LLM invocation tokens during the optimization phase.
Context: Python 3.10+, automated LLM teleprompter evaluation harnesses, supports local vLLM and remote API providers.
View missionMLflow: A Machine Learning Lifecycle and Foundation Model Evaluation Platform
Created MLflow, an open-source platform managing the end-to-end machine learning lifecycle, including experiment tracking, model registry, artifact packaging, and evaluation metrics for classical ML and modern LLM pipelines.
Self-hosted deployments require configuring and maintaining persistent artifact object storage (S3/GCS) and relational backend metadata databases.
Context: Python, REST, Docker packaging, native support for PyTorch, Hugging Face, LangChain, and OpenAI SDKs.
View mission