Dr. John Schulman
Anthropic / Thinking Machines
Co-founder of OpenAI and now research scientist at Anthropic. Invented Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Led OpenAI's reinforcement learning from human feedback (RLHF) and post-training alignment teams that built InstructGPT and ChatGPT.
Areas of focus
Professional niches
Proof of Work
Training Language Models to Follow Instructions with Human Feedback
Led the landmark InstructGPT research demonstrating that applying RLHF with fine-grained human preference data to a 1.3B parameter model produces superior instruction compliance compared to a 175B raw model.
Reward models can over-index on output verbosity and confident tone rather than factual accuracy.
Context: Supervised fine-tuning (SFT), Bradley-Terry reward modeling, PPO reinforcement learning over GPT-3.
View missionProximal Policy Optimization Algorithms (PPO)
Invented PPO, a clipped objective reinforcement learning algorithm that maintains policy stability during stochastic gradient descent, which became the global algorithmic standard for aligning LLMs via RLHF.
Hyperparameter sensitivity to KL penalty coefficients and reward model reward hacking.
Context: Clipped surrogate objective, generalized advantage estimation (GAE), actor-critic networks.
View mission