This episode with Lukasz Kaiser, co-author of the seminal "Attention Is All You Need" transformer paper and former researcher at both Google Brain and OpenAI, is a wide-ranging conversation about the fundamental limits of current AI architectures and whether transformers will continue to dominate or eventually give way to something new. Lukasz brings a rare dual perspective: deep belief in how far the current paradigm has taken us (he's an enthusiastic daily Codex user who's seen 10x productivity gains in his own research), while maintaining genuine intellectual humility about whether transformers can truly generalize the way humans do. The episode weaves together questions about data efficiency, the non-verifiable RL frontier, the coding agent revolution, the open vs. closed source gap, and what the next architectural leap might look like: all filtered through the lens of someone who helped build the foundation the entire field is standing on.
This episode with Lukasz Kaiser, co-author of the seminal "Attention Is All You Need" transformer paper and former researcher at both Google Brain and OpenAI, is a wide-ranging conversation about the fundamental limits of current AI architectures and whether transformers will continue to dominate or eventually give way to something new. Lukasz brings a rare dual perspective: deep belief in how far the current paradigm has taken us (he's an enthusiastic daily Codex user who's seen 10x productivity gains in his own research), while maintaining genuine intellectual humility about whether transformers can truly generalize the way humans do. The episode weaves together questions about data efficiency, the non-verifiable RL frontier, the coding agent revolution, the open vs. closed source gap, and what the next architectural leap might look like: all filtered through the lens of someone who helped build the foundation the entire field is standing on.
(0:00) Intro
(1:12) Transformers vs. Human Learning
(8:37) How Do We Get Physical World Generalization?
(10:52) What Comes After Transformers
(13:59) How Much Have Agents Improved Lukasz's AI Research Productivity?
(17:21) How Close Is an AI Research Intern?
(26:06) RL Beyond Verifiable Tasks
(35:38) App Companies: Build Models or Lean on Labs?
(46:21) Multimodal Is Still Missing Something
(49:46) OpenAI's Bet on Reasoning
(55:26) The AI Coding Wars
(59:26) Focus vs. Keeping Embers Burning
(1:02:09) Open Source vs. Closed Source Gap
(1:05:15) Quickfire