Free with a sign-in.
The best 20 min of 70
Noam Shazeer and Jack Rae: Scaling Test-time Compute, Reactions to Ilya & AGI
Noam Shazeer and Jack Rae argue that test-time compute can raise model capability, but reaching AGI also requires agents that act in complex environments.
Unsupervised Learning · Jacob Effron
- Noam Shazeer · OpenAI; co-author of the Transformer
- Jack Rae · Research Director, DeepMind
The best minutes
- 19:34Inference is cheap enough to justify much more thinking6 min
- 25:29AI math progress depends on asking better questions6 min
- 31:48Research teams need both bottom-up and top-down bets8 min
Also worth watching
- 9:39Qualitative tasks need broad reward criteria2 min
- 15:57Agent research depends on choosing the right environment4 min
- 42:03Use one model unless specialization earns its cost2 min
- 1:06:30Inference-heavy AI can use more distributed compute3 min
- 58:15Hallucinations made entertainment a natural early use2 min
Skip the extended discussion of AI in family life, human meaning, and social risks.
Full analysis · 6 min read
Real Alpha · Nov 8
Noam Shazeer says inference gets like over a million tokens per dollar
- Shazeer says efficient inference could yield like over a million tokens per dollar.
- That supports more thinking, but Rae says test-time compute alone will not take models to AGI.
- Teams should judge it by verified work, latency, and failure costs, not token prices alone.
Free with an account
Read the rest of the analysis
7 sections, the 4 other voices it tests Shazeer against, and what it means for you. Every episode we cut gets one.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Signed up before? The same buttons sign you in. The episode, its moments and the transcript stay free.
Inference prices leave room for more thinking
If inference runs relatively efficiently, Shazeer says, a very large model can deliver like over a million tokens per dollar. He compares that with like 10,000 tokens per dollar for reading a paperback book.
He says inference is probably four orders of magnitude cheaper than paying anyone for anything. He puts it at whatever six to eight orders of magnitude cheaper than a software engineer.
Those figures describe tokens per dollar, not the cost of a correct answer or finished job. The episode gives no measured quality gain or workload price for that extra inference. Nor does it account for review time, latency, or failed attempts.
Both guests work on Google's Gemini models. Google sells those models, so this is a builder's cost case, not independent accounting. Shazeer makes the value case conditional on finding useful ways to spend those extra flops.
Our read
The gap is a real economic opening, but not an ROI result. Budget around verified work, not tokens alone.
Consumers may tolerate added latency, but evidence is thin
Rae says users seemed willing to accept a bit more latency when using the Gemini app. He says a couple of seconds felt like a small price for a better answer.
Separately, he described his mother reading the model's thoughts after asking an open-ended philosophical question. She sat with both the answer and its thoughts, an anecdote about engagement rather than latency.
But the episode gives no conversion, retention, task completion, or productivity measure for that slower response. So the story cannot establish a work buyer's willingness to wait or pay.
For example, a product team can test a hard task at several delays and compare completion quality. Measure whether the slower answer reduces correction time, rather than treating preference as productivity.
Our read
The anecdote suggests curiosity, not willingness to tolerate delay. Teams need to test whether extra seconds buy useful work.
Broad reward criteria make evaluation design central
Rae says models are getting better at following more abstract instructions. He describes training them to take in broad criteria, including style or interest, and apply a reward signal.
Once that signal exists, he says, reinforcement learning can train against it. That is a route beyond tasks with automatically checkable answers, but does not make taste easy to measure.
Shazeer says difficult evaluations help distinguish reasoning from simply adding parameters and memorizing more. Yet the host says math benchmarks can saturate quickly, even after they seemed difficult.
Shazeer also warns that public tests lose value when their problems circulate. Private evaluations help, but no benchmark alone measures usefulness at a buyer's desk.
Jad Naous, on a16z Podcast in October 2026, says agents can improve query code. He ties that work to user workloads and performance benchmarks. His episode also says production experiments still answer questions such as memory needs.
Our read
Broad rewards can widen the tasks models learn from, but evaluation must stay connected to work. A score is only useful if it tracks an outcome that matters.
Agent environments shape what models can learn
Rae says selecting a useful environment is as important as a breakthrough in attention, long context, or reinforcement learning. He says no one has found the perfect setting.
For example, he points to a browser where models must scan screens and act across many websites. He also names codebases as settings where agents could do useful work.
At Google, he says a structured monorepo and build rules make AI coding easier to orchestrate. That evidence is specific to a well-structured codebase. It does not establish comparable reliability on every company's software.
Danila Shtan, on Unsupervised Learning in November 2026, says reliable outcomes will drive agent adoption. His episode separately describes internal work on tools and task-specific harnesses.
Richard Socher, on The TWIML AI Podcast in November 2026, says open environments can help ideas combine across fields. That supports open-ended settings as an exploration tool, not a guarantee of reliable action.
- DSDanila Shtan · Unsupervised Learning · Nov 2Agent reliability may matter more than model size
Richard Socher · The TWIML AI Podcast · Nov 3Scientific ideas grow by combining work across different fields
Our read
The environment is part of the research problem, not just a backdrop. Yet choosing one does not establish that an agent will work reliably elsewhere.
Useful math depends on asking better questions
Rae wants math benchmarks to move beyond exams toward useful discoveries and important problems. He favors a ramp from hard problems to findings that may be unpublished, while saying no single evaluation is perfect.
He says he feels confident about solving, but identifies asking worthwhile new questions as the harder step. Take the FrontierMath idea. It tries to bridge today's harder tests toward useful scientific contributions.
Shazeer uses mathematics to challenge the idea that models can only repeat existing material. He notes that mathematics can begin with little external data, though that observation is not proof of machine discovery.
Sam Altman, on Cheeky Pint in October 2026, describes agent teams exploring math at high compute, including Navier Stokes. That is a research effort, not evidence of a validated discovery or better evaluation.
Our read
The target is not only solving harder prompts. It is choosing consequential questions, then validating the result with people who know the field.
Rae says test-time compute will not reach AGI alone
Rae says test-time compute will not by itself take models to AGI. He points to acting in complex environments as another important component.
His Ramanujan example is more ambitious than longer answers. He imagines a model thinking deeply, incorporating useful knowledge, and learning from one math textbook until it becomes a strong mathematician.
That remains an aspiration, not a result described here. Rae says reinforcement learning that trains deeper thought is already helping models learn more from the same data.
But that is different from proving every inference-time token improves an answer. The transcript offers no measured transfer rate between unrelated tasks. So the case for extra inference is strongest where thought improves data efficiency or action, not merely answer length.
“I don't think it's all the way to AGI.”
Our read
Inference is a scalable input, not a complete theory of capability. The case is strongest where deeper thought improves learning or action.
Research needs bottom-up exploration and top-down bets
Shazeer says research benefits from both bottom-up exploration and top-down coordination. He saw Google Brain as mostly bottom-up and DeepMind as mostly top-down.
Top-down, he says, can coordinate people around large training runs. Bottom-up can give projects more resources when collaborators join, while making room for ideas that fit neither familiar category.
Rae describes a similar mix in Gemini thinking research, with open exploration alongside mandates teams must deliver. He says leaders can be surprised when bottom-up work proves more impactful than expected.
For example, Shazeer says a rule built only around pretraining and post-training can miss new categories. The episode gives no ratio for these funding choices.
For leaders, the implication is a portfolio, not a formula. Keep room for exploration while coordinating expensive bets. Yet the speakers do not prescribe how much compute belongs in either bucket.
Our read
The episode supports balancing exploratory work with coordinated commitments, not a fixed allocation formula. That balance also leaves room for ideas benchmarks and plans miss.
What it means for you
- Price a workflow. Compare inference spend with quality gains and human review for a hard task. Track cost per verified completion rather than token price alone.
- Test latency. Run the same work task at several response delays. Measure quality, correction time, and whether users return for the slower answer.
- Define reward criteria. Choose a qualitative task and write observable criteria before tuning. Check whether independent reviewers apply those criteria consistently.
- Select an agent setting. Start with a browser or codebase task that has repeatable states and outcomes. Track successful actions and recovery from failures before adding more reasoning.
What to watch
- Whether extra Gemini Flash Thinking latency improves completed work, not just preferred answers.
- Whether qualitative reward signals predict outcomes outside math and code.
- Whether agents act reliably in useful browser and code environments.
Send this to product and infrastructure leaders budgeting inference for agent workflows.
Watch the minute where Shazeer gives the token-cost comparison.
Written by Real Alpha from the episode's transcript and the other moments it links, then checked against the transcript. Quotes are the speaker's exact words.
Minute by minute
- 2:03Thinking can improve creative work
Rae says Gemini’s first test-time-compute effort focused on math and code, but thinking also improved creative writing by helping models explore and revise ideas. Shazeer says difficult benchmarks help separate reasoning gains from performance boosts due to more parameters or memorization, while Rae notes that evaluations can quickly saturate or leak.
- 6:44AI can accelerate AI development
Shazeer calls using AI to make AI researchers more productive the most important reinforcement loop, alongside user-feedback data flywheels. Rae says Google’s structured codebase and tools already let AI surface fixes and reviews, and he expects agents to handle more open-ended coding work.
- 9:39Open-ended work needs better rewards
Shazeer says less-verifiable fields need better ways to verify results or more human feedback. Rae says models can learn broad qualitative criteria, such as style or whether work is interesting, and be trained against those rewards with reinforcement learning.
- 11:51Thinking and vision broaden use
Rae says the Gemini app adds thinking to tools including Maps and Search, and users may tolerate a few seconds of latency for better answers. He recounts his mother reading the model’s thoughts after asking about the meaning of life. On multimodal work, Rae points to visual reasoning and Project Mariner’s browser actions, while Shazeer says image generation has fewer training examples than text.
- 15:57Reliable agents need useful environments
Shazeer says making models smarter helps, but agents need general solutions because users will find uses developers cannot anticipate. Rae says research also needs well-designed environments, such as a web interface or codebase, and that engineering agentic training is a substantial part of the challenge.
- 19:34Inference leaves room for more compute
Shazeer says efficient inference can deliver over a million tokens per dollar, compared with about 10,000 tokens per dollar for reading a paperback, leaving room to spend more compute when it improves results. Rae says test-time compute alone will not reach AGI; models also need to act in complex environments.
- 23:45Deep reasoning could support discovery
Rae says he wants models to think deeply enough to derive useful knowledge, not simply spend longer on each problem; reinforcement learning with test-time compute is already helping models learn more from available data. He sees a path from math evaluations toward useful scientific work, but says asking novel questions remains difficult. Shazeer argues that connecting known ideas can accelerate science and that math can grow from very little data.
- 31:48Research needs experiments and mixed bets
Shazeer compares AI research to experimental alchemy: researchers form hypotheses, test them and often struggle to assign credit. On resource allocation, he describes Google Brain as more bottom-up and DeepMind as more top-down. Rae says both approaches matter: concentrated bets need vision, but bottom-up work can reveal unexpected breakthroughs.
- 39:33General models can span domains
Both say choosing large language models was a good bet and discuss the move toward vision and native image generation. Shazeer argues that high-value areas such as healthcare do not need separate task-specific models unless cost justifies them. Rae says specialization makes sense if negative transfer appears, while positive transfer favors one general model.
- 44:03Open models are closing the gap
Rae says the speed at which labs adopted test-time compute surprised him: models explored the idea and appeared within months. Both say open-source models are staying close to frontier systems; Rae praises Gemma 3, while Shazeer says the time gap may be small even if a quality gap remains.
- 49:04AI could reshape education and purpose
Rae says his son, under supervision, uses Gemini to identify plants and lizards from photos, and he expects AI to make personalized education more powerful. Shazeer says human labor may not be materially necessary in the future, so people should do meaningful work now and consider where they will find purpose.
- 55:39Safety risks need active work
Rae says he is moderately concerned that more-intelligent systems may not remain predictable or useful, and he flags employment shifts and unintended consequences. Shazeer says he is not afraid but agrees that teams need to work on safety.
- 58:05AI companions may serve users differently
Shazeer says he left Google to start Character because he wanted an application that let people interact with LLMs and discover useful applications. He says entertainment worked partly because hallucinations could be a feature, and he expects people to use AI for both human-like interaction and practical help while still valuing human relationships.
- 1:03:00Evaluation and infrastructure still matter
Rae calls the ARC-AGI evaluation overhyped because progress on specific puzzles may not advance useful AGI, while Shazeer says AGI and coding are underhyped. Rae sees value in agentic applications beyond coding, while Shazeer would focus on engineering agents that help develop AI. Rae says inference-heavy systems could distribute actors across data centers; Shazeer adds that transformer inference can become memory-bound, leaving work for model architecture and hardware.
Published Mar 17, 2025.
Transcript · 53 min read
Unsupervised Learning, hosted by Jacob Effron, with Noam Shazeer and Jack Rae.
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
Noam Shazeer and Jack Ray really need no introduction. The two are at the forefront of Google's Gemini LLM efforts and have been involved in some of the most important discoveries in AI in the last decade. Noam as one of the co-inventors of the Transformer and mixture of experts. Jack, a key part of many DeepMind breakthroughs. It's a real privilege of the job to get to sit with these two and ask them literally every top-of-mind question in AI today. We talked about how far test-time compute will get us and the spaces where it will and won't work. We talked about how the infrastructure needs will be different for test-time compute versus the large kind of pre-training paradigm.
We also hit on the impressive pace at which open source models have caught up with closed source peers and their reactions to DeepSeq. We talked about their kind of reception and reaction to Ilya saying that this test-time compute paradigm won't get us all the way to AGI, as well as Jan LeCun saying this current generation of models can't actually have any novel thoughts. We talked about what it actually looks like to do cutting-edge AI research today and what their day-to-days look like, as well as the future model milestones that actually matter to them. And then we also got Gnome's reflections on character as well as both of their responses to what, you know, AGI means for the role of humanity.
I think folks are going to love this. It was really just a pleasure to get to speak with both Noam and Jack. Without further ado, here they are. Well, Noam and Jack, thanks so much for coming on the podcast. Oh, thank you. Are we sitting in the very office that the Transformer was invented? No, this is a new building. We were in 1965 Charleston, I think. Probably about half a mile. Half a mile. All right, so it's in the air. Pretty close, pretty close. Well, many things to dive into today. I mean, obviously want to start with some of the latest Gemini 2.0 models and obviously all the work you've been doing around test time compute and Gemini 2.0 flash thinking.
Read the whole transcript
I guess just at the highest level for our listeners, how do you characterize where these models work today, where they don't work as well? And as you were kind of experimenting with them, what surprised you most about those results? One surprising thing is kind of when we started the particular concerted effort to build a a lot of kind of research into test time compute, into Gemini, and then think about shipping it, is that we were really kind of focused on starting out with reasoning tasks. So math and code were like big areas of focus. And it wasn't really clear, you know, whilst we're kind of sprinting in that kind of domain, we obviously want to broaden it naturally over time, but it wasn't really clear like how that would work.
would there be any kind of sense of generalization? Would thinking be useful beyond those reasoning tasks if we're just concentrating on those as researchers? And I think it was pretty fun to see one of the early kind of models that we felt like had been trained to try and match the style of Gemini Flash. So it had been trained with thinking, but then it also undergone some kind of training to actually be just like generally a nice style, a nice like model to talk with. that it was actually very fun seeing thinking, interact, and improve creative tasks as well. You could ask the document to compose an essay on a particular topic.
And actually, A, the thought content was very fun to read. And it would go through various different ideas. And then it would go through revisions of the idea or things that it should cut. And that was kind of fun. And then also the output felt really nice. So that was one thing that kind of surprised me. Any surprise for you, Noam? Well, yeah, I mean, in general, like, I'm all for, like, generality, like, let's train something that's great at everything. It is important, you know, and I was skeptical at first of, like, okay, this intense focus on things like math. But it is very important to have good benchmarks that are going to encourage you to be able to reason about difficult tasks, because a lot of things will drop perplexity, like add more parameters to the model and memorize more.
So it's nice to have the evals that can distinguish. better, you know, some of the, you know, more difficult problems. I mean, what evals are even meaningful to you at this point? I mean, obviously, I feel like people are trying to hill climb the same set of evals that feel increasingly less relevant to day-to-day work. What do you guys, like, do when you're testing these models, or, you know, how do you vibe check them? It feels like we keep landing on an eval, and we're like, oh, actually, we overlook this eval, even if it's in math. It's like, okay, we've done a bunch of, like, math evals, but maybe, like, I don't know, Putnam answers only, AIME, they're still considered challenging.
And then it's like, done, okay. They're completely saturated and we really don't care and they're small. And we almost think, why did we ever even work on them? And it's easy to forget. That was so easy, but we thought six months ago. Yeah, a couple of months ago, it was considered really hard, maybe too hard for the model. And then it only snaps to being trivial. So right now, I do feel like there's always been a lot of concerted effort within DeepMind and within like Google as a whole to develop useful evals. But it's kind of very nice to see also this is like a shared responsibility across like many different AI labs and even like scale has like started really stepping up and developing like very challenging evals.
I think Chalent Humanity's last exam was a dangerous, you know, if every six months we think it wasn't too hard, it's a dangerous title. The really, really last exam. Yeah, exactly. But the whole thing is... V73 last exam. It's very, very challenging in general because you know evals get leaked you know like once people start talking about the evals then there's all this text out there about the evals and um and you know they're they're no good at anymore because everyone knows the problems and you know all the models will uh know the problems unless you're very very careful so there's you know i think there's still a lot of work that It goes on into having evals that are private.
Are there a set of milestones that are meaningful to you? It's like, hey, when Gemini 3.0 can do X, that's a really exciting milestone, whether it's an eval or just something that you've tried with these models and they can't quite do yet. I'd say when Gemini 3.0 writes Gemini 4.0. Or I should say Gemini X writes Gemini X plus one. Yeah, I think these reinforcement loops are are probably the most important thing to pay attention to and you know there are there are several reinforcement loops going on you know what the one I just mentioned is probably the most important one that we can actually use the AI we're building as as a tool to make ourselves more productive at at building AI but then you know that then there are other reinforcement loops around, you know, data flywheels, like, you know, you get, you have people use these models and provide feedback and make them better at the things that people care about. I think we'll see a huge acceleration from that. And then there's just the global excitement and funding flywheel, which is, I think, also seems to be kicking up in the last few years yes certainly so um you know to your point i guess of having you kind of uh armed with like a thousand ai engineers by your side uh making you even more productive like where are we
in you know do you have the equivalent of a 0.1 of an fte today of using some of these uh gemini models alongside your research yeah i think one benefit we have at google is like uh we work in a very like structured mono repo so uh and we have a lot of like amazing tooling around like contributing to the code base already so there's like lots of angles where ai is kind of uh like being pulled in as tooling for our own development like uh i i think jeff quoted the statistical i don't know but like the number of what we could call like pull requests that like 25 right yeah like useful like a just like ai like bug fixes or uh code reviews attached that's already like that just It gets pulled in one day. I notice it there. I'm like, oh, that's cool. I can now do a lot of apply fixes that it already spots.
But that's just one element where we're already pulling in AI towards our own coding development. I think we're incredibly excited about agentic coding is just definitely very important. and trying to get the model to be able to tackle more open-ended and difficult tasks is definitely something we're very excited about. And I think it's just like, in some ways, it's a lot easier to orchestrate when we have a very defined way of defining libraries. We have these build rules and things, and it's just like everything gels together in terms of the whole code base very well. So I can imagine it's going to be, as progress continues, use, it's going to be a very discreet moment where suddenly lots of libraries can be very quickly iterated on within our code base.
Yeah. You've got your AI engineers proposing experiments to you for what's good to try.
“take in a very broad set of criteria and then apply a reward signal once we have the reward signal we can train with reinforcement learning against it”
of your favorite kind of like ways you've seen people using them um and what would you like to see more people like trying and building with these things actually there's just like an update today happening in the gemini app where we're putting in a much stronger model and it's being integrated with basically the full suite of like Gemini app tools so it's like all of the apps that Gemini app supports uh from things like you know maps uh also like search integration and now has
like very long context all these things it's kind of like a fully featured Gemini thinking release in the app and it is actually you know it's a very enjoyable experience that's what I think so far people even for their day-to-day stuff people uh it wasn't clear to me whether they would like to have to pay a bit more latency to like get the model to think about stuff before responding, it does seem to be the case. But if people are going to like pull their phone out of their pocket and then type something in, that actually what we thought was a very long time, maybe a couple of seconds, is like a very small price to pay for something that they feel like is a better quality answer.
And also maybe they can like sometimes look at the thoughts and inspect them. Like I was kind of showing this to my mom like a couple of days ago. Like moms are the ultimate test of whether like something has broken the barrier from like the Twitter sphere to like the real world. Yeah, like the mum vibe check is a big deal. And yeah, I guess she asked a lot of what I would consider very generic questions to ask a model, like what is the meaning of life? Oh, meaning of life. She went for what is the meaning of life and she really like sat and read it for a long time. And then she read the thoughts as well and then she kind of like contemplated on it and she felt like, I don't know, she seemed to kind of very much appreciate the presence of like thoughts as to like how to even go around such like an open-ended question.
So more folks building philosophical conversations with these models. Well, I mean, one thing I think is super impressive about them is the multimodal capabilities, but it still feels like those are very underexplored from an application perspective. Not sure totally why that is, but... Yeah, I think right now, in some ways, we are quite modest about the multimodal capabilities of Gemini. I feel like the model has always been incredibly strong at image input. Image input plus thinking is actually remarkably good, I would say. I see a lot of people kind of red teaming the model on things like OnX and trying difficult or challenging images, visual reasoning problems.
I think that is working pretty well. There is also, at some point, some of those are kind of toy evals, but pulling in multimodal with then agentic tasks is super interesting as well. So, like, we launched Mariner last December, which was, like, an agent which, like, uses a browser and things. That has a lot of multimodal, like, aspects all built in. It was super important that they could get the model to be incredibly strong at, like, not only kind of scanning a screen, but, like, really understanding it and knowing how to act on many, many different types of websites. So, like, pairing that kind of capability, agentic, quite open-ended, maybe you really need visual understanding of potentially messy scenes with thinking is something that I think I'm feeling very excited about.
Yeah, that's one. I mean, it made sense to start with text. You know, a picture's worth a thousand words, but it's a million pixels, so the text is still a thousand times more information-dense just to do math on cliches. But and then there's also like a lot more just in the main training data for text. You know, we have so many examples of text, which actually kind of represents, you know, the way humans receive and produce information, you know, through language. But, you know, we have a lot fewer examples where for something like, say, image generation, because you don't have examples of people generating images.
So things are a little more challenging, but we, you know, it's, as Jack said, you know, great stuff, great stuff happening. Yeah. When you mentioned Project Mariner, I'm curious, like, you know,
“I don't think anyone has solved the perfect environment yet”
“If you think of like a really cheap pastime, like go like buy a paperback book and read it or something, you're paying like 10,000 tokens per dollar.”
particular, you know, spitting out a couple of thousands of tokens in order to solve the task, but maybe like think much more deeply, much more kind of like a researcher might think about a hard
and open-ended problem. I think so much of what you guys have touched on is like these models increasingly acting like researchers. What are like the early signs of that that you would look
“Whether it's possible, I'm not sure, but I would agree maybe the key crux is can I ask actually novel questions?”
“there are so many abstraction-breaking ideas that there's no great way to categorize them”
Yeah, I mean, reflecting back on the past decade, are there some of these inflection points or these decision points where it's maybe hard decisions, 51, 49, that ended up being kind of super impactful? I mean, I think for both of us, going in on large language models was good. Yeah, I'd say that was probably a good call. and it kind of seems obvious now but it kind of was at least I mean I found it was like in a state of being not obvious to most people but obvious I felt like to a small number of collaborators that like this was definitely gonna be a big thing and kind of had to go against the grain for a while. Yeah that's true there was definitely a time people were not excited about language models. Hard to remember now it feels like decades ago. I mean it always seemed like the best problem on earth to me but you know uh that's yeah it's kind of like the deep learning people at the time maybe thought machine translation was still a bit cooler or computer vision yeah vision was exciting for a while i don't know why was everyone into vision No, I guess, like, the ImageNet thing.
Yeah, or there's, like, a picture of a cat or something like that. Oh, that was, like, good one, the other cat. Did you work on the cat thing? No, no, no, never actually did much in Vision. I've run M&S. And now you've come full circle on these Gemini 2.0 models. I see all these demos that are, like, Vision-based, and you're kind of doing cooler things in Vision now than certainly identifying cats. I mean, it felt like, I think, even almost every early LLM person, world model is on their mind, really. I actually don't feel like the early LLM researchers were even really from a language-oriented background.
They weren't really like linguists. It was more, I don't even feel like understanding language was really part of the motivation for that early group. It was like train unsupervised learning at scale, do language first because it's the most like knowledge compressed, but then gobble everything up into like a big generative model and like understand everything. And it's very, very cool to see that just, like, continually proving out. Yesterday, we just, like, launched native image generation. It's amazing. You know, I think a lot of image generation right now is just, like, focused purely on getting, like, absolutely maximally aesthetic images.
But also having native image generation allows you to, like, really do a lot more with images, understanding, editing. You can have, like, interleave sequence of images and text. And, yeah, once again, it's just train a generative model on lots of data to arrive at that.
“If there's positive transfer, put it in the same model, as long as it doesn't then become too expensive to serve.”
What's one thing you've like changed your mind on in the past year? I feel like timelines have shifted forward. And I don't mean that like in a vague sense. Like I think the rate of progress is much faster right now. I felt like a year ago, there was obviously the field is advancing. But whenever you have a new paradigm shift, it creates this sudden acceleration. And also, actually, no, this is a pretty. Good one. One thing I've changed my mind on, like, my mental model of, like, the propagation of information and how, like, how people, like, adopt a scientific advance has completely changed. So when the Transformer came out, I think, so I was over in London at DeepMind. People thought it was a cool paper, were a bit suspicious. I eventually implemented it in our, like, code base over the, like, holiday break, actually. But it was, like, three months after the paper had come out.
I eventually implemented it and tried it for like language modeling but it was like not really getting picked up I eventually then collaborated with someone that wanted to use it for reinforcement learning but I'm going to say really it was like from paper coming out maybe it was like six to nine months before it was like you saw just transformers dotted around all areas of deep mind and that's within out like we're all within alphabet and it's like much easier to propagate information um I would say the the the speed at which the field has like picked up this test time compute paradigm you have like many labs have already trained and released models that like looking very good exploring the space that was like very surprising to me the fact that these things can like if you make an announcement say and like this is important the fact and it's just like a blog post or something and then you'll have people that are able to like uh make breakthroughs in that space and release models in order of months that is like that was a wake-up call for me there's a lot more compute and a lot more smart people working in the ai i often think of things in a bit of a rose-tinted glasses way and think about like 2016 or something and i think well
we're very smart then we're very creative people are very very smart and creative now and they have way more compute and there's way more of them and so if anything is going to like be very impactful and then it can just like spread and like this idea can like spread all across uh all across the world and people act on it which is kind of crazy yeah it is it is kind of crazy and just the amount of compute out there that yeah that like now uh you know whatever a kid in the garage has more compute than was necessary to invent the transformer so like uh you know um yeah i mean people always worry about oh just how much can compute do they have but you know it is it is definitely possible to make breakthroughs, you know, with way less compute than you would imagine.
Yeah. Anything you've changed your mind on in the past year? I mean, I've been continuously impressed with the success of RL. I'd never really worked with it much before. It's like, oh, that's actually pretty good. Well, I mean, you kind of alluded to this, Jack. Like, obviously, I think in the reaction to DeepSeq and all these kind of models that kind of fast-followed in the test-time-compete space, going forward, do you expect the open-source models to be able to keep up with each subsequent generation of these models? It obviously seemed to happen faster than a lot of people would have expected.
Yeah, that's actually something I'm changing my mind on. I do feel like the open-source, like the ability for open source models to stay very close and competitive with the frontier is persisting. I actually thought we were getting maybe a false sense of assurance that it's happening because maybe it felt like it was converging, but then these things can pull away again. But actually, that seems to have been very impressive. I'm really, really impressed at the performance of Gemma 3 that just got released yesterday. It's amazing. It's completely incredible. The team did a really good job. And other open source models.
Yeah, so, you know, DeepSeat V3 was a very good model when they released it. Yeah, so, it seems like people are very, like, passionate in the open source space. And they're very creative and smart. And they have compute. So, I don't really see why. why they wouldn't be able to continually innovate. Yeah, what do you think, Naan? Yeah, I mean, it seems like the time gap between closed source and open source is, you know, has been shrinking. You know, I think that the technology will continue to accelerate. So, I mean, it could be that the quality gap will be large and the time gap will be very small.
But, yeah, we'll have to see how it plays out. It's super exciting to see all of these companies getting great results. Switching to some of the broader implications on all the AI progress we've been talking about for society. I'm curious. Obviously, it seems like both of you in the last year have been impressed with the power of RL. You've been surprised by the pace at which we've scaled a lot of this test-time compute. um have you like changed anything in your own lives like based on kind of this you know probably uh more clarity or or or belief that you know a lot of this ai driven future is coming like you both have kids i know like any anything that that you've adapted uh it sounds like no i know you don't clean your garage but i guess that was a prior to this year as well you thought about not cleaning yeah yeah i don't worry too much about global warming you know we'll have ai to take care of the carbon stuff soon enough anyway you know yeah but i guess you know any anything that you've uh that you've thought about differently in your own lives or you know uh in how you think about the life your kids will have ai and education is like uh i didn't think people are really talking about it enough yet my son like under supervision but he likes to talk to gemini it's
actually insane like how powerful it is especially if he can like he goes out to the garden he like takes pictures of plants takes pictures of lizards and you and like he now has this like very accurate personalized encyclopedia which can give him information and do they adapt to that like i wasn't sure how that would work they do my four-year-old son walks around like talking very detailed about like the plants he'll use the latin name he's like they absorb so much stuff they're sponge i feel like i'm seeing kind of like a type of education that i don't really think has ever existed for like humanity happening like ai and education is going to be incredible uh he went to school and he was like um he was like oh yeah i caught a lizard to his teacher and his teacher was like oh that's cool um and it's like oh that looks like a big lizard he's like no it's not a big lizard it's a western fence lizard very particular about his type varieties of lizards And he's like, and I also saw a blue-tailed skink.
And she's like, what's that? He's like, it's an amphibious lizard that, you know. So he starts like kind of reeling these things out. And, you know, that's just like, it's just obvious when you see it, but children are very curious. They're like sponges for information. And if you can like combine that productively with AI, I think that's going to be really incredible. I do feel like the next generation will just seem like smarter people. That's what I'm feeling hopeful about. No, anything you've changed? Yeah, it is extremely hard to predict what the future will be like. We will all do our best to make sure that AI will be safe and beneficial.
But, you know, it does, you know, it does make you think that, hey, you know, what I do now really, you know, really, really matters, you know, like, okay, we don't know if, you know, human labor will be, you know, like, materially necessary in the future. but uh that that just means a you know like uh you know it makes uh makes more difference uh well if you want if you want to make uh if you want to do something uh that matters materially go uh go do it now and then other than that like uh you know just uh try to be a good person you know whatever you uh find uh find uh spiritually uh meaningful uh you know go do it because that's that maybe that's uh uh you know the purpose of humanity uh uh in the future uh you know may not not be about providing for physical needs.
So we've got to figure out where we find meaning in the future. But we'll have plenty of time for, you know, the relationship or something else. Yeah, well, especially if your mom's already having the deep philosophical conversations with the models. Maybe we'll be able to reason our way to do that. But I am struck. I mean, I feel like some other people have come on this podcast, you know, Bob McGrew, the chief research officer of the AI was like, look, you know, the humanity is always going to have a role in asking the questions. The models will go off and do things. But I think to our conversation earlier, it's like, I think the big question is, will, you know, people always be the best people to ask questions? Or, you know, will the models actually ask better questions over time?
And obviously, it has a ton of implications. I feel like every generation thinks they're living through the most important moment in history. But it does, you know, I guess, you know, you're biased when you're going through it. But it does feel like we are certainly in that. Yeah. And I think sometimes like a technological advancement kind of scares people and it is people have right to feel trepidation at this stage as well. But there is like, I mean, this isn't such a good example, but like even like the introduction of the, like the television, people are like, oh, is this going to make us all just like lose our attention span? Are we going to like completely lose the will to like go out and walk outside and like have friends and things?
that was like a thing people freaked out about i think uh obviously now we know that was like unnecessary there was like it was a small piece of technology that was entertaining maybe even has provided like a net positive to society i think it like kind of went okay right now it's like there's like it's it's like that but kind of on steroids there's like very strong signs of how this helps us there's like very good reason to be concerned about how this could not help us so it's kind of like you you have like very demonstrable reasons that have already proven out of how it's helping us you have very like concrete arguments for how this could not go well and i think that makes it a very interesting time too in some ways you know we have less uh kind of um meaning than than in the past in the sense that like uh you know in the distance past everyone was like at the break of starvation and you're like okay i have meaning in my work because i have to go work hard today and get some money so my family doesn't starve tomorrow and like today we're living in America. Nobody's family is starving tomorrow if you don't work hard. So that, you know, okay, that's less meaning than we used to have. And we've found other, you know,
other sources of meaning. So, you know, future, more of that in the future as hopefully, you know, AI improves our physical situation. How worried are you both about AGI risks? I would say moderately, yeah. It is hard. It's often difficult to find examples of creating something which becomes far more intelligent than its creator, but then still acts in predictable and useful ways for its creator. And I think that that, like, class of argument is concerning. There's also just, like, more practical, like, kind of AI and society implications that we've kind of touched upon. But, like, making sure that AI is, like, constructive to the economy and the, like, and the, and people can kind of offload their lifestyle.
And we don't have, like, sharp changes in, like, the employment landscape and things. That's super important. Both of those are often on my mind. And then there's just very pragmatic things when we're always putting more capable models out. We're very excited as technologists to develop and ship things. But we also, I think we have a pretty good balance of like, of then internally having another group that's going to be thinking about this much more holistically. And like, how can we make this launch safe? What are some unintended consequences? Which I think has been pretty good and is like super important.
So I'm glad that happens. Yeah, I mean, we, hey, I agree. We need to, yeah, I mean, I'm not afraid, but we definitely need to be working on all the safety aspects of all of that. And there are examples of, you know, we create something that becomes smarter and more powerful in that, you know, we have kids. and they're smarter than us and or they're already detailed encyclopedias of amphibians and then they become teenagers and then you solve the alignment problem yeah yeah so I mean hopefully if we respect our parents and you know treat them well the AI will learn from us you have to have a lot of tokens on the internet of people being really respectful toward their parents exactly yeah we have to stop pushing the robots over respect for creators yeah exactly that that feels dangerous um you know one thing i did want to ask you about noam is obviously you know uh in your uh previous in between google stints you'd spent a lot of time
“nobody's figured out how to make this thing not hallucinate. So people are going to use it for applications where hallucination is actually a feature, such as entertainment.”
Wait, what were you asking? I guess, like, what do you think the future of that? Obviously, like, there's this early behavior of people interacting. Yeah, yeah, what is the future? What does it look like, like, five, ten years from now? That's a good question. I do not know. I mean, I do think, you know, people are going to, I think people will always want relationships with humans because it, you know, it's spiritually more meaningful. But, you know, I think people will, you know, like having AIs that are kind of more in human form for, you know, for, you know, for things they want.
I mean, like, imagine, you know, you just get elected president. You get your piece in the cabinet to, like, advise you, like, wherever, you know, wherever you go. I mean, that could be cool. Yeah, you get your own personal AI cabinet. that you get your whole... A good AI summarization of all the secrets that he needs now. Yeah, or maybe you're a CEO of your own AI company, so you get a lot more productivity. I guess that's maybe in a lot of those cases, it's less about the personality and more about the productivity. But I think to the degree that people like interacting with something that feels human, probably we'll see a lot of AI that feels more human in various ways.
And do you think progress in that space is just about the models getting better? Or is there a whole other set of human-computer interaction product questions that need to be solved? That's a good question. I mean, I think the models getting better is pretty big. But then, yeah, part of it is for whoever is running the application to decide, okay, what are we going to let people do? I think users are probably pretty good at, will be pretty good at specifying, okay, what do I want for an interface? And it'll be mostly about, okay, do we want to let them specify that? Totally. Well, look, both of you, you know, fascinating conversation.
We always like to end with just a quick fire to get your, you know, take on some overly broad questions that we cram into the end. Sounds good. Okay. So maybe to start, you know, what do you feel like is overhyped in the AI world today and what's underhyped? I personally feel like the Arc AGI eval is overhyped. Oh, that's very spicy. Yeah, I think actually maybe the progress has been quite slow because a lot of researchers just don't feel particularly inspired to do like kind of these very specific types of puzzles. puzzles we did a lot of that in like 2015 2016 and then we kind of felt like yeah you can if you if you know the puzzle domain you spend a lot of time like fixing all the like actual bottlenecks like maybe acting on these large grids is like a little bit finicky and stuff and and then and then you make a lot of progress but but then you don't necessarily continue on like building something that's really agi and useful so i feel like i personally had a transition from going to from like all these like synthetic tasks like uh to then going to just like just model natural language and i felt like that was a dragging in in that direction was like a much more agi thing in in the long run anything else comes to mind for you to overhyped or underhyped i don't
know i think i think agi is underhyped agi lms are just still still massively massively underhyped I think people are, you know, I don't know, people are still, like, thinking about it like it's only, like, you know, that it's going to be about some silly trillion-dollar products, you know. I heard you say in another pod that a trillion wasn't cool anymore, quadrillion was... exactly you gotta put the dr evil up um i obviously you know it'd be a gross misallocation of societal resources to take you away from building models to like build applications but i am curious like if you were to if we were to say today go build an application like what what do you think are the most interesting you've talked about education before but any other areas that come to mind that you think would be fun to to go build uh you know apps on top of these models well i i do think um i think it's actually very cool how many like apps that have been trying to break into this agentic space and people then expose it and say, oh, this is just a wrap around a known model, but there seems to be a lot of value in actually having the right app experience if you want to actually have a model do something useful for you,
act and do something useful for you. So I feel like, yeah, I feel like, okay, in the agentic space, I think code is very crowded now, but I do think there's a lot of other things that I would find it useful for a model to automate for me that goes beyond maybe like a chat experience but it's actually going to be going out and doing useful things. Yeah, I mean, I'd say code is underhyped. I think it's huge. I think it's huge because, you know, for one, humans aren't even that good at it. Things like code and math are not super designed for that. And then it's one of these things that will help self-accelerate.
You know, if it builds an automated software engineer, researcher, and then it'll build the next better AI. So, yeah, the combination of, you know, of engineering and agentic, you know, something that can control surfaces broad enough to do the job of an engineer. I think if I were to focus on applications, that is what I would be focused on.
“if it turns out that building AGI becomes mostly an inference problem, an inference problem that can be much more distributed than maybe synchronous large batch training”
Yeah, well, you know, we have a new and updated Flash model that applies thinking that's considerably stronger than the last model we released in January. It's out on the Gemini app. I would definitely encourage people to try that out and give us feedback. We have been incorporating developer feedback, user feedback into each model series. So, like, I think that's a good, like, that would be one thing I would encourage people to do. what jack said yeah well thank you both so much seriously it's uh such a pleasure to be able to do all this with you real pleasure yeah You
